Efficient data management practices, real-time data access and robust security measures are critical to successful implementation of AI technologies, according to a TheCUBE Research and Starburst survey of 300 U.S. and Western European IT professionals.
The survey found 87% of organizations are eager to implement AI within the next 12 months, with 86% already reporting significant progress.
Ninety percent of respondents said they believe their data management practices are somewhat or very aligned with their AI innovation goals; however, the journey is not without challenges.
A significant portion of organizations face hurdles in organizing data for AI applications, with 52% struggling with structured data and 50% with unstructured data. Data privacy and security concerns, alongside the sheer volume of data, were cited as major barriers.
Real-time data access is highlighted as a critical factor for AI success, with 62% of respondents emphasizing its importance.
The survey also found 90% of participants believe that enhanced data literacy would significantly impact the success of AI projects.
Adrian Estala, vice president, field chief data officer, Starburst, said solving for real-time analytics involves several challenges.
“One of the biggest challenges is finding a reliable and cost performant approach for ingesting large volumes of real-time data,” he said.
Another challenge is finding an efficient approach to integrating ‘streaming’ data with other data assets.
“Integrating distributed data across the enterprise often takes weeks or months for the identification and migration of new data,” Estala said.
This means AI teams need rapid discovery, ideation and experimentation–the data should be easily accessible to enable data science teams to accelerate their unique AI use cases.
Estala recommended building a strategy designed to enable materially greater access to data for traditional data science teams and an evolving self-service business consumer.
“AI innovation will be driven by the business; empower them to discover and ideate at market pace,” he said. “Modernize your data governance approach.”
That means IT departments should “shift-right” and focus on governing the data the organization’s consumers need and use.
“Establish quality and ownership at the data product layer, where end-to-end lineage can be successfully tracked,” Estala said.
Building a Data-Driven Culture
The survey also found that to maximize AI’s potential, organizations are increasingly focusing on building a data-driven culture.
Estala explained data literacy is a broad concept with many critical elements that will support an organization’s AI strategy and the broader business strategy.
“The data literacy program should cover all the ways that data is used, including AI,” he said.
Other data literacy strategies include raising awareness of data’s value (69%), fostering cross-functional collaboration (66%) and adopting cloud-based platforms and agile methodologies for scalability and effective data management.
“In a data-driven culture, insight is the real fuel,” Estala said. “When you empower teams to gain rapid insight, you power them to drive with market speed.”
The results point to the growing adoption of data governance and federated access strategies, with 52% of respondents implementing these to improve data quality and accessibility across systems, both on-premises and in the cloud.
“You can’t be data-driven if you are always waiting for or looking for data,” Estala said. “Data consumers need a data marketplace where they can ‘shop’ for and obtain secure access to the data they need.”
He pointed out not all AI projects need real-time ‘streaming’ data as it depends on the use case.
“One thing is for sure–whether that data is real-time or a daily archive, all AI teams need the ability to quickly find and access the data they need,” he said.
