Managing resources with AI took on a whole new meaning when at Next ‘24, last April, Google Cloud launched a new capability for its cloud platform enabled by Gemini. Storage insight generation promises to relieve analysts and storage administrators of the pains of manually analyzing and managing storage footprint on Google Cloud Storage. The new feature, Google said, can produce deep analytics to natural language queries, making management and optimization of the storage infrastructure easier than was formerly possible.
According to Q2 reports, Google Cloud commands 12% of the global cloud market share, with a run rate of $41 billion. Cloud Storage makes a central product on its portfolio, and is a core platform for cloud deployment across organizations.
“Google Cloud Storage is cheap, easy, pay-as-you-go, and scales almost infinitely,” said Manjul Sahay, group product manager, at the AI Data Infrastructure Field Day event hosted by Tech Field Day, part of The Futurum Group, where he demoed storage insight generation to the audience. “But when you have billions of objects, thousands of buckets, tens of projects and petabytes of data, there’s something different about management.”
Sahay described storage management as a two-step exercise that begins with analyzing the estate and subsequently, performing the required actions. The standard way of doing the analysis entails exporting all of the metadata and running it through an analytics engine to run queries and get insights.
“If you find some great insights, you build scripts to take actions. You want to make sure nothing is inadvertently opened to public internet because that’s where you see some of the worst security scenarios happen,” warned Sahay.
Scale, he said, is a formidable obstacle when pushing changes downstream across a near-infinite number of objects and projects. It is neither easy, nor quick.
“The management solution must analyze the storage estate and take actions, both at the scale of billions of objects.”
Google Cloud addresses this with a shift-left approach where management, instead of being concentrated to a handful of infrastructure and storage administrators, is shifted closer to the users.
Built on top of Insights Datasets, a managed metadata warehouse powered by BigQuery, generate insights with Gemini is a solution designed to perform algorithmic-based environment-specific analysis, and produce automated insights for administrators to work with.
“We take all the metadata across billions of objects, bring that through a pipeline and put it in BigQuery, our analytics offering. Once the data is available there, it gets snapshotted daily.”
Insights Datasets resides inside Google Cloud’s object storage. Insights can be generated on any dataset by selecting the dataset from a drop-down menu, he explained. As the application analyzes the selected dataset, it produces a short summary of the corresponding metadata that includes the total number of projects, size of the object, number of projects per TB and buckets per project, used and free storage, highest storage project to lowest storage project, and so on.
Storage insight generation reports when an object is created, who created it, the object locks and bucket locks applied, metadata including custom metadata which a customer may have created.
On the dashboard, insights are grouped under three broad categories – Security and Compliance, Cost Management, and Operational Insight.
“Every day we give you tens of billions of object metadata,” said Sahay.
A set of pre-curated natural language prompts allows users to get validated responses to questions without any effort. Information is communicated via graphs, charts and other visual representations.
There are no hallucinations in the verified responses for top questions and customers can have complete trust in the data, said Sahay.
Alternatively, users can also craft their own natural language prompts to carry out investigation. The multi-turn chat feature of Gemini supports a highly interactive analysis with follow-up questions, and lets users dig for more context.
“They can do deep-dive analysis by starting with one question, and asking a second, and a third, doing their own custom analysis and generating on-demand graphs.”
The AI-generated query that the system runs behind the scenes is shared to let users reuse it on BigQuery to run more complex analysis.
“Essentially storage insight generation gives power users like analysts a great point to start. It’s like a cold start problem solution.”
Storage insight generation is available through Google Cloud console and is currently in experimental preview.
Head over to techfieldday.com for the solution demo and other presentations by Google Cloud at AI Data Infrastructure Field Day event.
