Not all object stores are made equal. Of the ones that were built in the pre-AI years, many are ill-equipped to fully support modern-day AI workloads. Then again, certain high-performance solutions live up well to AI’s demands. MinIO, at the recent AI Data Infrastructure Field Day event in Calif, said that “AI bona fides” built into its object store make MinIO a default fit for AI workloads.
As a quick primer, MinIO is a software-defined, high-performance, S3-compatible object store for large exabyte-scale data infrastructure. Launched years before AI became a mainstay in enterprises, lead product manager, Rakshith Venkatesh, said, the system is targeted at production environments where everything matters – performance, scale, security and manageability.
Checking All of the Boxes
AI is continually pushing up the demand curve for storage systems. The workloads hunger for more powerful, more capacious storage systems that can not only sustain the power-hungry engines, but also come baked with specially built management features that obfuscate many of the operational pain points.
Several of MinIO’s core capabilities deeply rooted in the cloud operating model meet these specific needs, Venkatesh said. He narrated a customer success story to support the argument. In this case, MinIO helped a certain automaker deploy a 100PB cluster over a weekend. Using the AMD EPYC 64-core processor over a 100 Gb ethernet connection, a throughput of 2.2 terabytes per second was achieved, according to benchmark results.
Supporting features like MinIO’s signature metadata structure that allows data and metadata to be stored together instead of in separate databases, quick scaling, and fast distributed deployment, helped meet the specific needs of the project efficiently.
“The time it takes MinIO to bring up on a single node is the same time it takes to bring it up on 300 nodes,” he told.
Control and flexibility are a mainstay in MinIO’s solution. He noted that the company follows what it calls a “smart software, dumb hardware” philosophy. Emphasis is laid on the software and what it can do to get optimum gains from the hardware.
Scaling and the bulk of innovation happen at the software layer, making operations cheaper and more resilient. A pure Linux distribution OS with MinIO running on top of it is adequate for the solution, said Venkatesh.
Users are allowed to bring any off-the-shelf commodity hardware of their choosing, and even though MinIO recommends NVMe drives for AI-specific workloads, it also runs on HDDs, Venkatesh said.
Since public cloud, customers expect the same degree of ease and flexibility with every solution. MinIO’s built-in cloud capabilities are designed to serve this need.
Venkatesh singled out a few features to explain. One of the capabilities MinIO has is a Kubernetes native operator that lets users “scale in and out like they do application scale-in and scale-out”.
RESTful APIs behind every service, automated data management features, and a simple way of consuming the solution further enforce the cloud operating model.
To get around Day 2 challenges, MinIO has worked in a set of features that eliminates or abstracts many of the processes. For example, to keep the solution compatible with third-party tools, MinIO offers features like key management system, global console and object store catalogue all of which reduce operational pain points and add to the cloud-like experience.
MinIO’s log monitoring solution offers visibility of logs, metrics and traces across the swathe of servers and drives. However, as of today, it does not support alerting.
Out-of-the-box, MinIO offers many data integrity and security features. Some examples are erasure coding that ensures data redundancy and availability, and Bit Rot protection for prevention of data corruption. Other useful capabilities include versioning and object locking, encryption and tamper-proofing, and data firewall.
“We are extremely light on the memory footprint we use,” said Venkatesh.
MinIO’s data source-aware caching service automatically retrieves and caches all frequently accessed objects missing from the existing cache keeping them handy for future requests.
“If your workload involves small, repeated objects that your AI app is accessing, with our all-cluster cache, you can set the limits on the memory you want to use,” he said.
To learn more, head over to Techfieldday.com and watch MinIO’s technical presentations from the AI Data Infrastructure Field Day event.
