TL;DR — Key Takeaways
- MLPerf Inference v6.1 adds end-to-end RAG and agentic AI tests as the benchmark adapts to more complex inference workloads.
- More than half of submitters used MLPerf’s new API-based harness, which will underpin MLPerf Endpoints and eventually replace the datacenter Inference benchmark.
- Nvidia previewed Vera Rubin performance, while AMD and Intel reported sizable gains from software optimization on existing hardware.
MLCommons released MLPerf Inference v6.1 today with new tests for retrieval-augmented generation and agentic AI, expanding the benchmark to cover more complex, multi-step inference systems.
A new End-to-End RAG benchmark measures both document ingestion and a complete question-answering pipeline, including embeddings, vector retrieval, reranking and generation. A second new benchmark measures agentic inference at the edge, where models work through multi-step tasks with growing context and repeated reasoning.
Those additions come as MLCommons prepares a larger change to its inference testing. More than half of v6.1 submitters used a new API-centric harness built around a client/server architecture that sends inference queries and results to the system under test over industry-standard APIs. MLCommons said the harness will serve as the foundation for MLPerf Endpoints, which will replace the current Inference benchmark for datacenter systems.
How MLPerf Inference Is Changing
MLCommons is an open AI engineering consortium that brings together companies, researchers and academic institutions to develop shared evaluations for AI systems. Its MLPerf benchmark suites are designed to provide reproducible, comparable tests of AI hardware and software across areas including training, inference and storage.
For MLPerf Inference, submitters run defined workloads under common rules, with Closed Division results intended for direct comparisons within the same benchmark and scenario. The suite has evolved along with the workloads it measures, adding larger generative AI models and tests that account for longer context windows and interactive serving. MLPerf Inference 6.1 continues that evolution by bringing RAG pipelines and agentic workloads into the benchmark.
The v6.1 results also show how Nvidia, AMD and Intel are approaching inference performance from different directions, including new hardware and software improvements.
Vera Rubin Makes Its MLPerf Debut
Nvidia submitted preview results for its upcoming Vera Rubin NVL72 platform, reporting up to 3.7 times the throughput of its GB300 NVL72 Blackwell Ultra system on the Qwen3-VL benchmark and up to 2.5 times the throughput on DeepSeek-R1. The results on these demanding vision-language and reasoning workloads offer an early look at the generational performance gains Nvidia is claiming for Rubin.
The company also reported 99% scaling efficiency in the DeepSeek-R1 Offline test when expanding from one 72-GPU GB300 NVL72 rack to four racks. That result addresses whether throughput can continue growing roughly in proportion to the amount of hardware added as deployments expand across racks. Nvidia attributed the scaling efficiency to its interconnects, networking and request orchestration working together across the system.
AMD and Intel Show Software-Driven Gains
AMD’s most notable results came from performance gains on existing Instinct MI355X hardware rather than a new accelerator generation. On the same eight-GPU configuration used in MLPerf 6.0, AMD said GPT-OSS-120B throughput improved 28% in the Offline test and 38% in Server, while Wan 2.2 single-stream performance rose 70%.
Brendan Burke, research director at Futurum, wrote that the same-hardware results make ROCm’s improvement easier to measure from one MLPerf cycle to the next, particularly now that AMD plans to release ROCm updates approximately every six weeks. Future benchmark rounds should make it easier to see whether those gains continue at a similar pace. Burke cautioned, however, that one round of large improvements is not enough to establish a sustained trend.
AMD’s submission also demonstrated the MI355X at a much larger scale. AI infrastructure firm Crusoe submitted GPT-OSS-120B and DeepSeek-R1 results using 512 MI355X GPUs, the largest GPU count submitted in MLPerf Inference to date.
Intel’s results also told a software optimization story. Using the same two-socket Xeon 6980P hardware as MLPerf 6.0, Intel reported 2.4 times the Llama 3.1 8B throughput in the Server test and 56% higher throughput in Offline. Intel attributed the gains to software improvements. The company also co-developed the new end-to-end RAG benchmark. Its submission used a Xeon 6787P CPU for embedding, reranking, vector search and a small language model, while four Arc Pro B70 GPUs ran the LLM generation stage. The company said the result demonstrates how optimized CPU and GPU compute can work together across a complete AI workflow.
Hardware and software optimization aside, the most important change in v6.1 may be what MLPerf is choosing to measure. As inference moves toward more complex production systems, MLPerf is adapting its benchmarks to match. Read more and see the results at this link.
