The Kubernetes-native project adds traffic priorities, upgrade controls and stable APIs to improve performance and reduce AI inference costs.
SAN JOSE — At the PyTorch Conference, lead developers of llm-d, the Cloud Native Computing Foundation (CNCF) open-source, Kubernetes-native framework for running large language model (LLM) inference across a cluster, rolled out llm-d 1.0.
Keeping a fleet of models serving thousands of competing AI agents, without wasting accelerator capacity or taking the service down for maintenance, is a major pain. That’s where llm-d, which Red Hat helped launch in 2025, comes in.
Stephen Watt, a Red Hat Distinguished Engineer, explained during the launch presentation that they created it to bridge the gap between running inference on one machine and operating it as a shared service.
With the release of version 1.0, this Kubernetes-native distributed inference stack adds multi-tenant traffic controls, high-availability routing, and a stable configuration API to its existing performance-oriented serving capabilities. The project brings together model-serving engines such as vLLM and SGLang with Kubernetes used to operate distributed services.
This, in other words, is an infrastructure project for running LLMs, not an AI project per se. Its existing capabilities include prefix-cache-aware and load-aware routing, key-value cache offloading, prefill/decode disaggregation, and wide expert parallelism for distributing mixture-of-experts models across nodes.
The challenge was not simply fitting model weights into accelerator memory. It was supporting development and production workloads, different model sizes, and operational requirements while getting more useful work from existing infrastructure. As Watt put it, the stack needed to meet “the day-to-day operational expectations of the IT and SRE teams for running production services.”
Version 1.0 focuses on keeping those capabilities useful after the initial deployment, when traffic patterns change, users compete for capacity, and operators need to upgrade software. As Carlos Costa, an IBM Distinguished Engineer, explained during the presentation: “When your traffic is competing for priority, you have to do those rollouts for updates and maintenance. All of these without your service going down.”
To accomplish this, llm-d has added four new features. These are:
- Multi-tenant quality of service uses priority- and fairness-aware flow control to let latency-sensitive requests and bursty agent workloads share infrastructure without starving one another.
- High-availability (HA) operation includes router hot standby, rolling updates and coordinated management of prefill and decode replicas.
- A stable v1 configuration application programming interface (API) and hardened request path give platform teams a foundation they can standardize on.
- Agent-aware serving combines session-aware routing, turn-priority scheduling, and native Responses API state support to help preserve and reuse context across agent turns.
The upgrade controls are especially important for disaggregated serving, which separates processing an incoming prompt, or prefill, from generating output tokens, or decoding. Its DisaggregatedSet component manages these pools as one unit for coordinated updates while allowing each phase to scale independently.
That all sounds good, but does it work? Abdullah Gharaibeh, a Google Cloud Senior Staff Software Engineer, showed it with a live demo featuring 1,000 sandboxed agents sending requests to a shared inference service. The demonstration compared a baseline Kubernetes setup with the llm-d router and then with llm-d flow control enabled. It used vLLM and SGLang across a mixed fleet of Google Cloud TPUs and GPUs.
According to Gharaibeh, in the baseline setup, the fleet produced roughly 900 to 930 tokens per second, with a cache hit rate of about 30%. However, after switching llm-d routing on, we saw a much more impressive cache hit rate of 75%. Throughput roughly doubled, while latency fell by half. That was enough to make the audience sit up and take notice.
The point was that routing, not additional hardware, drove the performance improvement. “This is the same hardware,” Gharaibeh said. “Same model server, same traffic,” but drastically better performance. I was impressed, and I’m not easily impressed.
It does this by context reuse. Multi-step agents repeatedly submit requests containing shared conversation history, instructions, or other prompt prefixes. Sending a request to a server that already holds the corresponding cached state avoids recomputing that work. llm-d’s router considers cached context as well as load and predicted latency when choosing a model-server replica.
That makes routing more than a matter of distributing an equal number of requests to each server. The useful destination depends partly on what each server already remembers.
The demo also showed why higher aggregate throughput does not, by itself, solve the production problem. It needed more, and enabling flow control provided it. Without flow control, requests from different user tiers waited in a shared queue. That meant, Gharaibeh noted, that paid-tier users could wait as long as free-tier users. That doesn’t make for happy customers. But after enabling flow control, the demo showed much shorter waits for the highest-priority users while lower-priority requests still received service. Or, as he said, “The pro members waiting the least, the free members waiting the most, but they’re all still getting served.”
That matters a lot for providers supporting interactive customers alongside background agents or other bursty workloads. A system can be busy and efficient overall while still failing to meet response-time commitments to its highest-priority customers. llm-d 1.0’s flow-control features let operators enforce those priorities while using remaining capacity for other work.
Mistral’s inference engineers explained in a Google press release that llm-d’s prefix-cache-aware routing, flow control and disaggregated serving allow the company to enforce priority bands across its commercial tiers. When capacity is saturated, the system can hold or shed lower-priority requests to protect latency objectives for paying customers.
llm-d 1.0 is already in production. DigitalOcean uses llm-d to underpin its Serverless Inference and Managed Agents services, which serve trillions of tokens. JetBrains says the software runs in production behind its Junie Lite coding-agent plan and handles a significant share of that traffic.
Looking ahead, Google is integrating the llm-d stack with Google Kubernetes Engine (GKE). This deployment will include a managed routing path via the GKE Inference Gateway and support for TPU and GPU pools behind a common endpoint.
The companies behind llm-d also claim substantial performance and cost improvements. They state it can deliver estimated per-token costs five to 10 times lower than equivalent commercial API pricing. Of course, you might not get the same results, but if llm-d can deliver even half of what it promises, it will be a success.
As Mark Collier, the PyTorch Foundation’s executive director, recounted at the demo’s end, the team’s aim was straightforward: “We want to be able to take it into these production environments, cloud native environments, and have people really be able to trust it.” I’m pretty sure after what I saw they’ll see that happen.
