llm-d 1.0: What Changes for Multi-Tenant AI Inference

The llm-d maintainers released version 1.0 on October 20, bringing a stable v1 configuration API to the distributed AI inference stack. The release focuses on running shared model fleets reliably, including multi-tenant policy, coordinated upgrades and agent sessions, rather than introducing another model server.

For teams already serving open models, the practical question is what happens when several workloads compete for the same accelerators. A coding agent with a growing conversation, a customer-facing chat and an overnight batch job should not necessarily receive the same treatment.

What llm-d does alongside vLLM and SGLang

llm-d adds a routing and coordination layer around inference engines. The engine executes the model; the surrounding system decides where a request should run and how resources should be shared.

The original architecture already included three useful ideas: directing requests toward replicas with reusable prompt context, separating prompt processing from token generation, and storing attention state across multiple memory tiers. Google described these in its original llm-d introduction.

That attention state is the key-value, or KV, cache. Reusing it avoids processing the same prefix repeatedly. Prefill processes the input prompt, while decode generates the answer token by token. Separating these phases lets operators allocate resources to their different compute and memory demands.

A faster engine and a better routing policy solve different parts of the serving problem. A spare replica can still be a poor destination if another replica already holds most of the context a request needs.

What changed in llm-d 1.0

The 1.0 release announcement emphasizes a stable configuration surface and operating the components together. Its agent-aware features include session affinity, turn-aware scheduling and Responses API state through previous_response_id.

Session affinity keeps follow-up requests near their existing cache, then allows rebalancing when that replica becomes saturated. This is useful for agents because successive turns often reuse much more input than they add.

It is important not to describe every operational feature as brand new. The August 28 version 0.9 release already introduced router high availability, graceful draining, plugin maturity gates and revision-aware routing for split prefill/decode deployments.

Version 1.0 is therefore a milestone in stabilization and integration. Teams evaluating an upgrade should compare their actual configuration with the release, rather than assume they need to replace working cache or routing components simply because the major version changed.

How multi-tenant flow control works

The flow-control documentation explains the underlying mechanism: requests are grouped by tenant and priority, held in queues at the Endpoint Picker, then dispatched when the backend has room. Priority decides which class goes first; fairness decides how tenants within that class take turns.

Keeping the queue at the router delays a routing decision until useful capacity is available. Sending work too early into an overloaded model server's private queue makes that decision harder to change.

There is an important deployment detail here. Tenant and priority headers should come from a trusted gateway or identity layer, not directly from an untrusted client. Otherwise, every caller could label their own traffic as premium.

The release says the default behavior remains first-come-first-served until operators choose multi-tenant policy. Installing the software does not decide the business priority of different workloads for you.

High availability does not remove failure handling

The 1.0 announcement still lists request failover and migration as future work. High-availability routing should not be read as a guarantee that every in-progress request survives every failure.

The versioned flow-control documentation also describes in-memory queues, retryable responses during graceful draining and connection loss after abrupt failure. It explains another tradeoff clearly: flow control moves waiting to a safer place, but cannot manufacture missing accelerator capacity.

For an operator, that means testing retries, queue limits and shutdown behavior alongside throughput. A benchmark run with one tenant will not show whether an interactive customer remains responsive while a large batch workload arrives.

Who should evaluate llm-d 1.0

This is most relevant to platform teams hosting multiple workloads or tenants on their own inference infrastructure. A small application using a managed model API will not gain much by adding another orchestration layer it does not need.

For a shared fleet, the next useful test is a realistic traffic mix: repeated agent turns, short interactive requests and background batches. Measure waiting time by priority class, cache reuse and recovery during an upgrade, not only total tokens per second.

The meaningful outcome is a fleet that can do more useful work without making its most important users wait behind everyone else's queue.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders turn infrastructure launches and complex engineering work into clear stories developers can understand and share.

Book a 30-minute call