Kinference Brings Cache-Aware Routing to vLLM, With Important Alpha Limits

A Kubernetes cluster can keep inference servers running without knowing which server has already processed the beginning of your prompt. That gap matters when requests repeatedly carry the same system instructions, documents or conversation history.

Buoyant introduced Kinference on November 4 to give inference routing more of that context. The launch announcement on the vLLM blog presents an inference control plane for Kubernetes and vLLM, with ambitions spanning cache-aware routing, token-based controls and policy enforcement.

The useful distinction for developers is between that broader product direction and the code available today. The current release is explicitly alpha, and several advertised capabilities remain outside its implementation.

What the current project does

According to the Kinference repository, its core service examines an inference request and returns a chosen endpoint plus a modified request body over gRPC. It watches vLLM's KV-cache events and engine metrics, using NVIDIA's dynamo-kv-router component to handle cache events.

Kinference is designed to work alongside a proxy, such as Buoyant Gateway. Kubernetes custom resources configure its behavior, and the included kinfctl tool can validate policies before they reach the server. The project uses the Apache 2.0 license.

This makes the routing decision a separate concern from generating tokens. For a platform team, that separation offers a place to evaluate request placement without changing the model itself.

Read the alpha limitations first

The repository says Kinference has not had sustained production exposure. It does not yet support Anthropic-style /v1/messages requests, rate limiting, semantic classifiers or multiple replicas.

Those gaps matter. A team that needs token quotas immediately should not treat the launch's broader feature list as a description of completed functionality. The replica limitation also deserves attention before relying on the control plane for a production service.

The launch reports substantial performance gains in selected workloads. Those are vendor results, rather than a guarantee for an existing deployment. Cloud Native Now's reporting adds context from Buoyant about self-hosting and service quality, but does not independently reproduce the benchmarks.

How to evaluate it without overreading the launch

A useful trial would compare Kinference with the team's existing routing on the same hardware, model and request mix. Include requests with repeated prefixes, unrelated prompts and longer conversations. Cache reuse has a different opportunity in each case.

Measure time to first token, total response time and completed requests within the application's latency target. Track queueing and failed requests too. An improvement in a friendly cache-reuse workload is useful evidence, but it should not stand in for the traffic the service actually receives.

Operational tests matter alongside those measurements. Check what happens when an inference endpoint disappears, when routing information becomes stale and when the control plane restarts. The alpha label makes these questions part of the evaluation, not a detail to defer until rollout.

Kinference is an interesting new option for teams already running vLLM on Kubernetes. Its immediate value is a concrete implementation to inspect and test, with a clear list of unfinished pieces to factor into the decision.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI infrastructure teams turn technical launches into clear stories developers can use.

Book a 30-minute call.