PARAS MADAN
All writing

Latest in AI

vLLM 0.34 Upgrade: Extensible KV Cache and Memory Settings to Recheck

The new default sizes cache after warmup. Check shared-GPU headroom, removed runner flags and the difference between a memory cap and disabling the feature.

vLLM 0.34 changes how an inference server decides how much GPU memory to give its KV cache. The cache holds attention state used during generation, so its capacity affects how much concurrent work the server can keep resident.

The 0.34.0 release, published November 16 UTC, enables extensible KV cache by default on CUDA and ROCm. It also removes Model Runner V1. An upgrade can therefore change memory allocation or reject an old launch setting before an application sends its first request.

I would separate those two checks. First make the existing workload start correctly. Then compare the new memory policy with the one the deployment actually used, not a remembered default from an older tutorial.

A memory cap is not the same as disabling the feature

The tagged cache configuration resolves an unspecified GPU-memory utilization to 1.0 with extensible caching and 0.92 otherwise. An explicit value takes precedence. Setting kv_cache_memory_bytes overrides the utilization setting and prevents automatic enablement of the new cache path.

That matters because the release summary suggests either a 0.9 utilization setting or disabling extensible caching to retain old sizing. Those are not identical experiments. A utilization cap restricts the memory budget; disabling the feature changes the sizing mechanism. The tagged implementation is more precise about the effective defaults.

For a rollback comparison, record the previous launch arguments and effective configuration. If production explicitly used 0.9, retain that value in the comparison rather than substituting 0.92. If it left the value unset, inspect the previous version before describing any new number as equivalent.

Why sizing happens after warmup

The implementation proposal explains the underlying problem: captured GPU graphs depend on stable cache addresses, but graph capture and warmup also consume memory. Estimating cache capacity before those allocations can leave unnecessary slack or underestimate later demand.

The new path reserves an address range first, then commits physical memory using measurements taken after warmup. This does not mean cache entries can consume every byte without a safety reserve. The implementation accounts for measured transient demand and a margin.

The proposal also reports that explicitly keeping the same utilization cap can yield a slightly smaller cache because measured allocations are now subtracted. A larger percentage in a configuration file is therefore not a direct measure of additional usable tokens.

Treat the cache-token capacity reported by the server as an observation to record. Do not infer it from GPU capacity alone, or use a larger cache as proof that every request will decode faster.

Shared GPUs still need an allocation plan

The tagged configuration calls out demand that vLLM cannot fully measure, including other GPU processes, later allocations, weight-update buffers and connector-owned buffers. A utilization setting is per instance, not a coordinator for everything using the device.

For a shared accelerator, inventory the other workloads before adopting an unset utilization value. Test the overlap that occurs in normal operation, not just an isolated startup while the GPU is empty. Keep your existing resource limits and rollback artifact available during that test.

Similarly, successful startup is not the end of a memory check. Exercise the request lengths, concurrency and optional features that the application uses. Record out-of-memory failures separately from latency regressions so the response is not simply to reduce every setting at once.

Remove the obsolete runner override deliberately

The merged Model Runner V1 removal says VLLM_USE_V2_MODEL_RUNNER=0 now causes startup failure. Setting it to 1 is redundant. Its custom logits-processor interface moves to vllm.v1.worker.gpu.sample.logits_processor.

If an old deployment set the variable to zero as a workaround, removing it is not evidence that the original problem has disappeared. Find why it was there and rerun that workload on the remaining runner. Review any custom processor against the new interface rather than assuming a successful import proves equivalent generation.

Avoid bundling that migration with a model change. Keeping the checkpoint and request fixtures fixed gives you a better chance of identifying whether a regression comes from the runtime rather than different model behaviour.

Compare serving outcomes, not just startup success

Use two candidate configurations with the same model, hardware and workload: your explicitly recorded baseline settings, and the new default policy. Measure startup time, resident cache capacity, first-token latency, inter-token latency and completed requests at representative concurrency.

These are proposed acceptance checks, not benchmarks I ran. Include output validation and a burst of longer requests; a fast short-request run does not establish adequate headroom for production traffic.

More cache is useful only when the workload benefits without breaking its latency or memory constraints. Promote the new policy when that comparison supports it, rather than treating an upgrade's default as a substitute for a capacity decision.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help infrastructure teams turn release changes and operational trade-offs into practical developer guides.

Book a 30-minute call