AKV Cache Engineering: How Agents Can Reduce Inference Costs

Tsinghua researchers published Cache Engineering for Agents on October 20, 2026, alongside an implementation called AKV. It lets an agent control the attention cache directly instead of repeatedly shortening its conversation. The reported benefit is lower inference cost on long tasks without an average loss in benchmark accuracy.

For developers running their own models, the interesting question is where the optimization happens. Changing an agent's prompt is relatively easy. Changing which internal states an inference server retains is a different kind of integration.

What cache engineering changes

A key-value cache stores intermediate attention states so a language model can reuse earlier computation. As an agent reads files, searches and calls tools, this cache grows with its working history.

The AKV project page explains the problem with editing the beginning of that history: later cached states depend on the original prefix. Removing text can require the server to process the remaining context again.

AKV separates the conversation record from the states used for future attention. Its drop operation excludes selected cache entries while retaining the other computed states. A separate reposition operation compacts their positional encodings. The original message sequence remains available.

That distinction also affects reuse. Two requests containing identical text may have different internal states if they dropped earlier information at different times. AKV tracks cache operations alongside tokens rather than assuming a matching text prefix is enough.

What the benchmark numbers actually measure

The paper reports a 27.7% average reduction in calculated inference cost and a 1.7-point increase in average benchmark score against its standard summarization baseline. These are research results across three models and four agent benchmarks, not a price reduction offered by an API provider.

The evaluations use GPT-OSS-120B, MiniMax-M2.7 and MiroThinker-1.7-Mini. Tasks include software engineering and deep research. The main cache policy retains the latest 12 tool responses, rather than preserving every old tool result indefinitely.

The cost calculation applies published token rates to active input and generated output, including summary calls. It also charges repositioned tokens at a cache-hit rate. That accounting is useful for comparing strategies, but it is not a measured saving on every production bill.

Serving tests use eight A800-80GB GPUs. At the compared peak-throughput configurations, AKV reports about 1.18 times the output throughput and 1.51 times faster P95 request latency than the standard baseline. Individual task scores do not improve uniformly: MiniMax-M2.7's SWE-bench Verified result falls from 75.2% to 74.4%.

How developers can use AKV

The Apache-2.0 repository contains an agent-side policy component and an SGLang-based serving system. The policy decides which message caches to discard and when to reposition them. The server executes those operations through an extended OpenAI-compatible Chat Completions interface.

The quickstart demonstrates GPT-OSS-120B and adds cache-operation fields to a normal message payload. OpenAI-compatible does not mean an ordinary hosted endpoint will execute these controls. The serving implementation has to understand them.

This makes AKV most relevant to teams that own their inference stack and run long tool-heavy tasks. A product that simply calls a closed hosted model cannot assume the same optimization is available by adding fields to its request.

What to test before adopting it

Dropping an old search result may be harmless after the agent has extracted what it needs. Dropping the only source of a constraint may make a later answer wrong. Cache selection remains a task-quality decision, even when the underlying server handles reuse correctly.

A useful evaluation would pair completed-task quality with GPU time, tail latency and cache pressure. It should also include tasks that revisit older evidence, where an aggressive rolling policy is more likely to hurt.

AKV's practical contribution is a clearer boundary between the history an application keeps and the history a model actively attends to. For long-running agents, owning that boundary may become as important as choosing a larger context window.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders explain AI systems and engineering tradeoffs in content that developers and buyers can understand and share.

Book a 30-minute call.