# Amazon ElastiCache for LLM KV Caching: Test TCP Before RDMA

Use KVTrace to decide whether a shared cache pays off, choose the right LMCache transport, and prove that a second GPU can reuse the first GPU's work.

By Paras Madan | 2026-11-22


A shared KV cache is worth testing when a user returns to a long conversation and lands on a GPU that has never seen it. If the request lands on a worker that already has those tokens in memory, sending the same state over the network adds little. That difference is the first thing to measure before you pay for another cache.

Amazon [made distributed KV caching generally available on November 18](https://aws.amazon.com/about-aws/whats-new/2026/11/announcing-distributed-kv-caching-for-self-hosted-llms-with-amazon-elasticache/) and [published an evaluation and setup workflow on November 20](https://aws.amazon.com/blogs/machine-learning/cut-self-hosted-llm-cost-and-latency-with-the-distributed-kv-caching-agent-skill/). The service stores attention state that workers can reuse during prefill. It does not store a finished answer or make new output tokens cheaper to generate.

## Measure what the current fleet recomputes

Start with time to first token by input length, local prefix-cache hits and how often resumed sessions change workers. A workload dominated by short requests or mostly warm local prefixes is a poor first candidate. A long, repeated document prefix that frequently crosses replicas is more promising because the alternative is to prefill it again.

Use KVTrace to replay a representative slice of cache events across your workers. Include the afternoon return to a morning conversation if that is where users wait. The tool estimates hit rate and break-even cost for different shared cache sizes. AWS says traces store block hashes and token counts rather than prompt text. For sensitive traffic, review the trace handling and use the documented anonymization option before recording.

Separate GPU time saved from money saved. If the fleet size and traffic stay fixed, faster prefill may buy better response times and headroom, but your GPU bill will not drop by itself. Compare the cache bill with the capacity you can actually remove or the traffic you can now serve.

## Test the transport you intend to use

AWS documents [three LMCache routes](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/distributed-kv-caching-how-it-works.html). They're useful at different stages:

- `tcp`: ordinary Valkey `SET` and `GET` over TLS. The values stay in node memory. Use this for a functional smoke test.
- `blob-tcp`: sends large objects over TLS. This exercises the large-object path without requiring Elastic Fabric Adapter (EFA) on the client. It won't measure RDMA performance.
- `rdma`: transfers the data over EFA, the route AWS recommends for production workloads that need high bandwidth. This requires GPU hosts that support EFA RDMA writes.

AWS's documentation calls for LMCache 0.5.8rc1 or later for `blob-tcp` and `rdma`; its stable-release note describes only `tcp` support. Check the version you plan to pin. Start with vLLM or SGLang if you want the documented setup. TensorRT-LLM support is listed in the LMCache development branch.

## Keep both workers on the same layout

The [AWS setup guide](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/distributed-kv-caching-getting-started.html) calls for Valkey 9.1 or newer on `cache.i8ge.48xlarge` nodes in the same Availability Zone as your GPU hosts. Plan shard count before provisioning: the setup guide says you cannot change it later.

In an isolated staging deployment, use the same checkpoint revision, tensor-parallel layout, KV data type, hash settings and LMCache version across workers. Use separate [key prefixes](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/distributed-kv-caching-configuration.html) for incompatible builds or key types. In particular, regular `tcp` must not share a prefix with the two large-object transports. Keep the cache dedicated rather than reusing a cluster that holds application state.

## Force a hit on a different host

Write one test prefix on worker A. Use a second GPU host without a local copy of that prefix and send it the same input and generation settings. Confirm worker B read from the shared tier, then compare the output against a cache-disabled baseline using fixed decoding settings. A hit counter can tell you that bytes moved. It can't tell you those bytes belong to the right model layout.

Repeat with short and long prefixes under load. Compare median and p95 time to first token against local-only serving and check that disabling the shared tier restores normal inference. There is no backup or replication for the cached data. Treat a missed read as an expected event and keep enough prefill capacity to handle cold starts.

I'd start with one model in one zone. If a forced cross-host hit produces the expected output and improves latency at a cost the team can justify, then test the production transport before moving more traffic.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI infrastructure teams turn deployment details and benchmark findings into technical writing developers can use.

[Book a 30-minute call](https://cal.com/paras-madan/30min)
