# SageMaker KV-Aware Routing: When to Switch From Prefix-Aware Routing

How to configure SageMaker AI's new cache-aware routing, check cache reuse and avoid mistaking an enabled setting for a faster endpoint.

By Paras Madan | 2026-11-18


AWS introduced KV cache-aware routing for SageMaker AI real-time endpoints on November 17. It targets a specific waste: sending a prompt to an instance that must recompute context another instance has already cached. The practical question is whether your traffic contains enough reusable context to justify changing how requests are routed.

## Prefix-aware guesses; KV-aware checks

The [routing API reference](https://docs.aws.amazon.com/cli/latest/reference/sagemaker/create-endpoint-config.html) distinguishes two strategies. `PREFIX_AWARE` groups requests using a shared starting segment and redirects traffic when a concurrency threshold is reached. `KV_AWARE` considers actual cached matches and instance load. Without a suitable available match, it falls back to least-outstanding-requests behavior.

That distinction matters for conversations whose history grows and caches that change under load. A fixed prefix can be a useful routing hint without proving that the destination still holds the cached work. I would consider the new strategy for a busy assistant with long, repeated context before trying it on short, unrelated requests.

## Check compatibility and isolation first

The [AWS launch guide](https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware-routing/) requires a supported AWS vLLM container, enabled prefix caching and cache-event reporting. Routing needs at least two endpoint instances or inference-component copies to have a choice of destination. A single-copy deployment can accept the setting without benefiting from it.

There is also a migration constraint: KV-aware routing ignores the prefix-aware ID header and `prompt_cache_key`. A non-null top-level `cache_salt` produces a 400 error. Do not switch a multi-tenant application on the assumption that its existing prefix-isolation controls carry over. Review that requirement before testing performance.

## Configure the model and production variant

AWS's [example notebook](https://github.com/aws-samples/sagemaker-genai-hosting-examples/blob/main/03-features/kv-aware-routing/kv_aware_routing.ipynb) uses the vLLM 0.33.0 CUDA 13 image with the compatible `al2023-ami-sagemaker-inference-gpu-4-1` AMI. Its model environment enables events with this JSON-serialized configuration:

```python
kv_events_config = {
    "enable_kv_cache_events": True,
    "publisher": "zmq",
    "endpoint": "tcp://*:5557",
    "topic": "kv-events",
    "replay_endpoint": "tcp://*:5658",
}
env["SM_VLLM_KV_EVENTS_CONFIG"] = json.dumps(kv_events_config)
env["SM_VLLM_ENABLE_PROMPT_TOKENS_DETAILS"] = "true"
```

This is a configuration excerpt, not a standalone deployment script. In the endpoint configuration's production variant, use `"RoutingConfig": {"RoutingStrategy": "KV_AWARE"}`. The API reference reserves `PrefixAwareRoutingConfig` for `PREFIX_AWARE`; do not carry its tuning fields into the new strategy.

Check GPU quota, execution-role permissions and the image/AMI combination before creating resources. A routing choice cannot fix a container that fails to start.

## Look for cache reuse, not just successful responses

With prompt-token details enabled, inspect `usage.prompt_tokens_details.cached_tokens`. A follow-up request should provide evidence of reused context, not merely an HTTP success. The notebook records client-side time to first token alongside cache reuse and CloudWatch latency metrics.

For ongoing monitoring, AWS's [observability documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-detailed-observability-promql.html) provides PromQL examples for throughput, first-token latency, KV-cache utilization and queue depth. Its CloudWatch endpoint uses SigV4 authentication and requires `cloudwatch:GetMetricData` and `cloudwatch:ListMetrics` permissions.

I would examine queue depth alongside cache reuse. More cache hits are not the goal if users spend longer waiting for a busy instance.

## Compare against the strategy you already use

I have not run this benchmark. My proposed trial would compare the current prefix-aware endpoint with KV-aware routing using the same model, capacity and request sequence. Separate initial turns from follow-ups, and record errors as well as median and tail latency. Include peak traffic rather than evaluating only an idle endpoint.

The sample runs random and KV-aware routing sequentially, so its baseline is not an existing prefix-aware deployment. Adapt the comparison before drawing a migration conclusion. Repeat with fresh documents to check variation, and verify cleanup after any interrupted run. Test endpoints continue billing while they exist.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams explain AI infrastructure and product changes in terms their users can act on.

[Book a 30-minute call](https://cal.com/paras-madan/30min).
