Prompt Caching Not Working? Diagnose OpenAI and Claude Cache Misses

A cache problem can survive an entire release cycle because the application still answers correctly. The failure shows up in token usage, a higher bill or slower responses, while the usual success-rate dashboard stays green.

Start with two requests that you expected to share context and find what changed between them. Increasing the cache lifetime before doing that can leave the underlying problem untouched, especially when the application rebuilds its instructions or tools differently on every turn.

The examples below are checked against the provider documentation, but have not been run against live APIs. They make billable requests if you execute them. Use a non-sensitive test document and a model your account can access, and check that your installed SDK supports the shown fields.

Record the symptom before changing configuration

Pick one failing conversation or repeated task. A useful diagnostic record contains the provider and endpoint, exact model ID, SDK version, request and response IDs, timestamps, relevant cache settings and usage fields. Keep the baseline and the disappointing request together so you can compare them after making a change.

Decide what you are investigating. Zero cached tokens, unexpected cache-write charges and slow responses need different evidence. A request can reuse some context while still processing new input, and output generation can dominate the time a user waits. Timing alone is a poor cache debugger.

For a first reproduction, remove application complexity: use one model, a fixed document and the same short question twice. Run the second request after the first has completed, close enough together to avoid introducing a long idle gap. Leave load tests and routing experiments until this simple case is understood.

Do not use a tiny greeting to prove that a provider's caching is broken. OpenAI's prompt-caching guide documents a minimum of 1,024 visible tokens for GPT-5.6 and later; earlier models have different rules. Claude's caching reference has model-specific minimum lengths and requires a caching configuration. Check the model you actually send, rather than copying a threshold from an older tutorial.

Compare OpenAI Responses requests

OpenAI's cache diagnostics support the Responses API on supported GPT-5.6-and-later models. Supply a recent completed response ID through prompt_cache_options.comparison_response_id, then read prompt_cache_diagnostics alongside the usage data.

This example reads a document you provide. Set OPENAI_MODEL to a supported model ID and OPENAI_API_KEY through your normal secret-management setup. The document must satisfy the cacheable-length requirement.

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI()
request = {
    "model": os.environ["OPENAI_MODEL"],
    "instructions": Path("cache-test.txt").read_text(),
    "input": "Summarize the document in one sentence.",
}

baseline = client.responses.create(**request)
comparison = client.responses.create(
    **request,
    prompt_cache_options={"comparison_response_id": baseline.id},
)

result = comparison.model_dump()
print("diagnostics:", result.get("prompt_cache_diagnostics"))
usage = result.get("usage") or {}
print("input details:", usage.get("input_tokens_details"))

The comparison ID requests diagnostics; it does not restore conversation content or change caching behavior. Measure reported reuse with usage.input_tokens_details.cached_tokens. A cache_hit comparison can still include new, uncached input.

If the result is cache_miss, inspect its reason. comparison_response_not_found and unavailable are inconclusive: select a recent baseline or check support, rather than counting them as misses. Diagnostics report the first classified cause, so repeat the comparison after correcting it. Keep the same baseline while testing a particular repair.

Opt in separately on Claude Messages

Claude's cache-diagnostics workflow requires the diagnostics object on both requests. Start with previous_message_id: null, then supply the prior response ID. The cache-diagnosis-2026-04-07 beta header is no longer required.

Set ANTHROPIC_API_KEY and CLAUDE_MODEL, and use the same eligible document. The example follows the documented client.beta.messages.create SDK surface.

import os
from pathlib import Path
import anthropic

client = anthropic.Anthropic()
request = {
    "model": os.environ["CLAUDE_MODEL"],
    "max_tokens": 256,
    "system": Path("cache-test.txt").read_text(),
    "cache_control": {"type": "ephemeral"},
    "messages": [
        {"role": "user", "content": "Summarize the document briefly."}
    ],
}

baseline = client.beta.messages.create(
    **request, diagnostics={"previous_message_id": None}
)
comparison = client.beta.messages.create(
    **request, diagnostics={"previous_message_id": baseline.id}
)

result = comparison.model_dump()
print("diagnostics:", result.get("diagnostics"))
print("usage:", result.get("usage"))

Read usage.cache_read_input_tokens alongside diagnostics, remembering that the first request normally has no reads. For a completed comparison, diagnostics: null means no divergence, while cache_miss_reason: null, missing previous records and unavailable leave the comparison inconclusive.

Fingerprints expire and must share an organization and workspace, and this feature is direct Claude API only, excluding Bedrock, Google-managed Claude and Managed Agents.

Investigate changes in the assembled request

Once the clean reproduction is useful, run the same comparison through your application. Keep a copy of the request structure in a controlled test environment and change one component at a time. Avoid logging entire production prompts just to discover that a timestamp moved.

The OpenAI caching guide identifies several settings that affect the rendered prefix, including tools, output format, reasoning effort, verbosity and compaction. The text visible in your prompt template is only part of the request. A tool registry that returns the same tools in a different order can therefore matter even when the user's message is unchanged.

Use the following checks to turn a diagnostic reason into an investigation:

What you observe	What to inspect next
The model changed	Compare the configured model with the one actually used by routing or a fallback.
Tools changed	Compare names, order, descriptions and schemas, including changes introduced by a tool-server update.
Early input changed	Look for timestamps, request IDs or user-specific material inserted before the reusable document.
Format or effort changed	Check middleware defaults and per-request overrides, not only the values in your main function.
History changed	Look for edited earlier messages, replaced tool results or a compaction step.
Matching requests still have low reuse	Check eligibility, breakpoints and time between requests before changing application code.

Treat these as questions to answer from the request pair, not proof that a particular component is faulty. For example, compare your configured tool list with the serialized list actually sent over the wire. If those differ, editing the prompt text will not address that difference.

Claude's caching guide distinguishes cache_creation_input_tokens from cache_read_input_tokens; both being zero can indicate that the prompt was too short to cache. Its default lifetime is five minutes, measured from the start of a request that writes or reads the entry. A long-running response can consume much of that window before the next request starts. Check timestamps before assuming that two consecutive turns were close together enough.

Preserve necessary changes, even when reuse falls

Some misses are a reasonable consequence of doing the job correctly. If a customer's access changes, update the context they are allowed to use. If a policy document is revised, send the revision. Keeping an obsolete prefix to improve a dashboard number would make the application worse.

A useful review question is whether the changed material needed to be at the beginning. Stable instructions and reference material can come before changing task details, provided that ordering still communicates the task correctly. Make the change in a fixture first, then check both answer quality and token usage.

Avoid deleting tools or weakening their permission checks merely to keep a request stable. When the application genuinely needs a different model or a smaller conversation, judge the resulting quality, cost and latency together. Reduced cache reuse can accompany a worthwhile reduction in total input.

Turn the repair into a regression check

Keep a small fixture containing a representative document, a fixed tool configuration and two requests that should share context. Run it when you change prompt assembly, model routing or tool registration. Save the usage fields and diagnostic outcome so a future failure has a comparison point.

Separate this integration check from a local test of your request builder. The local test can assert that the stable document and tool definitions remain unchanged when only the user's question changes. It costs no model tokens and helps catch accidental mutations before the API is involved. The integration check then tests the provider behavior that a local assertion cannot establish.

Compare several requests after a repair, including cache reads, writes, total cost and response timing. Choose an alert threshold from your workload's observed behavior rather than borrowing a universal hit-rate target. If diagnostics are inconclusive, retain that state explicitly instead of converting missing information into zero reuse.

For a handoff to another engineer, include the redacted request pair, model and endpoint, SDK version, timestamps, usage fields and the one change you tested. That gives them a reproducible problem without asking them to reconstruct it from a billing screenshot.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn API behavior and engineering decisions into clear developer guides and product stories.

Book a 30-minute call