Prompt Cache Hit Rate: Why Your Cached Prompt Isn't Caching

The Prefix Invariant, and Its Order

Anthropic's documentation states the render order plainly: cache prefixes are created in the order tools, system, then messages, and each level builds on the previous ones. Everything else follows from that single fact.

Three mechanics are worth internalising, because they explain almost every zero hit rate:

• A write happens only at your breakpoint. Marking a block with cache_control writes exactly one entry: a hash of the whole prefix ending at that block. No entries are written for earlier positions. The hash is cumulative, so changing anything at or before the breakpoint yields a different hash next time.
• A read walks backward looking for earlier writes. On each request the system hashes the prefix at your breakpoint and looks for a match. If there is none, it steps back one block at a time and checks each earlier position. It is hunting for entries previous requests actually wrote — not for content that merely looks stable.
• That backward walk is bounded at 20 blocks, counting the breakpoint itself as the first. If no match turns up inside the window, checking stops, or resumes at the next explicit breakpoint if you set one.

The 20-block bound is the one that ambushes agent loops. A single turn that appends fifteen tool_use and tool_result pairs adds thirty blocks; the next request's breakpoint never reaches back far enough to find the previous turn's entry, and a conversation that was hitting reliably starts paying full price with nothing in the code having changed. The documented fix is to add another explicit breakpoint close enough to the previous write before the growing conversation puts the final breakpoint 20 or more blocks past it.

One more line from the documentation deserves to be read literally: cache hits require 100% identical prompt segments, including all text and images, up to and including the block marked with cache control. Not equivalent, not semantically the same — identical bytes.

The Invalidators You Wrote Yourself

Almost every "my cache does nothing" report is a prompt-assembly bug rather than a caching bug, and the same handful of causes keep recurring. Grep your prompt-building path for these before you touch anything else.

A clock or a request id in the system prompt. This is the most common one by a distance, because "current time" feels like context and lands at the front of the prefix, where it poisons everything behind it.

# Every request produces a unique prefix. Nothing after this line can ever hit.
system = f"""You are the support assistant for {tenant.name}.
Current time: {datetime.now().isoformat()}
Session: {uuid4()}

{POLICY_TEXT}"""

if tenant.beta_summaries:          # a second prefix variant, silently
    system += BETA_SUMMARY_RULES

Three separate faults sit in six lines. The tenant name gives every customer a private cache namespace. The timestamp and session id make every single request unique. The conditional block splits whatever is left into one distinct prefix per flag combination — four flags give sixteen prefixes, each warming and expiring on its own.

The repair is not to delete the context. It is to move it past the breakpoint, because a value that appears at message position five invalidates nothing before message position five.

system = [
    {"type": "text", "text": POLICY_TEXT,          # frozen: no interpolation at all
     "cache_control": {"type": "ephemeral"}},
]

messages = [
    {"role": "user", "content": (
        f"<request_context>tenant={tenant.name} time={now.isoformat()} "
        f"beta_summaries={tenant.beta_summaries}</request_context>\n\n{question}"
    )},
]

Non-deterministic serialization. A tool schema rendered with json.dumps(schema) depends on insertion order; anything built by iterating a set depends on hash seeding and can differ between processes on the same deploy. That last one produces the worst version of this bug: a hit rate that varies by which worker served the request.

canonical = json.dumps(schema, sort_keys=True, separators=(",", ":"))
tools = sorted(build_tools(user), key=lambda t: t["name"])

A tool list that varies per user. Tools render at position 0, so build_tools(user) is a decision to give every user their own cache. It is also a decision you almost never made deliberately — it is usually a permission filter that was correct in the executor and got applied one layer too early. Send the full deterministic list and enforce permissions when the call comes back, or, if the tool surface genuinely has to be dynamic, look at tool search, which appends schemas rather than swapping the set.

The Invalidators the API Documents

The other half of the problem is request parameters that are not part of your prompt text but are rendered into the prefix anyway. Anthropic publishes an invalidation table organised by the same three levels, and the useful reading of it is which level each change reaches down to:

• Tool definitions — names, descriptions, parameters. Invalidates the entire cache, tools included. Adding one tool mid-conversation throws away the whole prefix.
• Web search toggle, citations toggle, speed setting. These modify the system prompt, so the tools cache survives and system plus messages do not.
• Tool choice, images, thinking configuration, effort setting. These reach the message blocks only; tools and system survive.

The third row is the one that catches teams doing sensible cost work. A router that drops output_config.effort from high to low for short questions invalidates the message cache on the request where it switches, and switching back invalidates it again. On a long conversation, per-turn effort tuning can cost more in re-processed message blocks than it saves in reasoning tokens. Measure that pairing before shipping it; it is not obvious from either feature's own documentation.

Read in the other direction, the same table is permission: tool_choice can vary per request and images can come and go without touching the cached tools and system prefix. Those are cheap to change.

The published table is organised by level rather than by every possible request field, and it does not enumerate the model id. Treat a model change as a cold start unless you have measured otherwise on your own traffic — and note that a rendered prefix that clears one model's minimum may not clear another's, which the next section covers.

Put Breakpoints at Stability Boundaries

You get up to four cache_control breakpoints per request. The instinct is to spend them at the end of the prompt, and that instinct produces the most expensive failure mode available: if the marker sits after content that differs on every request, each call writes its own distinct entry at the write multiplier and no call ever reads one. A cache that only writes is strictly worse than no cache.

Sort your inputs by how often they change, then check that the rendered order matches:

• Never changes — the frozen system prompt, the deterministic tool list. First in the prefix, before every breakpoint.
• Per session — the retrieved document set, the conversation so far. After the global prefix; give it its own breakpoint if sessions are long enough to pay for one.
• Per request — the question, the timestamp, the request id. After the last breakpoint, always.

Two placements cover most workloads. For a large shared preamble with a varying question, the breakpoint goes at the end of the shared portion, not the end of the prompt. For a multi-turn conversation, it goes on the last content block of the most recently appended turn, so each request reuses everything before it and hits accrue as the conversation grows.

Then there is the case where the honest answer is not to cache. If the first thousand tokens differ per request, there is no reusable prefix, and adding cache_control only buys you the write premium. Leave it off and spend the effort on the prompt structure instead.

The Minimum Cacheable Prefix Is Not Monotonic

Below a per-model token floor, a marked prompt is processed without caching and no error is returned. This is the failure that looks least like a failure: the code is right, the marker is there, and the number stays at zero.

The floor differs by model, and — the part that costs people a debugging session — it does not fall reliably as models get newer. These are the published minimums as checked on Aug 16 2026:

• 512 tokens — Claude Opus 5, Claude Fable 5, Claude Mythos 5.
• 1,024 tokens — Claude Opus 4.8, Claude Opus 4.1, Claude Opus 4, Claude Sonnet 5, Claude Sonnet 4.6, Claude Sonnet 4.5, Claude Sonnet 4.
• 2,048 tokens — Claude Opus 4.7, Claude Mythos Preview, Claude Haiku 3.5.
• 4,096 tokens — Claude Opus 4.6, Claude Opus 4.5, Claude Haiku 4.5.

Read the Haiku line twice. Haiku 3.5 caches from 2,048 tokens and Haiku 4.5 requires 4,096 — the newer model has the higher floor. The Opus line runs the other way across recent releases, from 4,096 on 4.5 and 4.6 down to 2,048 on 4.7, 1,024 on 4.8 and 512 on Opus 5. A 3,000-token prefix therefore caches on Opus 5, Opus 4.8 and Sonnet 4.5, and silently does not on Opus 4.6 or Haiku 4.5. There is no rule of thumb here; look the number up for the exact model in your config, and look it up again when you change models. This is also the reason a prompt you wrote off as "too short to cache" on an older model is worth re-testing after an upgrade.

Verify From the Response, Not From the Code

Reading your prompt builder and concluding that the prefix is stable is exactly the reasoning that produced the bug. The response tells you the truth, in three fields:

• cache_creation_input_tokens — tokens written to the cache on this request, billed at the write multiplier.
• cache_read_input_tokens — tokens served from the cache on this request, billed at the read multiplier.
• input_tokens — the uncached remainder only.

That last definition is worth stating explicitly, because it is routinely misread as the prompt size. It is not. Total prompt size is the sum of all three. An agent run that reports 4,000 input_tokens after two hours of work is not a small prompt; it is a large one that mostly came from cache.

Two field patterns give you a diagnosis without any further instrumentation:

• Both write and read are zero, on every request. Nothing is being cached at all. The documented first suspect is that the prefix did not meet the model's minimum length.
• Write is non-zero every time and read stays zero. The cache is working; your prefix is not stable. Something before the breakpoint differs between requests — go back to the previous two sections.

If you use both lifetimes in one request, the write is broken out by TTL in a cache_creation object, as ephemeral_5m_input_tokens and ephemeral_1h_input_tokens; the two sum to cache_creation_input_tokens.

The cheapest possible probe is two identical requests in a row. Run it against the prompt you actually ship, not a reduced one, since the reduced version is often the one that stays under the minimum.

#!/usr/bin/env python3
"""Prompt cache probe. Exit 0 = the prefix was written, then read back."""
import os, sys, anthropic

client = anthropic.Anthropic()
MODEL = "claude-opus-5"

SYSTEM = [{
    "type": "text",
    "text": open(os.environ["SYSTEM_PROMPT_FILE"], encoding="utf-8").read(),
    "cache_control": {"type": "ephemeral"},
}]

def ask():
    response = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        system=SYSTEM,
        messages=[{"role": "user", "content": "Reply with the single word: ok."}],
    )
    usage = response.usage
    return (usage.input_tokens,
            usage.cache_creation_input_tokens,
            usage.cache_read_input_tokens)

first = ask()
second = ask()          # byte-identical prefix, byte-identical suffix
print("first  uncached=%d write=%d read=%d" % first)
print("second uncached=%d write=%d read=%d" % second)

problems = []
if first[1] == 0 and first[2] == 0:
    problems.append("nothing cached on the first call - prefix is likely below "
                    "this model's minimum, or cache_control never reached the request")
if second[2] == 0:
    problems.append("second call did not read the cache - the prefix is not "
                    "byte-identical between the two requests")

for problem in problems:
    print("FAIL", problem)
print(("FAIL" if problems else "PASS") + " - %d cache problem(s)" % len(problems))
sys.exit(1 if problems else 0)

When the probe passes and production still does not, two documented behaviours usually explain the gap. Concurrency is the first: an entry only becomes available once the first response begins, so a fan-out of ten parallel requests with the same prefix all miss. Send one, wait for it to start returning, then send the rest. Isolation is the second: caches are isolated between organizations, and on the Claude API, Claude Platform on AWS and Microsoft Foundry they are isolated per workspace as well, while Bedrock and Google Cloud isolate at organization level only. A hit rate that halves the week you split traffic across two workspaces is not a prompt bug.

What a Hit Is Actually Worth

Checked Aug 16 2026, Anthropic documents cache-write tokens at 1.25× the base input price for the 5-minute lifetime and 2× for the 1-hour lifetime, with cache reads at 0.1× in both cases. Those multipliers set the break-even directly. At the 5-minute lifetime a write plus one read comes to 1.35× against 2× for two uncached requests, so the first read already pays for the write. At the 1-hour lifetime the write alone is 2×, so you need two reads before you are ahead. The documentation also states that the cache is refreshed at no additional cost each time the cached content is used, which is why steady traffic keeps an entry alive without repeatedly re-writing it.

That gives the longer lifetime a narrow but real job: bursty traffic with gaps longer than five minutes, where the default entry would expire between bursts and be re-written at 1.25× each time. Compare the re-write cost against the one-off 2× before assuming the longer TTL is the safer default — on continuous traffic it is simply more expensive. Recalculate whenever the multipliers move; they are versioned, and this page states the date it checked them for that reason. For turning them into a monthly figure alongside model choice and routing, the AI cost modeling guide carries the arithmetic.

The last thing worth saying about hit rate is that it is not the goal. A prefix restructured purely to maximise the number is easy to build and frequently ships a worse prompt. The number you are steering is cached share of input tokens on the requests you actually serve, and the work that moves it is almost always the same: freeze what does not change, push what does behind the last breakpoint, and let the response tell you whether you were right.