Caching AI Responses Without Serving the Wrong Answer

The Key Is the Whole Design

Every cache is a bet that two requests are equivalent. In HTTP that bet is written down: RFC 9111 makes the request method, the target URI and the Vary header part of selecting a stored response, so a server can declare which inputs matter. An AI feature has no such declaration. The inputs live in your prompt template, your retrieval layer, your tool definitions and your authorization middleware, and nothing forces you to write them down. So write them down as the key.

key = sha256(canonical([
  "v3",                    # key schema version: bump to abandon every entry at once
  model_id,                # the exact dated model id, never a floating alias
  sha256(system_prompt),
  sha256(tool_schema),     # canonicalised JSON: sorted keys, no insignificant whitespace
  corpus_version,          # version of the data this answer is derived from
  tenant_id,
  permission_scope,        # the caller's effective read scope, not their user id
  locale,                  # often lives outside the prompt text entirely
  sha256(user_question),
]))

Each component is there because of a specific failure. These are the ones worth knowing before you meet them:

• Model id, pinned to the exact version. Key on a floating alias and the day the alias moves, your evals run against the new model and pass while production keeps replaying answers the old model wrote. Nothing in the response distinguishes them, and the discrepancy survives every rollback of your own code.
• System prompt hash. You ship a fix to the output format. It works in staging, where the cache is cold, and does nothing in production for every question anyone has already asked. The bug report says "the fix didn't deploy", and the deploy was fine.
• Tool schema hash. Add a required parameter to a tool and cached turns keep emitting tool calls in the old argument shape. Your executor rejects them, so you get a wave of validation errors with no corresponding deploy of the executor.
• Version of the underlying data. Without it, a summary keeps citing a document that was corrected, superseded or deleted — and deletion is the case that turns a stale answer into a compliance problem.
• Tenant id. The obvious one, and the only one most teams add. It is necessary and nowhere near sufficient.
• The caller's effective read scope. Two users of the same tenant ask the same question; one may read the legal folder and one may not. Key on the scope rather than the user id: keying on the user id is technically safe and destroys the hit rate, because no two people share a key. Scope is the coarsest value that is still correct.
• Locale. The one people forget, because locale often is not in the prompt text at all — it arrives as a parameter to a formatting step or a per-request instruction appended downstream. If it is not in the key, the first language to ask a question wins it for everyone.

TTL Answers the Wrong Question

A TTL says how stale you are willing to be. For an answer derived from data that changes, the question is not how old the entry is — it is whether the source has moved since. Those two only coincide by luck.

The usual shape: an answer is generated from a policy document at 09:00 with a one-hour TTL. The document is corrected at 09:05. For fifty-five minutes the service confidently serves the old policy, and the team's instinct is to shorten the TTL. For any key asked often enough to matter, a five-minute TTL multiplies regeneration cost by about the same factor it divides the error window by, and still leaves a window. There is no TTL that makes this correct, because the clock is not the variable.

Put a version of the source in the key instead. When the document changes, its version changes, the key changes, and the old entry is simply never looked up again. Invalidation becomes a key change rather than a delete, which removes an entire class of race: a request that read the data before the write can only ever store its result under the superseded version, where no future reader will look. Delete-based invalidation has the opposite property — a slow reader can write pre-write data after your delete lands, and that entry is live.

TTL still has two honest jobs. It bounds storage for keys nobody will ask for again, and it is the only tool available for sources you cannot observe, such as a third-party API with no change feed. Set it for those reasons and say so in a comment, so the next person does not read it as a correctness control. If you want a middle setting for slow-changing data, the pattern already has a name and a specification: RFC 5861's stale-while-revalidate serves the known-stale entry immediately and refreshes behind it. That is a latency decision made deliberately, which is a different thing from not having noticed.

Errors and Empty Results Need Their Own Policy

If you never cache failures, a broken dependency gets hammered: every retry is a full miss, and on this kind of cache a miss costs a model call rather than a database read. If you cache failures like successes, a thirty-second outage becomes as many hours of confidently wrong answers as your positive lifetime allows. The resolution is old and well tested — RFC 2308 gave DNS negative answers their own separately bounded lifetime in 1998, for exactly this reason.

Apply it with three distinctions your code should make explicit:

• Transient failures — timeouts, rate limits, upstream 5xx. Cache for seconds, not minutes, purely to absorb a retry storm. Store the reason alongside the entry.
• Deterministic empties — the tenant genuinely has no document matching the question. This is data-derived, so it belongs under the normal version-keyed lifetime; it will be invalidated the moment a matching document is added, which is precisely when it should be.
• Authorization denials — never cache these under a key that omits the permission scope. Grant a user access and the cache keeps replaying the denial, which reads as a broken permissions system and is very hard to reproduce, because the person debugging it usually has different rights.

One more entry that looks like a success and is not: a response truncated by the token limit, or one that ended on any stop reason other than the normal one. It parses, it renders, and it will be replayed for as long as you let it. Refuse to store anything whose stop reason you did not expect.

One Hot Key, Many Simultaneous Misses

The moment a popular entry disappears, every in-flight request for it misses at once. A hundred concurrent callers become a hundred identical model calls, all computing the same answer, and the latency they were caching to avoid arrives for all of them simultaneously. Version-keyed invalidation makes this sharper than TTL does: a TTL expires one key, while a version bump retires every key derived from that source in the same instant, with no jitter to spread it out.

The countermeasure is small. On a miss, take a short-lived lock on the key; the winner generates and stores, and everyone else waits briefly for the result rather than calling the model. Two details decide whether it works:

• The lock's lifetime must exceed your p99 generation time, or the lock expires mid-generation and you get the stampede anyway, now with extra moving parts.
• The lock must be released on failure, not only on success. A lock leaked by an exception makes the key dark for its full lifetime, and a hot key going dark is worse than the stampede you were preventing.

For entries you can refresh before they are needed, probabilistic early recomputation is quieter still: as an entry ages, each request has a rising chance of regenerating it early, so refreshes spread out instead of arriving together. Either way, the waiters need a bounded wait and a fall-through path — the queueing and timeout behaviour is the same problem the AI API reliability guide treats in general, and a cache is just one more place it shows up.

Streaming: Cache the Result, Not the Stream

Streaming is a delivery mechanism, and delivery is the part that does not cache. The token timings, the chunk boundaries and the event log are properties of one generation; the assembled text is not. So accumulate on the server as you relay to the client, and write the cache entry only when the turn terminates normally.

Three consequences follow:

• A client disconnect must not make a partial buffer cacheable. If your handler stores whatever it accumulated when the connection dropped, a user closing a tab mid-answer pins a truncated response into the cache for everyone who asks that question next. The backend may continue generating after the client leaves, but it may write an entry only if the turn later terminates normally.
• Cache the completed turn, not the raw event sequence. A turn that contains tool calls interleaves model output with your own execution; replaying the event log replays the request for a tool call, not the answer that came after it.
• A hit is much faster than a miss, and that difference is visible. Front ends written against a progressive stream can misbehave when the whole answer lands in one frame — a typewriter animation that never runs, a scroll anchor that jumps. Re-chunk the stored text back into the same event shape on a hit. The client code stays identical, and you keep the option of measuring hits without the UI announcing them.

An Isolation Test You Can Actually Run

Reasoning about a key is not evidence about a key. This probe asks the same question as several identities, in both orders, and fails if any of them is shown a marker it is not entitled to see. Plant each marker inside a document only its owner can read, point ASK_URL at your own endpoint, and run it against staging.

#!/usr/bin/env python3
"""Cross-identity cache isolation probe. Exit 0 = isolated, 1 = leak."""
import json, os, sys, urllib.request

ENDPOINT = os.environ["ASK_URL"]
QUESTION = "Summarise our current refund policy."

# label -> (bearer token, markers this identity must never be shown)
IDENTITIES = {
    "acme/admin":   (os.environ["ACME_ADMIN"],   ["GLOBEX-CANARY"]),
    "acme/support": (os.environ["ACME_SUPPORT"], ["GLOBEX-CANARY", "ACME-LEGAL-CANARY"]),
    "globex/admin": (os.environ["GLOBEX_ADMIN"], ["ACME-CANARY", "ACME-LEGAL-CANARY"]),
}

def ask(token):
    request = urllib.request.Request(
        ENDPOINT,
        data=json.dumps({"question": QUESTION}).encode("utf-8"),
        headers={"Authorization": "Bearer " + token,
                 "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        return json.load(response)["answer"], response.headers.get("X-Cache", "unknown")

failures = []
order = list(IDENTITIES)

# Round 1 warms the cache. Round 2 runs in reverse order, so the test cannot
# pass merely because the same identity always asks first.
for round_number, sequence in ((1, order), (2, list(reversed(order)))):
    for label in sequence:
        token, forbidden = IDENTITIES[label]
        answer, cache = ask(token)
        for marker in forbidden:
            if marker in answer:
                failures.append("round %d: %s was served %s" % (round_number, label, marker))
        if round_number == 2 and cache.lower() != "hit":
            failures.append("round %d: %s did not hit the cache (%s)" % (round_number, label, cache))

for failure in failures:
    print("FAIL", failure)
print(("FAIL" if failures else "PASS") + " - %d isolation failure(s)" % len(failures))
sys.exit(1 if failures else 0)

The second round does double duty. It catches order dependence, and it asserts that the cache was actually involved: if a repeated identical request is not reported as a hit, the run is meaningless, because a probe that passes with caching switched off proves nothing at all. That check needs your endpoint to say whether it served a stored answer — an X-Cache header or an equivalent field. If yours does not, add it before you add the cache; the same signal is what makes the hit rate measurable later.

Isolation in the durable store beneath the cache is a related but separate problem — retention, deletion and the boundaries of what you keep per tenant are covered in the conversation memory guide. A cache entry is easier: it is always safe to throw away, which makes deleting it the cheap answer to almost any doubt.

Measure the Hit Rate Honestly, Then Decide

A single site-wide hit rate is the least useful number in this system. It is usually one hot key — a demo query, a health check, the example in your docs — sitting on top of a long tail where the real hit rate is a few percent. Four habits keep the number honest:

• Report per key namespace and per tenant. A tenant stuck at zero almost always means something tenant-specific leaked into the prompt: a timestamp, a display name, a session id. That is a key bug found by a dashboard rather than by a customer.
• Count only cacheable requests in the denominator. Otherwise the day you exclude streaming or a permission tier from caching, the hit rate improves and nothing got better.
• Report cost avoided, not hits. Hits multiplied by the miss-path cost of that key. A modest hit rate on long document summaries can be worth more than a high one on short answers, and only cost avoided tells you which.
• Alert on both directions. A rate that crashes means the key gained a varying component. A rate that jumps to nearly total on a key that should be diverse means it lost one — and that failure is silent, fast and popular, which is why it is usually found last.

The plumbing for all of this is ordinary instrumentation; if you have already followed the trace observability pattern, add the cache decision and the key namespace as span attributes and you are done.

Finally, the option that is easy to skip: not caching. Before building any of this, log a hash of the canonical key for a week with no cache in place and count exact repeats. If repeats sit at a few percent, a cache buys you a small saving and a permanent correctness liability. Skip it outright when the model call has side effects through tools, when a repeated question deserves a different answer because it is a follow-up, or when a stale answer would cause real harm rather than mild embarrassment. In that last case, cache only on a version-keyed entry and never on a clock.