
1. Decide what the shadow path may see
LLM shadow testing sends a copy of real production requests to a candidate model, keeps the candidate's answers away from users, and compares them with what production actually returned. One reason to test is a forced model change. Anthropic's model deprecations page, for example, lists claude-sonnet-4-5-20250929 as deprecated on September 30, 2026, with retirement on November 30, 2026 and claude-sonnet-5-5 as the recommended replacement. Those dates apply to Anthropic-operated platforms; Amazon Bedrock and Google-managed Vertex AI set their own schedules. Requests to a retired model fail, and the page promises at least 60 days' notice for publicly released models. It also advises testing your applications with newer models well before the retirement date. Shadowing is one way to do that testing on the traffic you really have.
Start with a written scope. List the routes you will mirror, the tenants that are excluded by contract or opt-out, and the request types that never qualify, such as requests that carry uploaded files you are not allowed to send to a second model. Offline suites still matter: the AI regression test plan template covers known cases, and shadowing adds the long tail you did not think to write down. The demonstration below shadows a support-triage route that reads an order and may issue a refund.
2. Mirror without awaiting the candidate
The Istio mirroring task defines traffic mirroring, also called shadowing, as sending a copy of live traffic to a mirrored service. Istio mirrors fire-and-forget and discards the responses, appends -shadow to the Host header, and can mirror a fraction of traffic with mirrorPercentage. A network mirror is a useful mental model; this guide mirrors in the application to capture model behavior. You need the candidate's answer for comparison, and a proxy cannot see the tool calls that happen inside your agent loop.
So mirror in the application, after the primary answer exists. In the code below, handle() awaits only the primary model, then schedules the candidate with asyncio.create_task and returns. The shadow task has its own timeout, catches its own exceptions, and writes a record when it finishes. Sampling hashes the request id, so the same request always gets the same decision and a replay produces the same sample. A cap on pending shadow tasks keeps a slow candidate from piling up work in your process; when the cap is full, the request is skipped and the skip is counted.
Not awaiting the candidate is not a latency guarantee. These tasks still share an event loop and process resources; blocking work can delay other requests. Use nonblocking clients, measure primary latency with and without shadowing, and isolate the shadow worker when shared resources affect production.
3. Deny tool side effects by default
This is the step that makes shadowing safe for agents. If the candidate decides to issue a refund, send an email, or write a row, nothing may happen. The production path wraps the real tool executor in RecordingTools, which runs each call and stores its result. The shadow path gets ShadowTools instead, which has no reference to the real executor at all. Effectful tools return blocked_in_shadow and are logged as intended actions. Read-only tools replay the primary's recorded result for the same name and arguments, so both models see the same order data and you do not double the load on your backends. A read the primary never made returns unavailable_in_shadow and is counted, because a candidate that asks for different data is itself a finding.
Allow replay only for an explicit list of read-only tools; every other name is blocked and counted as an intended effect. The production recorder keeps effectful calls in a list, so repeated calls with the same arguments remain distinct. Replaying reads also has a cost. The candidate cannot take a genuinely different investigative path, so the shadow underestimates behavior that depends on new reads. Report the unmatched-read count instead of hiding it.
4. Cap cost and keep personal data out of the records
Every shadowed request is a second model call that you pay for. The router reserves an estimated token count before it launches a shadow and skips with reason budget once the reservation would exceed the cap. The 500-token estimate and the caps in the demo are placeholders; set them from your own usage data and reconcile against billed usage, because a reservation is not an invoice. Combine the cap with a sample percentage so a traffic spike does not spend the whole budget in an hour.
For a like-for-like comparison, the shadow path uses the same authorized input as production. If the data cannot be shared with the candidate, exclude that request or compare both models offline on an approved sanitized input. Check that sending production data to the candidate is covered by the same data agreements as the primary, and exclude opted-out tenants before sampling. The demo recursively masks email addresses in stored request text and structured outputs, while computing output agreement before masking; a real system needs a fuller redactor, a retention limit on comparison records, and access control on the review queue.
5. Compare pairs, including actions
Each record holds the primary output, the candidate output, any error, and a comparison of side effects: the effectful calls production executed against the effectful calls the candidate attempted and the shadow blocked. That last field matters most. In the synthetic run, the candidate agrees with production on 351 of 353 categories, yet on three chargeback tickets it tries to issue a refund where production escalated. A text-only diff would rate that candidate excellent.
Exact match works for this demo because the output is a small category. For free text, compare structured fields you extract from both answers, or use a pairwise grader that sees both outputs without labels for which is which, and calibrate it against human review before you trust it.
6. Write the promotion rule before you look
CRITERIA fixes four conditions in advance: at least 150 samples, a Wilson lower bound on agreement of at least 0.90, an error rate of at most 1%, and zero effect mismatches. The thresholds are illustrative; the point is that they exist before the first report. A lower bound rather than the raw rate keeps a small sample from looking better than it is; the LLM eval confidence intervals article explains the interval. Changing the rule after seeing the numbers turns the test into a justification.
A HOLD is a result, not a failure of the process. Investigate the listed request ids, fix the prompt or tool descriptions for the candidate, and run a fresh shadow window. A PROMOTE moves you to a staged rollout with a tested way back, which the model fallback drill walks through. No shadow window proves the absence of regressions: it covers only the traffic you sampled, and replayed reads hide some behavior.
7. Run the side-effect check and the report
Save the block as shadow_demo.py and run python shadow_demo.py with Python 3.11 or later. It uses only the standard library, fake models, fake tools, and synthetic tickets, and makes no network calls. The six checks assert that the user answer returns while the shadow is still blocked, that only the primary's refund reaches the real executor, that candidate errors and timeouts never change the primary answer, that opt-outs and both caps skip with a reason, that sampling is stable and stored text is redacted, and that the report decides HOLD on 800 synthetic tickets.
import asyncio
import hashlib
import json
import math
import re
from collections import Counter
READ_ONLY = {"lookup_order"}
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
def redact(text):
if isinstance(text, dict):
return {redact(k): redact(v) for k, v in text.items()}
if isinstance(text, list):
return [redact(item) for item in text]
if not isinstance(text, str):
return text
return EMAIL.sub("<email>", text)
def sampled(request_id, percent):
digest = hashlib.sha256(request_id.encode()).digest()
return int.from_bytes(digest[:8], "big") % 10_000 < percent * 100
def key(name, args):
return name, json.dumps(args, sort_keys=True)
class RealTools:
def __init__(self):
self.executed = []
async def __call__(self, name, args):
self.executed.append(key(name, args))
return {"order": args["order"], "status": "ok"}
class RecordingTools:
"""Production path: execute for real and record every result."""
def __init__(self, real):
self.real = real
self.recorded = {}
self.effects = []
async def __call__(self, name, args):
result = await self.real(name, args)
self.recorded[key(name, args)] = result
if name not in READ_ONLY:
self.effects.append(key(name, args))
return result
class ShadowTools:
"""Shadow path: deny by default and never reach the real executor."""
def __init__(self, recorded):
self.recorded = recorded
self.blocked = []
self.unmatched = []
async def __call__(self, name, args):
call = key(name, args)
if name not in READ_ONLY:
self.blocked.append(call)
return {"status": "blocked_in_shadow"}
if call in self.recorded:
return self.recorded[call]
self.unmatched.append(call)
return {"status": "unavailable_in_shadow"}
class ShadowRouter:
def __init__(self, primary, candidate, tools, percent=50.0,
token_budget=100_000, estimate=500, max_pending=64,
timeout=30.0, opted_out=()):
self.primary, self.candidate, self.tools = primary, candidate, tools
self.percent, self.budget, self.estimate = percent, token_budget, estimate
self.max_pending, self.timeout = max_pending, timeout
self.opted_out = set(opted_out)
self.reserved = 0
self.pending = set()
self.records = []
self.skipped = Counter()
async def handle(self, request):
tools = RecordingTools(self.tools)
answer = await self.primary(request, tools)
self._maybe_shadow(request, answer, tools.recorded, tools.effects)
return answer
def _maybe_shadow(self, request, answer, recorded, primary_effects):
if request["tenant"] in self.opted_out:
reason = "opted_out"
elif not sampled(request["id"], self.percent):
reason = "not_sampled"
elif len(self.pending) >= self.max_pending:
reason = "queue_full"
elif self.reserved + self.estimate > self.budget:
reason = "budget"
else:
reason = None
if reason:
self.skipped[reason] += 1
return
self.reserved += self.estimate
task = asyncio.create_task(self._shadow(request, answer, recorded, primary_effects))
self.pending.add(task)
task.add_done_callback(self.pending.discard)
async def _shadow(self, request, answer, recorded, primary_effects):
tools = ShadowTools(recorded)
candidate, error = None, None
try:
async with asyncio.timeout(self.timeout):
candidate = await self.candidate(request, tools)
except Exception as exc: # Shadow failures stay in the shadow.
error = type(exc).__name__
self.records.append({
"id": request["id"],
"text": redact(request["text"]),
"primary": redact(answer),
"candidate": redact(candidate),
"outputs_match": error is None and candidate == answer,
"error": error,
"effects_match": sorted(primary_effects) == sorted(tools.blocked),
"unmatched_reads": len(tools.unmatched),
})
async def drain(self):
await asyncio.gather(*list(self.pending), return_exceptions=True)
def wilson_lower(hits, n, z=1.96):
if n == 0:
return 0.0
p = hits / n
center = p + z * z / (2 * n)
margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
return (center - margin) / (1 + z * z / n)
CRITERIA = {"min_samples": 150, "min_agreement_lower": 0.90,
"max_error_rate": 0.01, "max_effect_mismatches": 0}
def report(records, criteria=CRITERIA):
n = len(records)
agree = sum(r["outputs_match"] for r in records)
errors = sum(r["error"] is not None for r in records)
mismatches = [r["id"] for r in records if not r["effects_match"]]
lower = wilson_lower(agree, n)
reasons = []
if n < criteria["min_samples"]:
reasons.append(f"only {n} samples")
if lower < criteria["min_agreement_lower"]:
reasons.append(f"agreement lower bound {lower:.3f}")
if n and errors / n > criteria["max_error_rate"]:
reasons.append(f"error rate {errors / n:.3f}")
if len(mismatches) > criteria["max_effect_mismatches"]:
reasons.append(f"effect mismatches {mismatches}")
return {"samples": n, "agreement": agree, "lower_bound": round(lower, 3),
"errors": errors, "effect_mismatches": mismatches,
"decision": "HOLD" if reasons else "PROMOTE", "reasons": reasons}
def order_of(text):
return re.search(r"order (\w+)", text).group(1)
async def primary_model(request, tools):
text = request["text"]
if "refund" in text:
await tools("lookup_order", {"order": order_of(text)})
await tools("issue_refund", {"order": order_of(text)})
return {"category": "refund"}
if "chargeback" in text:
await tools("lookup_order", {"order": order_of(text)})
return {"category": "escalate"}
return {"category": "general"}
async def candidate_model(request, tools):
text = request["text"]
if "refund" in text or "chargeback" in text:
await tools("lookup_order", {"order": order_of(text)})
await tools("issue_refund", {"order": order_of(text)})
return {"category": "refund" if "refund" in text else "escalate"}
if request["id"].endswith("17"):
return {"category": "billing"} # Synthetic disagreement.
return {"category": "general"}
def fixture(i):
kinds = ["Please refund order A{i}", "Where is my invoice?",
"Reset my password", "Change my plan"]
text = kinds[i % 4].format(i=i)
if i % 97 == 5:
text = f"I filed a chargeback on order C{i}"
return {"id": f"req-{i:04d}", "tenant": "acme" if i % 9 else "globex",
"text": f"{text} (reply to user{i}@example.com)"}
class Gate:
def __init__(self, model):
self.model, self.release = model, asyncio.Event()
async def __call__(self, request, tools):
await self.release.wait()
return await self.model(request, tools)
async def fails(request, tools):
raise RuntimeError("candidate unavailable")
async def checks():
passed = 0
one = fixture(0) # A refund request with an email address.
always = dict(percent=100.0)
# 1. The user gets the primary answer while the shadow is still blocked.
gate = Gate(candidate_model)
router = ShadowRouter(primary_model, gate, RealTools(), **always)
assert await router.handle(one) == {"category": "refund"}
assert len(router.pending) == 1 and not router.records
gate.release.set()
await router.drain()
passed += 1
# 2. The candidate's refund is blocked; only the primary refund executed.
real = RealTools()
router = ShadowRouter(primary_model, candidate_model, real, **always)
await router.handle(one)
await router.drain()
assert [name for name, _ in real.executed] == ["lookup_order", "issue_refund"]
assert router.records[0]["effects_match"]
assert router.records[0]["unmatched_reads"] == 0
passed += 1
# 3. Candidate errors and timeouts never change the primary answer.
for candidate, timeout, error in [(fails, 30.0, "RuntimeError"),
(Gate(candidate_model), 0, "TimeoutError")]:
router = ShadowRouter(primary_model, candidate, RealTools(),
timeout=timeout, **always)
assert await router.handle(one) == {"category": "refund"}
await router.drain()
assert router.records[0]["error"] == error
passed += 1
# 4. Opt-outs, the token cap, and the pending cap skip with a reason.
router = ShadowRouter(primary_model, Gate(candidate_model), RealTools(),
token_budget=1_000, max_pending=3,
opted_out={"globex"}, **always)
for i in [9, 1, 2, 3]: # 9 is a globex request.
await router.handle(fixture(i))
assert router.skipped == Counter(opted_out=1, budget=1)
router.budget = 10_000
await router.handle(fixture(4)) # Third pending shadow.
await router.handle(fixture(6))
assert router.skipped["queue_full"] == 1
for task in router.pending:
task.cancel()
await router.drain()
passed += 1
# 5. Sampling is stable per request id, and stored text is redacted.
assert sampled("req-0042", 50.0) == sampled("req-0042", 50.0)
router = ShadowRouter(primary_model, candidate_model, RealTools(), **always)
await router.handle(one)
await router.drain()
assert "@" not in router.records[0]["text"]
passed += 1
# 6. A pre-registered rule decides; agreement alone is not enough.
router = ShadowRouter(primary_model, candidate_model, RealTools(),
percent=50.0, token_budget=200_000,
opted_out={"globex"})
for i in range(1, 801):
await router.handle(fixture(i))
await asyncio.sleep(0)
await router.drain()
result = report(router.records)
assert result["decision"] == "HOLD" and result["effect_mismatches"]
passed += 1
print(json.dumps(result, indent=1))
print(f"PASS {passed}/6 shadow checks")
if __name__ == "__main__":
asyncio.run(checks())
Expected output is the report followed by PASS 6/6 shadow checks. In our local run on November 18, 2026 it showed 353 samples, 351 agreements, a lower bound of 0.98, no errors, effect mismatches on req-0393, req-0490, and req-0781, and the decision HOLD with a single reason. Those numbers come from the synthetic fixture, not from any real model. To see a PROMOTE, remove the chargeback branch from candidate_model and run it again; to see the agreement rule fire, make the synthetic disagreement more frequent.
FAQ
Is shadow testing the same as an A/B test?
No. In an A/B test some users see the new model. In shadow testing users only see production answers, and the candidate's answers go to a comparison record.
Does shadow testing slow down production responses?
The router does not await the candidate before returning the primary answer. Shadow tasks still share process resources, so this is not a guarantee of unchanged latency. Measure the primary path under load and isolate shadow workers when needed.
How do I stop the candidate model from issuing refunds or sending email?
Give the shadow path a tool executor with no access to the real one. Effectful tools return a blocked status and are logged, and read-only tools replay the results production already recorded.
Can shadow testing prove a new model has no regressions?
No. It covers only the traffic you sampled, and replayed reads hide some behavior. Treat a passing window as evidence for a staged rollout, not as proof.
The Claude model deprecations page supplies the retirement dates, notice policy, and testing advice; the Istio mirroring task supplies the definition of mirroring and its fire-and-forget behavior. Both were checked November 18, 2026. The demonstration and its output are local test evidence with synthetic data.