
1. Decide which reads may share an answer
LLM request coalescing lets overlapping requests wait for one in-flight operation. Your service owns that operation; the first HTTP client merely happens to register it. If that client disconnects, another client should still receive its answer. This guide builds a local Python demonstration with a controlled fake backend so you can check that ownership before attaching an SDK.
The Go singleflight documentation describes duplicate callers waiting for an existing execution and receiving its result. We apply that idea to a narrower contract: explicitly approved, read-only operations whose callers accept the same completed answer. A subsequent request starts a new operation after completion. For retained answers, TTL, and invalidation, use the separate AI response caching guide.
For example, two support workers may accept the same summary of the same authorized policy snapshot. Two evaluation trials intended to sample independent outputs must remain separate, even if their prompts and temperature match. A temperature of zero is not our permission to combine calls.
2. Make identity an authorization decision
The immutable Request below serves as both the registry key and backend input. It includes tenant, effective scope, authorization revision, pinned model, prompt revision, data revision, locale, question, and temperature. Add every other input your adapter uses, including output format, tool configuration, seed, and generation limits. The demonstration deliberately has no tools or extra generation parameters.
Build these values from trusted server-side state after checking access. Do not accept a client-supplied scope label as evidence of permission. If two users cannot safely read precisely the same snapshot, give them separate keys, potentially including principal identity. Revoking access requires a new authorization revision for new requests and a delivery-time permission check for existing waiters. The key alone does not revoke an answer already being computed.
A Python tuple-like dataclass avoids delimiter collisions from joining unescaped strings. It does not encrypt prompts or authorize requests. Keep the registry private, avoid logging raw keys, and return immutable answer strings as this example does. Sharing a mutable response object would let one waiter change what another sees.
3. Run the complete offline demonstration
Save the following block as coalescing_demo.py and run python coalescing_demo.py with Python 3.11 or later. It uses the default asyncio task factory, one process, and one event loop. No packages, credentials, network access, or model account are required. The expected output is PASS 10/10 lifecycle checks; an assertion or unexpected exception means the contract failed.
The Python task documentation documents waiting on existing tasks. Here, asyncio.wait keeps the wait separate from the task result: canceling a waiter leaves service work running, as the tests below verify. The registry keeps a strong reference to that task, and its completion callback retrieves an exception even when all waiters have left. Execution uses an empty contextvars context so it does not silently inherit the first waiter's request-local credentials; pass the approved snapshot explicitly to your adapter.
import asyncio
import contextvars
import math
from dataclasses import dataclass, replace
@dataclass(frozen=True)
class Request:
tenant: str
scope: str
auth_revision: str
model: str
prompt_revision: str
data_revision: str
locale: str
question: str
temperature: float = 0.0
def observe(task):
if not task.cancelled():
task.exception() # Retrieve failures even if every waiter left.
class Coalescer:
def __init__(self, backend, lifetime=30.0):
if not math.isfinite(lifetime) or lifetime < 0:
raise ValueError("lifetime must be finite and nonnegative")
self.backend = backend
self.lifetime = lifetime
self.pending = {}
self.loop = asyncio.get_running_loop()
if self.loop.get_task_factory() is not None:
raise ValueError("demo requires the default task factory")
async def get(self, request):
if asyncio.get_running_loop() is not self.loop:
raise RuntimeError("one event loop per coalescer")
task = self.pending.get(request)
if task is None:
# No await between lookup and insertion on this event loop.
task = asyncio.create_task(
self._run(request), context=contextvars.Context())
self.pending[request] = task
task.add_done_callback(observe)
await asyncio.wait({task})
return task.result()
async def _run(self, request):
try:
async with asyncio.timeout(self.lifetime):
return await self.backend(request)
finally:
# Cleanup runs before the task publishes its terminal result.
if self.pending.get(request) is asyncio.current_task():
del self.pending[request]
async def close(self):
tasks = list(self.pending.values())
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
class FakeModel:
def __init__(self):
self.calls = []
self.started = asyncio.Queue()
self.release = asyncio.Event()
self.fail = False
async def __call__(self, request):
self.calls.append(request)
self.started.put_nowait(request)
await self.release.wait()
if self.fail:
raise RuntimeError("model unavailable")
return f"{request.tenant}/{request.scope}/{request.auth_revision}"
async def joined(coalescer, request):
# The default factory runs the scheduled get() before this resumes.
task = asyncio.create_task(coalescer.get(request))
await asyncio.sleep(0)
return task
async def expect_error(task, kind):
try:
await task
except kind:
return
raise AssertionError(f"expected {kind.__name__}")
async def checks():
request = Request("acme", "support", "a1", "pinned-model",
"p1", "d1", "en-US", "Refund policy?")
passed = 0
# Identical simultaneous reads share one call; completed reads do not.
model = FakeModel()
co = Coalescer(model)
first = await joined(co, request)
await model.started.get()
second = await joined(co, request)
assert len(model.calls) == 1
model.release.set()
assert await first == await second == "acme/support/a1"
assert not co.pending
assert await co.get(request) == "acme/support/a1"
assert len(model.calls) == 2
passed += 2
# Tenant, effective scope, auth revision, and context stay separate.
variants = [request, replace(request, tenant="globex"),
replace(request, scope="legal"),
replace(request, auth_revision="a2"),
replace(request, data_revision="d2"),
replace(request, prompt_revision="p2"),
replace(request, model="other-model"),
replace(request, locale="sv-SE"),
replace(request, temperature=0.5)]
model = FakeModel()
co = Coalescer(model)
tasks = []
for item in variants:
tasks.append(await joined(co, item))
assert await model.started.get() == item
assert len(model.calls) == len(variants)
model.release.set()
results = await asyncio.gather(*tasks)
assert results[:4] == ["acme/support/a1", "globex/support/a1",
"acme/legal/a1", "acme/support/a2"]
assert not co.pending
passed += 1
# Either the first or a later waiter can leave without owning execution.
for canceled_index in (0, 1):
model = FakeModel()
co = Coalescer(model)
tasks = [await joined(co, request), await joined(co, request)]
await model.started.get()
tasks[canceled_index].cancel()
await expect_error(tasks[canceled_index], asyncio.CancelledError)
assert len(co.pending) == 1
model.release.set()
assert await tasks[1 - canceled_index] == "acme/support/a1"
assert len(model.calls) == 1 and not co.pending
passed += 1
# A waiter deadline does not become the shared operation deadline.
model = FakeModel()
co = Coalescer(model)
survivor = await joined(co, request)
await model.started.get()
impatient = await joined(co, request)
await expect_error(asyncio.wait_for(impatient, 0), TimeoutError)
assert len(co.pending) == 1
model.release.set()
assert await survivor == "acme/support/a1"
passed += 1
# All concurrent waiters see the error; the next read retries freshly.
model = FakeModel()
co = Coalescer(model)
model.fail = True
tasks = [await joined(co, request), await joined(co, request)]
await model.started.get()
model.release.set()
errors = await asyncio.gather(*tasks, return_exceptions=True)
assert all(isinstance(error, RuntimeError) for error in errors)
assert len(model.calls) == 1 and not co.pending
model.fail = False
assert await co.get(request) == "acme/support/a1"
assert len(model.calls) == 2
passed += 1
# Zero is a deterministic expired budget, not a wall-clock speed test.
model = FakeModel()
co = Coalescer(model, lifetime=0)
await expect_error(co.get(request), TimeoutError)
assert len(model.calls) == 1 and not co.pending
await expect_error(co.get(request), TimeoutError)
assert len(model.calls) == 2 and not co.pending
passed += 1
# With no waiters, the owner still retrieves failures and cleans up.
model = FakeModel()
co = Coalescer(model)
orphan = await joined(co, request)
await model.started.get()
owner = co.pending[request]
orphan.cancel()
await expect_error(orphan, asyncio.CancelledError)
model.fail = True
model.release.set()
await expect_error(asyncio.shield(owner), RuntimeError)
assert not co.pending
passed += 1
# Explicit service shutdown cancels cooperative work and empties registry.
model = FakeModel()
co = Coalescer(model)
waiter = await joined(co, request)
await model.started.get()
await co.close()
await expect_error(waiter, asyncio.CancelledError)
assert not co.pending
passed += 1
print(f"PASS {passed}/10 lifecycle checks")
if __name__ == "__main__":
asyncio.run(checks())
4. Read the assertions as a lifecycle contract
The first pair overlaps behind an Event gate and must produce one backend call. After both answers arrive, the registry must be empty and the next read must increment the call count to two. That second assertion distinguishes this helper from a result cache. The identity test starts nine different requests before releasing them and checks both call count and tenant/scope markers.
The cancellation loop removes the first waiter and then, in a separate run, the later waiter. In each case the surviving waiter still receives the answer from one backend call. The waiter-timeout test applies an immediately expired wait budget to one client while another remains attached. It verifies that client patience is separate from the lifetime of the operation.
The failure test delivers a backend error to both attached clients, checks cleanup, and then succeeds on a fresh call. No completed error is retained as a negative cache entry. An upstream adapter with its own cache can still replay cached failures: disable that policy for this experiment or test it separately. The orphan test removes the sole waiter before the backend fails, exercising the service's cleanup responsibility.
These are synchronization tests, not latency benchmarks. Event and Queue gates establish overlap; a zero-duration sleep yields one event-loop turn rather than waiting an assumed number of milliseconds. The expired-budget test uses zero lifetime to trigger cancellation at the backend's first suspension point. It checks the deadline path reproducibly without asserting that a busy machine meets a short wall-clock threshold.
5. Bound service work and document the limits
The default operation budget is 30 seconds, chosen for this demonstration rather than measured for your workload. Python's timeout documentation describes cancellation-based deadlines. This budget bounds the cooperative fake backend; it is not a hard execution kill. A backend that blocks the event loop or suppresses cancellation can exceed it and delay shutdown. Require cooperative cancellation, configure the adapter's own transport limits, and test those properties before production.
The operation continues if every waiter leaves, until it completes, fails, reaches its cooperative deadline, or the service closes it. This intentionally avoids letting client arrival order decide ownership. Use close() during an orderly shutdown after stopping admission of new requests. A production service also needs a cap on distinct pending keys and attached waiters, plus per-tenant admission limits: sharing one hot key does not bound a flood of different keys. Keep queue limits, retry ownership, and overall request budgets aligned with the AI API reliability guide.
This registry is local memory. Two workers can each start an operation for the same key, and restarting the process loses the registry. It provides no distributed exactly-once guarantee. Do not attach writes, payments, or effectful tool execution and infer deduplication safety from these tests. Streaming needs a separate subscriber policy for buffering, slow readers, disconnects, and partial results; this helper returns one completed string.
Before attaching a real backend, record started operations, joining waiters, waiter cancellations, terminal errors, deadline expirations, and pending-key count without exposing private inputs. Then repeat the tests through your authorization and SDK adapters. Passing this fake-backend suite verifies the printed control flow; it does not establish provider cancellation, billing savings, authorization integration, or answer quality.
FAQ
Is request coalescing the same as response caching?
No. This helper shares an overlapping operation and removes its entry when it finishes. A later request runs the backend again.
Does canceling the first waiter cancel the model call?
No. The service owns the shared task, so another waiter can still receive its answer. Service shutdown and the shared cooperative deadline can cancel that task.
Can requests from different tenants share a task?
In this demonstration, different tenants have different keys. Effective scope, authorization revision, and the other request inputs also distinguish operations.
Does this work across multiple Python workers?
Each worker has its own registry. This demonstration does not coordinate processes or provide a distributed exactly-once guarantee.
Go singleflight supplies the duplicate-suppression contract; Python asyncio tasks supplies task, waiting, context, and timeout semantics. Both were checked November 16, 2026. The demonstration and its expected results are local test evidence.