ArticleReliability

LLM Retries: Why Backoff Needs a Retry Budget

Three attempts sounds bounded until every request gets three attempts. Put a ceiling on additional calls across operations, and decide which client, tenant, and fleet that ceiling actually covers.

Published: · Last reviewed:

Cyan paths converge on a dark gate with amber tokens while two other paths stop short.
A finite shared allowance gates additional attempts; this illustration is a metaphor for retry admission.
The decision

Assign one retry owner, cap attempts per operation, and require every additional provider call to acquire permission from a shared retry pool. Keep a deadline as a separate stop condition. A pool controls extra work; it does not admit original traffic or make repeated side effects safe.

Count calls at the dependency boundary

In this article, an attempt is one outbound provider call. The initial call is attempt one; two retries allow three total attempts. Write both numbers in your configuration review. A setting named retries is not interchangeable with one named attempts.

Consider a hypothetical application that makes at most three attempts, while an inner SDK makes at most three outbound attempts per application invocation. If every failure remains eligible, the operation can make 3 × 3 = 9 provider calls: one original and eight additional calls. With another independent three-attempt wrapper, the ceiling becomes 27. These are constructed settings, not claims about any SDK default. Deadlines and cancellation may stop the sequence earlier.

AWS's retry analysis describes this amplification and the overload risk. Inventory the application, SDK, HTTP adapter, queue redelivery, and fallback path. For this design, let the application own retries and configure inner layers for one outbound attempt. If an inner retry cannot be disabled or observed, an application-level counter cannot enforce the intended outbound-call ceiling.

Jitter changes timing; a budget changes admission

AWS explains that jitter spreads retries over time and describes local token buckets for limiting retry traffic. Those are different controls. If 1,000 independent operations can each make two retries, spacing their calls still leaves as many as 2,000 additional attempts eligible. A deadline may reduce that number, but jitter alone does not specify a shared allowance.

Choose a budget contract explicitly. One option is a fixed pool of 20 retry permits for a defined incident window, without refill. Another is a token bucket with capacity B and refill rate r permits per second. With atomic consumption, no resets, and one permit per outbound retry, the latter admits at most B + rT retries during any interval of T seconds. Rounding permits down only tightens that bound. Count original calls separately.

A success-earned budget is a third policy: successful original operations earn a fractional allowance, up to a cap. It ties retry volume to productive work but may need a deliberately small bootstrap allowance after a quiet period. State whether retry successes replenish it; otherwise a recovering request can silently finance more retries. These are application policy choices, not universal settings from AWS or Google.

Choose the scope before choosing the number

Share the pool across operations using the same dependency client and quota boundary. Creating a new pool inside each request gives every request its own allowance and defeats the aggregate limit. Creating several independent client objects can also multiply it. For a long-lived process, key the pool by the provider account and constrained route you actually intend to protect.

Now add replicas. With P independent processes, each holding B permits and refilling at r permits per second, the combined allowance is at most P(B + rT) for a fixed set of processes without resets. Ten replicas with 20 permits each start with 200 retry permits, not 20. Autoscaling, restarts, and rolling deployments can introduce fresh pools. A per-process setting is therefore a local bound, not a fleet-wide guarantee.

For a fleet ceiling, allocate shares whose sum respects the total, or consume permits through an atomic shared authority. Static shares can strand unused capacity; a shared authority introduces a dependency and an explicit behavior when it is unavailable. Choose whether retries stop during that failure. Do not quietly create full local pools as a fallback and keep claiming the original fleet ceiling.

Google's overload chapter discusses customer quotas and local throttling before a call reaches the backend. Its adaptive algorithm also covers original traffic; the retry-only pool here does not. For tenant fairness, combine an outer dependency ceiling with tenant sublimits or a scheduler. A single first-come pool lets a noisy tenant consume the permits before another tenant arrives. Independent tenant pools without an outer ceiling can exceed dependency capacity.

Call counts are deliberately coarse. The same Google chapter explains that requests can have different resource costs. For your LLM workload, consider separate workload classes or weighted permits for expensive generations. A retry-call ceiling does not by itself bound tokens, concurrency, latency, or money.

Reproduce a small overload window

The following Python 3 model starts 100 original operations, allows three total attempts each, and processes retries in synchronous rounds. Each round has capacity for either 40 successful calls or zero successful calls; every other admitted call fails transiently. All requests have equal cost. The shared pool starts with 20 retry permits and never refills during this window.

Save the code as retry_budget.py and run python retry_budget.py. It has no dependencies or network calls. The returned tuple is network attempts, successful operations, locally rejected retries, and operations that exhausted the attempt cap. A denied retry ends that operation; it is not a provider rejection.

def simulate(capacity, retry_tokens):
    originals, max_attempts = 100, 3
    pending = list(range(originals))
    per_operation = [0] * originals
    attempts = retries = successes = rejected = 0
    tokens = retry_tokens  # None means no shared retry budget.
    for round_index in range(max_attempts):
        admitted = []
        for operation in pending:
            if round_index > 0:
                if tokens is not None:
                    if tokens == 0:
                        rejected += 1  # Local denial, no provider call.
                        continue
                    tokens -= 1
                retries += 1
            admitted.append(operation)
            per_operation[operation] += 1
            attempts += 1
        # Deterministic service: first capacity calls succeed each round.
        winners = min(capacity, len(admitted))
        successes += winners
        pending = admitted[winners:]
    assert max(per_operation) <= max_attempts
    assert attempts == originals + retries
    assert successes + rejected + len(pending) == originals
    if retry_tokens is not None:
        assert retries <= retry_tokens
    return attempts, successes, rejected, len(pending)

cases = [
    (40, None, (180, 100, 0, 0)),
    (40, 20,   (120, 60, 40, 0)),
    (0, None,  (300, 0, 0, 100)),
    (0, 20,    (120, 0, 100, 0)),
]
for capacity, tokens, expected in cases:
    actual = simulate(capacity, tokens)
    assert actual == expected, (actual, expected)
    print(capacity, tokens, actual)
print("PASS: four deterministic overload scenarios")
Capacity per roundShared retry permitsNetwork attemptsSuccessful operationsLocally rejected retriesAttempt cap exhausted
40 callsUnrestricted18010000
40 calls2012060400
0 callsUnrestricted30000100
0 calls2012001000

With capacity 40, the pool trades 40 successful operations for 60 fewer calls in this finite window. That is a real cost, not evidence that a smaller allowance improves every outcome. With capacity zero, neither policy succeeds; the pool limits additional load to 20 calls instead of 200. The 100 local denials in that case include 80 operations denied their first retry and 20 denied their second.

The exact embedded code was executed locally and all four expected tuples and their accounting assertions passed. This model demonstrates bounds and rejected work. It does not simulate latency, jitter, continuous arrivals, provider billing, concurrent token acquisition, or recovery after the window. No provider or SDK integration was tested. Change arrival patterns and recovery timing before using a similar model to evaluate your production policy.

Keep deadlines and unsafe outcomes independent

A retry permit is necessary, not sufficient. Before dispatch, require a retryable failure, a safe operation, room under the attempt cap, and enough remaining end-to-end time for the required delay and useful work. Recheck after waiting and cancellation. Consume the permit atomically immediately before the outbound retry; avoid holding it through a long sleep. If rejected, return a declared degraded or failed outcome instead of starting an invisible fallback loop.

Keep one end-to-end deadline across waits, queues, and attempts rather than resetting a timeout for each retry. Use the implementation details in the AI API reliability guide. AWS also warns that a missing response does not establish that remote side effects failed. For write-capable workflows, use the separate idempotent tool-call design before deciding whether another attempt is safe.

In the release review, record the owner, maximum outbound attempts, pool key, initial permits, refill rule, replica accounting, and tenant policy. Measure originals, dispatched retries, budget denials, deadline denials, cap exhaustion, and successes after retry separately. A falling retry rate can mean fewer failures or more local denials; the outcome counters distinguish them.

FAQ

How many attempts do two retries allow?

Two retries allow three total attempts: one initial call and two additional calls. Nested layers can multiply their total-attempt limits.

Does jitter replace a retry budget?

No. Jitter spreads retry timing, while a shared budget decides how many additional calls may be dispatched across operations.

Is a per-process retry budget a fleet-wide limit?

No. Independent processes have independent allowances. A fleet ceiling requires explicit allocation or coordinated consumption, including new processes and restarts.

Does a retry budget make tool calls safe?

No. A permit limits extra traffic. Retrying a write-capable operation still requires an idempotency or reconciliation design and enough remaining deadline.

Sources

Primary sources checked 2026-11-19: AWS: Timeouts, retries, and backoff with jitter, for amplification, local token buckets, jitter, and ambiguous outcomes; Google SRE: Handling Overload, for local rejection, customer isolation, and resource-cost distinctions. All numeric settings and synthetic results in this article belong to the stated examples.

→
Back toHome