Reliable AI API Calls: Timeouts, Retries, Rate Limits, and Fallbacks

Start With the Failure Contract

Reliability begins before the request is sent. Define what the feature promises when the model is slow, unavailable, rate-limited, or returns an unusable answer. A chat suggestion, a document export, and an agent that edits production data do not deserve the same deadline or recovery behavior.

Write down five limits for every AI operation:

• Latency objective: how long the user or caller should normally wait.
• Hard deadline: when the entire operation must stop, including retries and fallbacks.
• Attempt budget: the maximum number of provider calls and total tokens or money one user action may consume.
• Side-effect policy: whether tools, writes, emails, or charges may occur and how duplicates are prevented.
• Degraded outcome: queue for later, return a simpler answer, switch models, use cached data, or fail clearly.

Separate interactive work from durable work. An autocomplete request can be abandoned when the user types again. A ten-minute report should become a background job with a persistent state, a result the client can retrieve, and a cancellation path. Keeping a browser request open does not make long work reliable.

Give Every Operation One End-to-End Deadline

A stack of independent timeouts can exceed the user's patience. The load balancer may allow one minute, your application may permit two minutes per attempt, and a retry helper may run three attempts. Each component looks reasonable alone while the total workflow is unbounded.

Start with one monotonic deadline for the user-visible operation. Each queue wait, provider attempt, backoff delay, fallback, parser, and tool call spends from that same budget. Before starting the next step, calculate the remaining time and decline work that cannot finish within it.

type Deadline = { expiresAtMs: number };

function remainingMs(deadline: Deadline): number {
  return Math.max(0, deadline.expiresAtMs - performance.now());
}

async function runAttempt(deadline: Deadline) {
  const remaining = remainingMs(deadline);
  if (remaining < 2_000) throw new Error("deadline_exhausted");

  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), remaining);
  try {
    return await callModel({ signal: controller.signal });
  } finally {
    clearTimeout(timer);
  }
}

Track at least queue time, time to response headers, time to first token, streaming duration, and total duration. A timeout before the first token points to a different problem than a stream that stalls after producing useful output. Keep the provider request identifier when available so an incident can be correlated without logging private prompts.

Classify Errors Before Retrying

Put provider-specific exceptions behind a small application-owned taxonomy. Your feature should reason about meaning: overloaded, rate-limited, invalid input, unauthorized, unsafe output, or unknown outcome. This keeps retry and fallback policy stable when providers or SDKs change.

Exponential backoff spaces repeated attempts; jitter prevents many workers from waking simultaneously. If the service supplies an explicit retry delay, treat it as an input to policy rather than sleeping beyond your own deadline. Cap both the delay and total attempts.

for (let attempt = 1; attempt <= policy.maxAttempts; attempt++) {
  try {
    return await runAttempt(deadline);
  } catch (error) {
    const failure = classifyProviderError(error);

    if (!failure.retryable || attempt === policy.maxAttempts) throw failure;

    const delay = retryDelayWithJitter({
      attempt,
      retryAfterMs: failure.retryAfterMs,
      capMs: policy.maxBackoffMs
    });

    if (remainingMs(deadline) < delay + policy.minAttemptMs) throw failure;
    await sleep(delay);
  }
}

Never nest invisible retries. SDK, HTTP client, queue worker, application service, and outer job runner may each retry by default. Inventory every layer, assign ownership to one layer, and include the attempt number and parent operation ID in traces.

Control Rate, Concurrency, and Backpressure Separately

Requests per minute, tokens per minute, concurrent streams, and your own CPU or database capacity are different constraints. A limiter that counts only request starts can still overload the system when prompts vary from one hundred to one hundred thousand tokens.

• Rate limit starts: smooth bursts before they hit the provider.
• Reserve estimated tokens: account for input and a bounded output before dispatch, then reconcile actual usage.
• Cap in-flight work: use a semaphore per provider, model class, tenant, or workload as appropriate.
• Bound queues: reject, shed, or downgrade when waiting work exceeds an age or size limit.
• Preserve fairness: prevent one tenant or batch job from consuming all interactive capacity.

Backpressure means the admission layer tells callers that capacity is unavailable instead of accepting infinite work. For interactive features, return a clear retryable status or a degraded result. For background work, expose queued, running, retry-scheduled, completed, failed, and cancelled states with timestamps.

Autoscaling alone does not solve a provider quota. Adding workers can make a rate-limit incident worse by increasing contention and retries. Scale dispatch according to the narrowest downstream capacity, not the number of messages waiting.

Make Repeated Work Idempotent

A client can lose the response after the provider or tool completed the operation. From the client's perspective it failed; from the system's perspective the outcome is unknown. Blind repetition can send the same email twice, create two tickets, or charge twice.

Assign one operation ID to the user intent, not one ID per retry. Persist its state before starting durable work. Every retry and fallback reads the same record and either resumes, reconciles, or returns the stored result.

type AiOperation = {
  id: string;                  // idempotency key
  tenantId: string;
  kind: "draft" | "agent_run" | "report";
  inputHash: string;
  status: "pending" | "running" | "succeeded" | "failed";
  attemptCount: number;
  sideEffectState: "none" | "started" | "committed" | "unknown";
  resultRef: string | null;
};

Tools need their own idempotency boundary. Validate arguments, authorize them in application code, and store a tool-call key before execution. If a timeout occurs after dispatch, query the destination or operation ledger before running the tool again. A model-generated tool-call identifier is useful correlation data, but your application should own the durable key.

Do not cache only by raw prompt text. The effective input also includes model policy, system instructions, tool definitions, retrieved data versions, tenant scope, safety settings, and output schema. A cache key that ignores these fields can return another policy version's answer or cross a data boundary.

Use Circuit Breakers and Fallbacks Deliberately

When a dependency is consistently failing, repeated calls consume the remaining capacity without helping users. A circuit breaker opens after a measured failure threshold, rejects or reroutes new work for a cooling period, then admits a small number of probes before closing.

Scope breakers carefully. A failure in one model, region, or operation type should not disable every AI feature. Base the signal on recent eligible failures, not validation errors caused by your own bad request. Emit each state change as an operational event.

A fallback is a product decision, not just a model name:

• Smaller or alternate model: useful only if it supports the required context, tools, schema, policy, and latency.
• Reduced capability: summarize without tools, provide search results instead of a synthesized answer, or produce a draft requiring review.
• Cached or last known result: appropriate only when freshness and user scope are explicit.
• Queue for later: best for durable work that does not need an immediate answer.
• Clear failure: safer than pretending success when correctness or authorization cannot be preserved.

Practice the route before an outage. The model fallback drill covers the operational exercise, while the fallback runbook template structures the first response window.

Observe One Logical Operation Across Every Attempt

A dashboard that counts only successful provider calls hides the user experience. Trace one logical operation through admission, queueing, attempts, backoff, fallback, parsing, tool execution, and final delivery.

Useful fields include operation ID, tenant-safe workload class, route, model class, attempt number, error class, retry decision, backoff, queue age, deadline remaining, input and output token counts, time to first token, total latency, cancellation source, fallback reason, circuit state, and final outcome. Keep raw prompts and outputs out of general logs unless a separate, consented and access-controlled process requires them.

Measure ratios and end states:

• operations that succeed on the first attempt, after retry, or through fallback;
• deadline exhaustion, cancellation, rate-limit, overload, and invalid-output rates;
• queue age and in-flight concurrency by workload class;
• extra tokens, latency, and cost caused by retries;
• duplicate side-effect attempts and reconciliation outcomes;
• circuit openings, probe results, and time spent degraded.

Alert on user-visible outcomes and capacity trends, not every isolated transient error. The AI Observability guide covers trace structure, safe logging, metrics, and incident dashboards in more depth.

Test the Failure Path on Purpose

Unit tests can verify classification and delay calculations, but reliability also depends on interactions between the queue, provider client, database, stream, and tools. Add a controllable fake or proxy that can delay headers, stall a stream, emit partial data, return chosen statuses, and drop connections at exact points.

• Rate limit: verify the delay respects policy, the deadline remains bounded, and traffic does not synchronize into another burst.
• Provider error: verify the exact maximum attempts and the final fallback or degraded state.
• Slow first token: verify cancellation reaches the provider and frees the concurrency slot.
• Mid-stream failure: verify incomplete output is never presented as complete and unsafe continuation does not occur automatically.
• Duplicate delivery: submit the same operation and tool idempotency keys concurrently and prove one durable side effect.
• Unknown outcome: fail after tool dispatch and verify reconciliation occurs before any repeat.
• Queue overload: exceed the bound and verify fair admission, shedding, queue-age expiry, and recovery.
• Fallback: force the primary route open and run quality, schema, tool, safety, latency, and cost assertions on the alternate route.

it("does not repeat a committed tool side effect", async () => {
  provider.failStreamAfterToolResult();

  await runOperation({ operationId: "op_42" }).catch(() => {});
  await runOperation({ operationId: "op_42" });

  expect(tool.executionsFor("op_42")).toHaveLength(1);
  expect(await operationStore.get("op_42")).toMatchObject({
    sideEffectState: "committed"
  });
});

Run a low-volume production drill with explicit safeguards after the test environment passes. A fallback or circuit breaker that has never seen real credentials, quotas, routing, and telemetry is still a hypothesis.

Copy-Paste Prompt: Audit an AI Call Path

Give this prompt the provider wrapper, queue worker, tool dispatcher, API route, and relevant tests. Ask for the map before accepting code changes:

Audit the AI operation in [files or service] for production reliability.

First map the complete path from user intent to final result, including:
- admission, authentication, tenant scope, queueing, and concurrency limits
- every provider call, SDK retry, HTTP retry, job retry, and fallback
- parsing, structured-output validation, streaming, tools, and side effects
- operation state, idempotency keys, cancellation, and result delivery

For each step, document:
1. timeout and the shared end-to-end deadline
2. possible failure classes and whether each is retryable
3. maximum attempts, backoff, jitter, and retry-delay handling
4. what happens if the outcome is unknown after dispatch
5. rate, token, concurrency, queue-size, and cost limits
6. idempotency and reconciliation for every durable side effect
7. fallback compatibility: schema, tools, safety, quality, latency, and cost
8. logs, traces, metrics, alerts, and privacy boundaries

Find nested or unbounded retries, fresh timeouts per attempt, missing abort
propagation, unlimited queues, retry storms, duplicate side effects, silent
fallbacks, cross-tenant cache keys, and failures reported as success.

Then produce:
- a failure matrix with retry, fallback, and final user-visible behavior
- one end-to-end deadline and attempt budget per operation class
- the smallest implementation plan in reviewable stages
- rollback and feature-flag controls
- deterministic tests for rate limits, server errors, slow first token,
  mid-stream failure, cancellation, duplicate delivery, unknown tool outcome,
  queue overload, circuit opening, and fallback behavior

Do not implement changes until I approve the map and policy.

Related Guides