Reliable AI API Calls: Timeouts, Retries, Rate Limits, and Fallbacks

Start With the Failure Contract

Reliability begins before the request is sent. Define what the feature promises when the model is slow, unavailable, rate-limited, or returns an unusable answer. A chat suggestion, a document export, and an agent that edits production data do not deserve the same deadline or recovery behavior.

Write down five limits for every AI operation:

• Latency objective: how long the user or caller should normally wait.
• Hard deadline: when the entire operation must stop, including retries and fallbacks.
• Attempt budget: the maximum number of provider calls and total tokens or money one user action may consume.
• Side-effect policy: whether tools, writes, emails, or charges may occur and how duplicates are prevented.
• Degraded outcome: queue for later, return a simpler answer, switch models, use cached data, or fail clearly.

Separate interactive work from durable work. An autocomplete request can be abandoned when the user types again. A ten-minute report should become a background job with a persistent state, a result the client can retrieve, and a cancellation path. Keeping a browser request open does not make long work reliable.

Give Every Operation One End-to-End Deadline

A stack of independent timeouts can exceed the user's patience. The load balancer may allow one minute, your application may permit two minutes per attempt, and a retry helper may run three attempts. Each component looks reasonable alone while the total workflow is unbounded.

Start with one monotonic deadline for the user-visible operation. Each queue wait, provider attempt, backoff delay, fallback, parser, and tool call spends from that same budget. Before starting the next step, calculate the remaining time and decline work that cannot finish within it.

A deadline does two different things. It decides when your code stops waiting, and it asks the work to stop. Only the first is fully in your hands. An AbortSignal is a request: an adapter that ignores it can finish after the deadline, keep a provider slot busy and still be billed. The module below therefore never returns a result that arrives after the abort. It reports it through onLateSettlement, so cost and side-effect reconciliation can see it.

// reliability.ts: one deadline and one retry owner for an AI operation.
// Needs AbortController, AbortSignal, setTimeout and performance.now()
// (Node.js 20+, Deno, Bun and current browsers).

export type Clock = () => number;
export type Deadline = { readonly expiresAtMs: number; readonly now: Clock };

export type RetryPolicy = {
  maxAttempts: number;   // total attempts, including the first
  minAttemptMs: number;  // shortest attempt worth starting for this workload
  baseBackoffMs: number;
  maxBackoffMs: number;  // caps your own backoff, never a server's retry-after
};

export type FailureDecision =
  | { retry: false; reason: string }
  | { retry: true; reason: string; retryAfterMs?: number };

export type LateSettlement =
  | { status: "fulfilled"; value: unknown }
  | { status: "rejected"; reason: unknown };

export class DeadlineExceededError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "DeadlineExceededError";
  }
}

export class OperationCanceledError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "OperationCanceledError";
  }
}

export class RetryStoppedError extends Error {
  constructor(message: string, readonly attempts: number, options: { cause: unknown }) {
    super(message, options);
    this.name = "RetryStoppedError";
  }
}

// setTimeout fires immediately for delays above 2^31 - 1 ms (about 24.8 days).
const MAX_TIMER_MS = 2_147_483_647;

export function createDeadline(budgetMs: number, now: Clock = () => performance.now()): Deadline {
  if (!Number.isFinite(budgetMs) || budgetMs <= 0 || budgetMs > MAX_TIMER_MS) {
    throw new RangeError("budgetMs must be a positive number of milliseconds below 2^31");
  }
  return { expiresAtMs: now() + budgetMs, now };
}

export function remainingMs(deadline: Deadline): number {
  return Math.max(0, deadline.expiresAtMs - deadline.now());
}

// Stops waiting when the deadline or the caller's signal fires and aborts the
// signal handed to `work`. It cannot stop work that ignores that signal: a
// result arriving after the abort is never returned, only reported.
export function runAttempt<T>(
  deadline: Deadline,
  work: (signal: AbortSignal) => Promise<T>,
  options: {
    minAttemptMs: number;
    signal?: AbortSignal;
    onLateSettlement?: (late: LateSettlement) => void;
  },
): Promise<T> {
  if (options.signal?.aborted) {
    return Promise.reject(new OperationCanceledError("canceled before the attempt started"));
  }
  const remaining = remainingMs(deadline);
  if (remaining < options.minAttemptMs || remaining === 0) {
    return Promise.reject(new DeadlineExceededError("not enough time left for another attempt"));
  }

  const controller = new AbortController();
  return new Promise<T>((resolve, reject) => {
    let settled = false;
    const settle = (complete: () => void): boolean => {
      if (settled) return false;
      settled = true;
      clearTimeout(timer);
      options.signal?.removeEventListener("abort", onCallerAbort);
      complete();
      return true;
    };
    const stop = (error: Error) => settle(() => {
      controller.abort(error);
      reject(error);
    });
    const onCallerAbort = () => stop(new OperationCanceledError("canceled by the caller"));
    const timer = setTimeout(() => stop(new DeadlineExceededError("deadline reached during the attempt")), remaining);
    options.signal?.addEventListener("abort", onCallerAbort, { once: true });

    let pending: Promise<T>;
    try {
      pending = Promise.resolve(work(controller.signal));
    } catch (error) {
      pending = Promise.reject(error);
    }
    pending.then(
      (value) => {
        if (!settle(() => resolve(value))) options.onLateSettlement?.({ status: "fulfilled", value });
      },
      (reason: unknown) => {
        if (!settle(() => reject(reason))) options.onLateSettlement?.({ status: "rejected", reason });
      },
    );
  });
}

export function backoffDelayMs(
  attempt: number,
  policy: RetryPolicy,
  random: () => number,
  retryAfterMs?: number,
): number {
  const ceiling = Math.min(policy.maxBackoffMs, policy.baseBackoffMs * 2 ** (attempt - 1));
  const jittered = Math.floor(random() * ceiling); // "full jitter"
  // A server delay is a floor: retrying earlier fails again.
  return retryAfterMs === undefined ? jittered : Math.max(retryAfterMs, jittered);
}

function sleep(ms: number, signal?: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    if (signal?.aborted) return reject(new OperationCanceledError("canceled during backoff"));
    const onAbort = () => {
      clearTimeout(timer);
      reject(new OperationCanceledError("canceled during backoff"));
    };
    const timer = setTimeout(() => {
      signal?.removeEventListener("abort", onAbort);
      resolve();
    }, ms);
    signal?.addEventListener("abort", onAbort, { once: true });
  });
}

function assertPolicy(policy: RetryPolicy): void {
  const finite = [policy.minAttemptMs, policy.baseBackoffMs, policy.maxBackoffMs];
  if (!Number.isInteger(policy.maxAttempts) || policy.maxAttempts < 1
    || finite.some((value) => !Number.isFinite(value) || value < 0)
    || policy.maxBackoffMs < policy.baseBackoffMs) {
    throw new RangeError("invalid retry policy");
  }
}

export async function runWithRetries<T>(
  work: (signal: AbortSignal, attempt: number) => Promise<T>,
  options: {
    deadline: Deadline;
    policy: RetryPolicy;
    classify: (error: unknown) => FailureDecision;
    signal?: AbortSignal;
    random?: () => number;
    onRetry?: (event: { attempt: number; delayMs: number; reason: string }) => void;
    onLateSettlement?: (late: LateSettlement, attempt: number) => void;
  },
): Promise<T> {
  const { deadline, policy } = options;
  assertPolicy(policy);
  const random = options.random ?? Math.random;

  for (let attempt = 1; ; attempt++) {
    try {
      return await runAttempt(deadline, (signal) => work(signal, attempt), {
        minAttemptMs: policy.minAttemptMs,
        signal: options.signal,
        onLateSettlement: (late) => options.onLateSettlement?.(late, attempt),
      });
    } catch (error) {
      if (error instanceof DeadlineExceededError || error instanceof OperationCanceledError) throw error;

      const decision = options.classify(error);
      if (!decision.retry) {
        throw new RetryStoppedError(`not retryable: ${decision.reason}`, attempt, { cause: error });
      }
      if (attempt >= policy.maxAttempts) {
        throw new RetryStoppedError(`attempt limit reached: ${decision.reason}`, attempt, { cause: error });
      }
      const delayMs = backoffDelayMs(attempt, policy, random, decision.retryAfterMs);
      if (remainingMs(deadline) < delayMs + policy.minAttemptMs) {
        throw new RetryStoppedError(`next attempt would not fit the deadline: ${decision.reason}`, attempt, { cause: error });
      }
      options.onRetry?.({ attempt, delayMs, reason: decision.reason });
      await sleep(delayMs, options.signal);
    }
  }
}

minAttemptMs is a workload choice: the shortest attempt that can still produce a useful answer for this feature. Two seconds can suit a short interactive draft; a long report needs a durable job rather than a larger number. Budgets above 2³¹ − 1 ms are rejected because setTimeout fires immediately for longer delays. The module is tested against work that ignores the abort, work that never settles, caller cancellation, an already aborted signal, invalid budgets and policies, and listener and timer cleanup.

Track at least queue time, time to response headers, time to first token, streaming duration, and total duration. A timeout before the first token points to a different problem than a stream that stalls after producing useful output. Keep the provider request identifier when available so an incident can be correlated without logging private prompts.

Classify Errors Before Retrying

Put provider-specific exceptions behind a small application-owned taxonomy. Your feature should reason about meaning: overloaded, rate-limited, invalid input, unauthorized, unsafe output, or unknown outcome. This keeps retry and fallback policy stable when providers or SDKs change.

Exponential backoff spaces repeated attempts; jitter prevents many workers from waking simultaneously. A server's retry-after is a floor, not something to cap: Anthropic's rate-limit documentation says earlier retries will fail. If that delay does not fit the remaining deadline, stop and degrade instead of sleeping past it. A 429 is not always temporary either. When an organization reaches its tier's monthly spend cap, the API returns 429 without retry-after and with the error code enforced_spend_limit_reached; retrying cannot help.

A "Usually" in the table is still conditional. A lost connection does not prove the server did nothing, and RFC 9110 says a client should not automatically repeat a non-idempotent request unless it knows the request is safe to repeat or can tell that the original was never applied. A plain Messages call with no acting tools is safe to repeat at the cost of tokens; an agent turn that may have created a ticket is not. Streaming adds one more case: an error can arrive as an event after the API has already returned 200, so it never reaches status-based handling.

// classify-anthropic.ts: maps @anthropic-ai/sdk errors to retry decisions.
// Checked against @anthropic-ai/sdk 1.3.0 and the Claude API error and rate-limit docs.
import Anthropic from "@anthropic-ai/sdk";
import type { FailureDecision } from "./reliability";

const IMF_FIXDATE = /^[A-Z][a-z]{2}, \d{2} [A-Z][a-z]{2} \d{4} \d{2}:\d{2}:\d{2} GMT$/;

// RFC 9110 allows delay-seconds or an HTTP-date; the Claude API sends seconds.
export function parseRetryAfterMs(value: string | null, nowMs = Date.now()): number | undefined {
  if (value === null) return undefined;
  const text = value.trim();
  if (/^\d+$/.test(text)) return Number(text) * 1000;
  if (IMF_FIXDATE.test(text)) return Math.max(0, Date.parse(text) - nowMs);
  return undefined;
}

function errorCode(error: InstanceType<typeof Anthropic.APIError>): unknown {
  const body: unknown = error.error;
  if (typeof body !== "object" || body === null || !("error" in body)) return undefined;
  const inner: unknown = body.error;
  if (typeof inner !== "object" || inner === null || !("details" in inner)) return undefined;
  const details: unknown = inner.details;
  return typeof details === "object" && details !== null && "error_code" in details
    ? details.error_code
    : undefined;
}

// repeatIsSafe: true only when sending the same request again cannot repeat a
// side effect, for example a plain Messages call with no tools that act.
export function classifyAnthropicError(error: unknown, repeatIsSafe: boolean): FailureDecision {
  if (error instanceof Anthropic.APIUserAbortError) return { retry: false, reason: "aborted" };
  if (error instanceof Anthropic.APIConnectionError) {
    // Includes APIConnectionTimeoutError: the request may have been processed.
    return repeatIsSafe
      ? { retry: true, reason: "connection failed" }
      : { retry: false, reason: "outcome unknown: reconcile before repeating" };
  }
  if (!(error instanceof Anthropic.APIError) || error.status === undefined) {
    return { retry: false, reason: "unclassified error" };
  }

  const retryAfterMs = parseRetryAfterMs(error.headers?.get("retry-after") ?? null);
  const status = error.status;
  if (status === 429) {
    // A tier spend cap also returns 429, without retry-after, until access resumes.
    if (errorCode(error) === "enforced_spend_limit_reached") {
      return { retry: false, reason: "monthly spend cap reached" };
    }
    return { retry: true, reason: "rate limited", retryAfterMs };
  }
  if (status === 529) return { retry: true, reason: "overloaded", retryAfterMs };
  if (status === 408 || status >= 500) {
    return { retry: true, reason: `server or timeout error ${status}`, retryAfterMs };
  }
  if (status === 409) return { retry: false, reason: "conflict: resolve the resource state first" };
  return { retry: false, reason: `request rejected with ${status}` };
}

Never nest invisible retries. SDK, HTTP client, queue worker, application service, and outer job runner may each retry by default. The Claude TypeScript SDK, for example, retries connection errors, 408, 409, 429 and 5xx responses twice by default, honors retry-after, uses a ten-minute default timeout and retries timed-out requests too. One application attempt can then become three HTTP requests. When the application owns retries, create the client with maxRetries: 0 and pass the remaining budget as the request timeout:

// call-claude.ts: one Messages call under the application's deadline and retry policy.
import Anthropic from "@anthropic-ai/sdk";
import { classifyAnthropicError } from "./classify-anthropic";
import { createDeadline, remainingMs, runWithRetries, type RetryPolicy } from "./reliability";

// The SDK retries twice by default; this module owns retries instead.
const client = new Anthropic({ maxRetries: 0 });

// A workload choice for an interactive draft, not a provider recommendation.
const interactivePolicy: RetryPolicy = {
  maxAttempts: 3,
  minAttemptMs: 2_000,
  baseBackoffMs: 250,
  maxBackoffMs: 2_000,
};

export function draftReply(model: string, prompt: string, callerSignal: AbortSignal) {
  const deadline = createDeadline(20_000);
  return runWithRetries(
    (signal) =>
      client.messages.create(
        { model, max_tokens: 1_024, messages: [{ role: "user", content: prompt }] },
        { signal, timeout: Math.floor(remainingMs(deadline)) }, // the SDK requires an integer
      ),
    {
      deadline,
      policy: interactivePolicy,
      signal: callerSignal, // e.g. the incoming Request's signal: Stop or disconnect aborts upstream
      classify: (error) => classifyAnthropicError(error, true), // no tools act in this call
      onLateSettlement: (late, attempt) => {
        // Usage may still be billed; record it for cost reconciliation.
        console.warn("late provider settlement", { attempt, status: late.status });
      },
    },
  );
}

The model ID comes from configuration rather than code, so a model change does not require editing the retry path. Inventory every other retrying layer, assign ownership to one, and include the attempt number and parent operation ID in traces.

Control Rate, Concurrency, and Backpressure Separately

Requests per minute, tokens per minute, concurrent streams, and your own CPU or database capacity are different constraints. A limiter that counts only request starts can still overload the system when prompts vary from one hundred to one hundred thousand tokens.

• Rate limit starts: smooth bursts before they hit the provider.
• Reserve estimated tokens: account for input and a bounded output before dispatch, then reconcile actual usage.
• Cap in-flight work: use a semaphore per provider, model class, tenant, or workload as appropriate.
• Bound queues: reject, shed, or downgrade when waiting work exceeds an age or size limit.
• Preserve fairness: prevent one tenant or batch job from consuming all interactive capacity.

Backpressure means the admission layer tells callers that capacity is unavailable instead of accepting infinite work. For interactive features, return a clear retryable status or a degraded result. For background work, expose queued, running, retry-scheduled, completed, failed, and canceled states with timestamps.

Autoscaling alone does not solve a provider quota. Adding workers can make a rate-limit incident worse by increasing contention and retries. Scale dispatch according to the narrowest downstream capacity, not the number of messages waiting.

Make Repeated Work Idempotent

A client can lose the response after the provider or tool completed the operation. From the client's perspective it failed; from the system's perspective the outcome is unknown. Blind repetition can send the same email twice, create two tickets, or charge twice.

Assign one operation ID to the user intent, not one ID per retry. Persist its state before starting durable work. Every retry and fallback reads the same record and either resumes, reconciles, or returns the stored result.

// operation.ts: the durable record every retry and fallback reads first.
export type AiOperation = {
  id: string;                 // idempotency key for the user's intent, not per attempt
  tenantId: string;
  kind: "draft" | "agent_run" | "report";
  inputHash: string;          // same ID with a different input is a client error
  status: "pending" | "running" | "succeeded" | "failed";
  attemptCount: number;
  sideEffectState: "none" | "started" | "committed" | "unknown";
  toolCallKey: string | null; // application-owned key sent to the destination
  resultRef: string | null;
};

Store the tenant and an input hash with the key. The same operation ID with a different input, or from another tenant, is a client error, not a cache hit. Claim the record atomically, for example with a unique constraint or a conditional write, so two deliveries of the same intent cannot both start the side effect.

Tools need their own idempotency boundary. Validate arguments, authorize them in application code, and store a tool-call key before execution. If a timeout occurs after dispatch, query the destination or operation ledger before running the tool again. A model-generated tool-call identifier is useful correlation data, but your application should own the durable key.

Do not cache only by raw prompt text. The effective input also includes model policy, system instructions, tool definitions, retrieved data versions, tenant scope, safety settings, and output schema. A cache key that ignores these fields can return another policy version's answer or cross a data boundary.

Use Circuit Breakers and Fallbacks Deliberately

When a dependency is consistently failing, repeated calls consume the remaining capacity without helping users. A circuit breaker opens after a measured failure threshold, rejects or reroutes new work for a cooling period, then admits a small number of probes before closing.

Scope breakers carefully. A failure in one model, region, or operation type should not disable every AI feature. Base the signal on recent eligible failures, not validation errors caused by your own bad request. Emit each state change as an operational event.

A fallback is a product decision, not just a model name:

• Smaller or alternate model: useful only if it supports the required context, tools, schema, policy, and latency.
• Reduced capability: summarize without tools, provide search results instead of a synthesized answer, or produce a draft requiring review.
• Cached or last known result: appropriate only when freshness and user scope are explicit.
• Queue for later: best for durable work that does not need an immediate answer.
• Clear failure: safer than pretending success when correctness or authorization cannot be preserved.

Practice the route before an outage. The model fallback drill covers the operational exercise, while the fallback runbook template structures the first response window.

Observe One Logical Operation Across Every Attempt

A dashboard that counts only successful provider calls hides the user experience. Trace one logical operation through admission, queueing, attempts, backoff, fallback, parsing, tool execution, and final delivery.

Useful fields include operation ID, tenant-safe workload class, route, model class, attempt number, error class, retry decision, backoff, queue age, deadline remaining, input and output token counts, time to first token, total latency, cancellation source, fallback reason, circuit state, and final outcome. Keep raw prompts and outputs out of general logs unless a separate, consented and access-controlled process requires them.

Measure ratios and end states:

• operations that succeed on the first attempt, after retry, or through fallback;
• deadline exhaustion, cancellation, rate-limit, overload, and invalid-output rates;
• queue age and in-flight concurrency by workload class;
• extra tokens, latency, and cost caused by retries;
• duplicate side-effect attempts and reconciliation outcomes;
• circuit openings, probe results, and time spent degraded.

Alert on user-visible outcomes and capacity trends, not every isolated transient error. The AI Observability guide covers trace structure, safe logging, metrics, and incident dashboards in more depth.

Test the Failure Path on Purpose

Unit tests can verify classification and delay calculations, but reliability also depends on interactions between the queue, provider client, database, stream, and tools. Add a controllable fake or proxy that can delay headers, stall a stream, emit partial data, return chosen statuses, and drop connections at exact points.

• Rate limit: verify the delay respects policy, the deadline remains bounded, and traffic does not synchronize into another burst.
• Provider error: verify the exact maximum attempts and the final fallback or degraded state.
• Slow first token: verify cancellation reaches the provider and frees the concurrency slot.
• Mid-stream failure: verify incomplete output is never presented as complete and unsafe continuation does not occur automatically.
• Duplicate delivery: submit the same operation and tool idempotency keys concurrently and prove one durable side effect.
• Unknown outcome: fail after tool dispatch and verify reconciliation occurs before any repeat.
• Queue overload: exceed the bound and verify fair admission, shedding, queue-age expiry, and recovery.
• Fallback: force the primary route open and run quality, schema, tool, safety, latency, and cost assertions on the alternate route.

The template below runs against your own operation code through a small harness. It replaces an earlier version that swallowed the first failure with .catch(() => {}). That test also passed when the failure was injected before the tool ran, when the operation swallowed the stream error, when two concurrent deliveries both executed, when a reused ID carried a different input, and when an unknown tool outcome was repeated without a lookup. Each case now checks the state that proves the failure happened where it was meant to.

// operation-harness-contract.ts: what your operationHarness.ts must provide.
import type { AiOperation } from "./operation";

export type OperationRequest = { operationId: string; tenantId: string; prompt: string };

export type Harness = {
  runOperation(request: OperationRequest): Promise<{ resultRef: string }>;
  operationStore: { get(id: string): Promise<AiOperation | undefined> };
  provider: { failStreamAfterToolCommit(): void };
  tool: {
    executionsFor(operationId: string): readonly unknown[]; // commits at the destination
    lookupsFor(operationId: string): number;                // status queries before a repeat
    loseResponseAfterCommit(): void;
  };
};

// operation-idempotency.test.ts: run with Vitest against your own operationHarness.ts,
// which wraps the real runOperation, operation store and tool dispatcher. Only the
// provider and the tool's destination are fakes with failure switches.
import { beforeEach, describe, expect, it } from "vitest";
import type { Harness } from "./operation-harness-contract";
import { createHarness } from "./operationHarness";

const order = { tenantId: "tenant_a", prompt: "Open a support ticket for order 1042" };

describe("one durable side effect per operation", () => {
  let h: Harness;
  beforeEach(async () => {
    h = await createHarness(); // fresh store, ledger and fakes for every test
  });

  it("does not repeat a committed tool call after the stream fails", async () => {
    h.provider.failStreamAfterToolCommit();
    await expect(h.runOperation({ operationId: "op_42", ...order })).rejects.toThrow();

    // The failure must come after the side effect, or this test proves nothing.
    expect(h.tool.executionsFor("op_42")).toHaveLength(1);
    expect(await h.operationStore.get("op_42")).toMatchObject({
      status: "failed",
      sideEffectState: "committed",
    });

    const retried = await h.runOperation({ operationId: "op_42", ...order });
    expect(h.tool.executionsFor("op_42")).toHaveLength(1);
    expect(await h.operationStore.get("op_42")).toMatchObject({
      status: "succeeded",
      sideEffectState: "committed",
      resultRef: retried.resultRef,
    });
  });

  it("reconciles an unknown tool outcome before any repeat", async () => {
    h.tool.loseResponseAfterCommit(); // destination commits, dispatcher sees a timeout
    await expect(h.runOperation({ operationId: "op_43", ...order })).rejects.toThrow();
    expect(await h.operationStore.get("op_43")).toMatchObject({ sideEffectState: "unknown" });

    await h.runOperation({ operationId: "op_43", ...order });
    expect(h.tool.lookupsFor("op_43")).toBeGreaterThan(0);
    expect(h.tool.executionsFor("op_43")).toHaveLength(1);
  });

  it("runs one side effect when the same operation arrives twice at once", async () => {
    const runs = await Promise.allSettled([
      h.runOperation({ operationId: "op_44", ...order }),
      h.runOperation({ operationId: "op_44", ...order }),
    ]);
    expect(runs.filter((run) => run.status === "fulfilled").length).toBeGreaterThan(0);
    expect(h.tool.executionsFor("op_44")).toHaveLength(1);
  });

  it("refuses a reused operation ID with another input or tenant", async () => {
    await h.runOperation({ operationId: "op_45", ...order });
    await expect(h.runOperation({ operationId: "op_45", ...order, prompt: "Refund order 1042" })).rejects.toThrow();
    await expect(h.runOperation({ operationId: "op_45", ...order, tenantId: "tenant_b" })).rejects.toThrow();
    expect(h.tool.executionsFor("op_45")).toHaveLength(1);
  });
});

Implement operationHarness.ts so that it exports createHarness() and wraps your real runOperation, operation store and tool dispatcher. Only the provider and the tool's destination should be fakes with failure switches. An in-memory store proves the template, not your database: run the concurrency case against the real unique constraint or conditional write, and keep a separate integration test for a destination that may commit after your timeout.

Run a low-volume production drill with explicit safeguards after the test environment passes. A fallback or circuit breaker that has never seen real credentials, quotas, routing, and telemetry is still a hypothesis.

Copy-Paste Prompt: Audit an AI Call Path

Give this prompt the provider wrapper, queue worker, tool dispatcher, API route, and relevant tests. Ask for the map before accepting code changes:

Audit the AI operation in [files or service] for production reliability.

First map the complete path from user intent to final result, including:
- admission, authentication, tenant scope, queueing, and concurrency limits
- every provider call, SDK retry, HTTP retry, job retry, and fallback
- parsing, structured-output validation, streaming, tools, and side effects
- operation state, idempotency keys, cancellation, and result delivery

For each step, document:
1. timeout and the shared end-to-end deadline
2. possible failure classes and whether each is retryable
3. maximum attempts, backoff, jitter, and retry-delay handling
4. what happens if the outcome is unknown after dispatch
5. rate, token, concurrency, queue-size, and cost limits
6. idempotency and reconciliation for every durable side effect
7. fallback compatibility: schema, tools, safety, quality, latency, and cost
8. logs, traces, metrics, alerts, and privacy boundaries

Find nested or unbounded retries, fresh timeouts per attempt, missing abort
propagation, unlimited queues, retry storms, duplicate side effects, silent
fallbacks, cross-tenant cache keys, and failures reported as success.

Then produce:
- a failure matrix with retry, fallback, and final user-visible behavior
- one end-to-end deadline and attempt budget per operation class
- the smallest implementation plan in reviewable stages
- rollback and feature-flag controls
- deterministic tests for rate limits, server errors, slow first token,
  mid-stream failure, cancellation, duplicate delivery, unknown tool outcome,
  queue overload, circuit opening, and fallback behavior

Do not implement changes until I approve the map and policy.

Related Guides

Official Sources and Review Trigger

Timeouts, retries and fallbacks are only as good as the failures they distinguish. These sources define what a client actually receives, what the SDK already retries and which retries are safe.

• Anthropic: API errors — status codes and error types, SDK default retries, and errors that arrive after a 200 stream response, checked Oct 28 2026.
• Anthropic: rate limits — retry-after, rate-limit headers and the spend-cap 429 that must not be retried, checked Oct 28 2026.
• Anthropic: TypeScript SDK — maxRetries, timeouts and the error classes used above, checked Oct 28 2026 with @anthropic-ai/sdk 1.3.0.
• RFC 9110: idempotent methods and Retry-After — when a repeat is safe and the two forms of a retry delay, checked Oct 28 2026.

The modules were checked with Node.js 24.13.0, TypeScript 7.0.2 in strict mode, Vitest 5.0.5 and the real SDK error classes behind an offline fetch. No live provider, network failure, production database or billing system was exercised. Review this guide again by Nov 27 2026, or immediately if a provider adds an error class, changes how rate limits are signaled or changes the SDK's default retries. A raised limit is not a reason to revisit; a new failure mode is.

For a controlled capacity experiment, follow LLM load testing with k6: run your app against a model mock and record latency, errors and dropped iterations at fixed arrival rates.