Batch API vs Real-Time Calls: When Async Pays Off, and What Breaks When It Doesn't

What Each Provider Actually Offers

All provider details below come from the official documentation as it stood on 2026-10-18. Limits and statuses change, so re-check the linked pages before you size a job.

Note the different clocks on retention. Anthropic counts 29 days from created_at, not from when processing ended; OpenAI counts 30 days from completion. Download results into your own storage as part of the job, not when someone remembers.

The OpenAI guide lists seven batchable endpoints, including /v1/responses, /v1/chat/completions and /v1/embeddings, and caps batch creation at 2,000 batches per hour. The Claude page accepts almost any Messages request, but rejects stream: true, the fast-mode speed parameter and max_tokens: 0. It also warns that batches may go slightly over a Workspace spend limit, so a spend cap is not a hard stop for batch traffic.

A Decision Tree for Batch or Real Time

1. Is someone or something waiting for this answer? A user in a chat, a webhook that must reply, a pipeline step that blocks: keep it real time. Retries, deadlines and backoff for that path belong in Reliable AI API Calls.
2. Can the deadline absorb one window plus a rerun? Plan for 24 hours, not for the typical hour. Anthropic notes that heavy demand can slow processing and cause more requests to expire. If expired requests go into a second batch, the worst case is close to two windows. A nightly report due at 07:00 fits; a same-afternoon export does not.
3. Does any request need another request's output? Each dependent step becomes its own batch round with its own wait. A three-step chain can take up to three windows. Independent work such as evals, classification, embeddings and bulk summaries fits best.
4. Does the job fit the per-batch limits? Split above 50,000 requests on OpenAI or 100,000 on Claude, and on OpenAI split by model, because one input file can only target a single model.

Only when all four answers point to batch does the 50% discount become the deciding factor. How much it saves per month depends on your token volume; model that in AI Cost Modeling rather than guessing from the headline rate.

Same Jobs, Two Submission Paths

On Claude, the batch request reuses the exact params you would send synchronously. The only difference is the custom_id wrapper. Use the synchronous call as your dry run: Anthropic validates batch params asynchronously and reports validation errors only when the whole batch has ended.

import anthropic
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

client = anthropic.Anthropic()

def params_for(job):
    return MessageCreateParamsNonStreaming(
        model=CLAUDE_MODEL,
        max_tokens=1024,
        messages=[{"role": "user", "content": job["text"]}],
    )

# Real time: one call per job, answer now
for job in jobs:
    message = client.messages.create(**params_for(job))
    store(job["id"], message)

# Batch: same params, keyed by custom_id, answer within the 24-hour window
batch = client.messages.batches.create(
    requests=[Request(custom_id=job["id"], params=params_for(job)) for job in jobs]
)

A Claude custom_id must be 1 to 64 characters of letters, digits, hyphens and underscores. Use your own record key, such as ticket-1042, not a list index: results come back in any order on both platforms.

On OpenAI, the same jobs become lines in a .jsonl file. Each line carries custom_id, method, url and a body with the same parameters as the synchronous endpoint.

import json
from openai import OpenAI

client = OpenAI()

with open("batch_input.jsonl", "w") as f:
    for job in jobs:
        f.write(json.dumps({
            "custom_id": job["id"],
            "method": "POST",
            "url": "/v1/responses",
            "body": {"model": OPENAI_MODEL, "input": job["text"]},
        }) + "\n")

batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/responses",
    completion_window="24h",
)

Statuses You Have to Handle

OpenAI reports eight batch statuses: validating, failed, in_progress, finalizing, completed, expired, cancelling and cancelled. Here failed means the input file failed validation, not that individual requests failed, and cancelling can take up to 10 minutes. Claude uses a simpler processing_status: in_progress, canceling, then ended. Per-request outcomes live in the results, with four types: succeeded, errored, canceled and expired.

Expiry is where the two platforms look most alike. OpenAI cancels unfinished requests, puts completed responses in the output file, charges for the tokens of completed requests, and writes each unfinished one to the error file with code batch_expired. Anthropic marks unsent requests as expired and says it does not bill errored, canceled or expired results. In both cases an expired batch is a partial success, not a failure to throw away.

Reconcile by custom_id, Then Retry

The failure mode that costs money is silent loss: a job that reads the output file, stores what it finds, and does not notice that some IDs are missing. Build the set of expected IDs before you submit, and check every one after the batch ends.

# Claude: one result per request, four result types
retry, fix = [], []
for result in client.messages.batches.results(batch.id):
    outcome = result.result
    match outcome.type:
        case "succeeded":
            store(result.custom_id, outcome.message)
        case "errored":
            if outcome.error.error.type == "invalid_request_error":
                fix.append(result.custom_id)    # body must change first
            else:
                retry.append(result.custom_id)  # retry as-is
        case "expired" | "canceled":
            retry.append(result.custom_id)

# OpenAI: successes and failures live in two files
def read_lines(file_id):
    if file_id is None:
        return []
    return [json.loads(l) for l in client.files.content(file_id).text.splitlines() if l]

batch = client.batches.retrieve(batch.id)
for line in read_lines(batch.output_file_id):
    store(line["custom_id"], line["response"]["body"])
for line in read_lines(batch.error_file_id):
    if line["error"] and line["error"]["code"] == "batch_expired":
        retry.append(line["custom_id"])
    else:
        fix.append(line["custom_id"])

missing = expected_ids - stored_ids() - set(retry) - set(fix)
assert not missing, f"unaccounted requests: {sorted(missing)}"

Send retry to a new batch if the deadline allows, or to synchronous calls if it does not; a submitted Claude batch cannot be modified. The OpenAI guide documents only the batch_expired error line, so the sketch sends every other OpenAI error line to fix for inspection. Do not resend the fix list until a person or a validator has changed the request. On Claude, also treat a succeeded message with stop_reason: "pause_turn" as unfinished: with server tools, the batch loop can pause a turn that you must continue in a follow-up request.

Prompt Caching Inside a Batch

Anthropic says the prompt-caching and batch discounts can stack, but cache hits in a batch are best effort because requests run concurrently and in any order. Its page states that users typically see hit rates "ranging from 30% to 98%, depending on their traffic patterns". That range is the provider's statement, not a number to plan with. To push toward the high end, it recommends identical cache_control blocks in every request, a steady stream of requests so entries do not expire after their 5-minute lifetime, and considering the 1-hour cache duration because batches can run longer than 5 minutes.

OpenAI's batch guide says nothing about caching, so check its caching documentation separately before you count on cached input rates. Whichever provider you use, read the cache fields from the returned usage, as described in Prompt Cache Hit Rate, instead of assuming the synchronous hit rate carries over.

Run a Side-by-Side Test Before You Switch

The numbers that matter are yours, not a vendor's. This is a test for you to run; we have not run it and report no results.

1. Pick a sample of real requests from the job, large enough to include its long and odd inputs, with stable IDs.
2. Send it twice: once synchronously at your normal concurrency, once as a single batch with identical parameters.
3. Measure cost from the token usage each response returns, priced at the synchronous and batch rates on the provider page on the day you test, including cache-read tokens where reported.
4. Measure time as wall-clock from first submission to the last usable result, including your retry pass, not per-request latency.
5. Measure failure share as errored plus expired, divided by the total, and check that the reconciliation leaves no missing IDs.
6. Decide with thresholds you wrote down before the test, for example the latest acceptable finish time and the highest failure share you will tolerate.

Repeat the batch half on a different day before moving a production job: one quiet run says little about how the 24-hour window behaves under load.

Related Guides