Resume an Agent Run After a Crash: Checkpoints, Durable State, and Safe Replay

What Dies, and What You Still Have

Start by naming the interruptions, because they have different survivors and teams usually design for only the first one.

• Process death. OOM kill, panic, container eviction. In-memory state is gone; anything already written to disk or a database survives.
• Deploy. A rolling restart during a long run. Same as process death, plus the new process may be running different code — a changed prompt, a renamed tool, an extra step in the plan.
• Timeout. A platform ceiling rather than a fault. Serverless request limits and job-runner wall clocks cut the run at a point unrelated to its logical structure.
• Budget or turn stop. A deliberate guard firing. Anthropic's Agent SDK documents this as a normal reason to resume: a run that ends with error_max_turns or error_max_budget_usd is resumed with a higher limit rather than restarted.
• Machine loss. The CI worker is gone entirely. This is the case that quietly breaks transcript-based recovery, because the SDK's session files are local to the machine that created them.

Only the last of those five destroys durable state, and only if you kept that state on the node. Everything else leaves your database untouched. The reason teams still restart from zero is not that the information was lost. It is that nothing wrote down which steps had finished.

A Transcript Is Not Run State

Every serious agent SDK will hand you conversation persistence, and it is genuinely useful. Anthropic's Agent SDK stores sessions as JSONL under ~/.claude/projects/<encoded-cwd>/, gives you resume to reopen a specific session id and fork_session to branch a copy without disturbing the original, and offers a session-store adapter that mirrors transcripts to your own backend so another host can pick them up. That covers the follow-up question and the process restart on the same box.

It does not cover the ledger question, and the same documentation says so — its third recommendation for resuming across hosts is to stop relying on session resume and instead capture the results you need, such as analysis output, decisions and file diffs, as application state you pass into a fresh session. That sentence is the whole design brief for this page. A transcript answers "what was said". A ledger answers "what is done".

The distinction matters most where the two disagree, and they disagree in exactly the situation you are recovering from. A run that crashed one second after a POST /charges returned 201 has a transcript ending in a tool call with no result. Replay that transcript and the model, reasonably, calls the tool again. Only a record written by your own code — before the call, and updated after it — can distinguish "never sent" from "sent, answer lost". This is also why a human-facing handover document such as the AI session resume packet is a different artifact: it is prose for a person restarting deliberately, not a machine-readable ledger for an orchestrator restarting automatically.

The Checkpoint Record: Six Fields That Earn Their Place

You do not need a workflow engine to start. You need one table and the discipline to write to it in the right order. Six fields cover the resumable case:

CREATE TABLE run_step (
  run_id        TEXT    NOT NULL,
  step_id       TEXT    NOT NULL,   -- stable across attempts: "plan.3.fetch_invoice"
  input_hash    TEXT    NOT NULL,   -- sha256 of the canonical inputs to this step
  status        TEXT    NOT NULL,   -- pending | started | succeeded | failed | abandoned
  result        JSONB,              -- the observation the next step will consume
  next_step_id  TEXT,               -- what the planner chose after seeing result
  attempt       INT     NOT NULL DEFAULT 1,
  effect_key    TEXT,               -- idempotency key, when this step has a side effect
  updated_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (run_id, step_id)
);

Three of those fields are the ones people leave out, and each omission has a specific failure attached.

• input_hash is what makes a resume safe rather than optimistic. On restart you recompute the hash of what this step's inputs are now. If it differs from the stored value, the retrieval set changed, an upstream step was edited, or the plan was regenerated — and the stored result is no longer an answer to the question you are asking. Without the hash, a resume silently splices a stale observation into a new plan.
• next_step_id records a decision, not just an output. In an agent loop the model chooses the next action. If you store only the tool result, the resumed run has to ask the model again, and it can pick differently — which loses the very property that made the first nineteen steps worth keeping.
• effect_key is the join to tool-level idempotency. The ledger knows a charge was attempted; the key is what lets you ask the payment provider whether it happened. Storing the provider's own returned id here as soon as you have it turns an ambiguous retry into a lookup.

If you would rather borrow a shape than invent one, LangGraph's checkpointer stores a StateSnapshot per super-step with the fields values, next, config (carrying thread_id, checkpoint_ns and checkpoint_id), metadata, created_at, parent_config and tasks. The mapping onto the table above is close enough to be instructive: values is your accumulated result set, next is next_step_id, and parent_config is the parent pointer that turns a flat table into a resumable chain.

Classify Each Step Before You Replay Anything

"Resume where it stopped" is not one operation. Every recorded step falls into one of three classes, and the resume procedure has to treat them differently.

• Pure. Parsing, formatting, arithmetic, deterministic routing. Safe to recompute from stored inputs. Storing the result is an optimisation; correctness does not depend on it.
• External read. A search, a database query, a file read, a model call. Re-executing is safe for the outside world but not free, and not guaranteed to return the same thing. Replay the stored result by default; re-read only when a freshness rule says the observation has expired.
• Effectful. A payment, an email, a deployment, a write to a shared system. Never re-executed on resume. Either the stored result is replayed, or — when the crash left the outcome ambiguous — the effect is reconciled through its effect_key before the run continues.

Model calls deserve a note of their own. They are external reads with a non-deterministic response, which means recomputing one during a resume does not restore the earlier state, it invents a new one. Treat a completion as a recorded observation exactly like an HTTP response: hash the inputs, store the output, and never re-derive it. Sampling temperature is beside the point here; even a greedy decode is not a guarantee you can build a resume on.

Two production systems make the same split in their own vocabulary and are worth reading as prior art. Temporal separates workflow code, which is replayed, from activities, which execute outside the replay path and are therefore allowed to be non-deterministic and retried on their own. LangGraph does it at a finer grain: when one node fails mid-super-step, it stores the pending writes of the other nodes that did complete, so a resume does not re-run them. If your own loop has no equivalent of that rule, a crash in a parallel fan-out will re-issue every sibling call.

Commit Order Decides What a Crash Can Break

There is no ordering of "do the thing" and "write that you did the thing" without a crash window. There is only a choice about which window you can clean up afterwards.

Write the intent first. Before an effectful step, commit a row with status='started' and the effect_key you are about to use, then perform the effect, then commit status='succeeded' with the result. A crash between the first and second commit leaves a started row, which is precisely the signal a resume needs: this step is ambiguous and must be reconciled, not retried and not skipped.

The reverse order — effect first, record after — leaves nothing behind at all, so the resumed run cannot tell an unstarted step from a completed one, and the only safe behaviour left is to stop and ask a person. That is a defensible design for two or three genuinely irreversible actions; see approval gates that keep production recoverable for where a human stop belongs. It is not a defensible design for thirty steps.

def run_step(conn, run_id, step, inputs):
    h = canonical_hash(inputs)
    row = load(conn, run_id, step.id)

    if row and row.status == "succeeded":
        if row.input_hash != h:
            raise StaleCheckpoint(step.id)       # inputs moved; do not reuse
        return row.result                        # replay, do not re-execute

    if row and row.status == "started":
        # Crash window: the effect may or may not have happened.
        settled = step.reconcile(row.effect_key)  # ask the downstream system
        if settled is None:
            raise NeedsReconciliation(step.id)    # stop the run, page a human
        commit(conn, run_id, step.id, "succeeded", settled, h)
        return settled

    key = row.effect_key if row else new_effect_key(run_id, step.id)
    commit(conn, run_id, step.id, "started", None, h, effect_key=key)
    result = step.execute(inputs, idempotency_key=key)
    commit(conn, run_id, step.id, "succeeded", result, h, effect_key=key)
    return result

The reconcile call is the part that cannot be generic, and it is the part most teams skip. For a payment provider it is a lookup by idempotency key. For an email service it may be a message-id search. For a system with no read path at all, reconciliation is impossible and the step has to be made safe some other way — which is the argument, again, for pushing deduplication down into the tool contract rather than up into the orchestrator.

Replay Puts a Determinism Constraint on Your Own Code

A resume assumes the resumed process walks the same path to the same place. Temporal states that requirement about as tightly as it can be stated: workflow code must make the same API calls in the same sequence given the same input, and a mismatch between the commands produced on replay and the recorded event history is a non-determinism error that stops the replay outright.

Agent loops break that rule casually, because the things that define the path get edited daily. So record what the path depended on:

• The model id and the prompt hash for every recorded model call. A prompt edit between crash and resume is a code change by another name.
• A hash of the tool schema set. A renamed or removed tool invalidates a stored plan that referenced it.
• A plan version. If the step list itself is generated, the resume has to know whether it is resuming into the same list.
• No wall clock and no unseeded randomness in the orchestration path. If a step's behaviour depends on now(), capture the timestamp into the checkpoint and read it back on replay, exactly as a durable-execution engine does.

Then decide the policy in advance, because this failure is silent otherwise. The two defensible answers are "refuse to resume across a plan-version change and start a clean run" and "resume, but re-plan from the last completed step while keeping the effect ledger". The indefensible one is to resume a stale plan into new code and hope the step ids still mean what they meant.

Resumed Runs Grow: Bound the History

Every resume appends. The ledger grows, the transcript grows, and the context fed to the model on the next turn grows with them. Durable-execution platforms hit this wall early enough that they publish the numbers, and those numbers are the cheapest available sanity check on your own design.

Temporal caps a workflow execution's event history at 51,200 events or 50 MB and issues a warning at 10,240 events or 10 MB, with Continue-As-New as the documented escape hatch: pass the state that still matters into a fresh execution with an empty history. AWS Step Functions attaches its limits to the recovery operation itself — an execution can be redriven only within a redrivable period of 14 days, only within a maximum open time of one year, and only if its event history count is below 24,999, because a redrive appends to the existing history rather than starting a new one.

Both sets of figures are as published on Aug 19 2026 and both are platform-specific, so do not copy the constants. Copy the two behaviours behind them. First, a resume budget: cap the number of resumes per run and fail loudly at the cap instead of looping forever on a step that will never succeed. Second, a compaction step at the boundary — carry forward the accumulated results and the ledger, not the raw transcript, which is the same trade the platforms make when they hand you a fresh history.

Step Functions also documents a trap worth stealing as a test case: a redrive normally reruns only the failed branches of a parallel state, but if that state failed with a data-limit error, the entire state is rerun including the branches that had succeeded. Partial-progress preservation has exceptions, and the exceptions cluster around the failures that look least like a clean crash. Assume yours has them until a test says otherwise. And if the work already sits behind a queue rather than an in-process loop, the retry and redelivery semantics of that queue are the layer to fix first — migrating an existing job queue without losing work covers that side.

The Kill Test Has to Pass Twice

A resume path that has never been interrupted on purpose does not work. This is the cheapest test in the whole design and almost nobody runs it, because it means killing your own process rather than asserting against a mock.

Two assertions carry the test, and both have to be present. Result equality alone passes happily while the customer is charged twice; effect-count equality alone passes for a run that quietly produced the wrong answer. Run the pair in CI against a real database, not an in-memory fake — the whole point is the durability boundary, and an in-memory checkpointer has none. LangGraph is explicit about that split in its own tooling: InMemorySaver is for experimentation, while SqliteSaver and PostgresSaver are the ones that survive the process.

When Not to Build This

The ledger costs one write per step, a schema to migrate, and a reconciliation path per effectful tool. Three situations do not repay that.

• Short, read-only runs. Five steps, no side effects, a one-minute wall clock: cheaper to re-run than to checkpoint. Measure before assuming, and if a restart costs less than the engineering, restart.
• Runs where a person is already the gate. If every consequential action waits for approval anyway, the operator is your checkpoint. Add the ledger when you remove the operator, not before.
• Work that genuinely belongs in a durable-execution engine. Long-running, multi-tenant, human-in-the-loop-for-days workflows are what Temporal, Step Functions and their peers exist for. Hand-rolling ten per cent of that feature set is a good way to own the hard ninety.

The version worth building is the small one: a step id, an input hash, a status, a result, a next pointer and an effect key, written in the right order and tested by killing your own process. That is about a day of work, it turns the most expensive failure mode of an autonomous run into an inconvenience, and it composes with the practices collected in the AI reliability guide. The engineering that follows is not clever. It is just written down.