Generate one operation key when the workflow decides what should happen, preserve it across every retry, and store a payload hash, state, lease, and result behind a uniqueness constraint. A duplicate with the same payload returns the stored result; the same key with different payload is rejected. For external effects, pass that key to a downstream system that also deduplicates, or reconcile its state before retrying. A local database row alone cannot make an email, charge, or deployment exactly once.
The Failure Is the Missing Reply
Imagine an agent calling issue_refund(invoice_842). The refund service commits the refund, but the network connection closes before the agent receives the response. From the agent's point of view, success and failure now look identical: both are a timeout. If the orchestration layer simply retries the tool call, the customer may receive a second refund.
This ambiguity is ordinary distributed-systems behavior, amplified by agents. Agent runtimes retry after timeouts, models repeat a call when they do not see the expected observation, queues deliver a message again, and operators replay failed runs. None of those behaviors is wrong. The mistake is exposing a non-idempotent business operation without a durable identity for the operation itself.
A run ID, trace ID, or tool-call ID usually changes when work is retried. Keep those values for observability, but do not use them to decide whether the business effect already happened. One intended refund can have five attempts and five traces while still having one operation key.
Define the Idempotency Boundary
HTTP defines an idempotent method by its intended effect: several identical requests should have the same intended server effect as one request. That is useful vocabulary, but an agent tool is a business command, not merely an HTTP verb. Putting POST /refund behind a retry loop does not make it idempotent. You must decide what counts as the same intended operation.
Good operation keys are created from a stable business decision. Examples include refund:invoice_842:approved_17, send_welcome:user_219:v1, or a random UUID stored on the workflow step when that step is first created. Two legitimate refunds for the same invoice need different approval identities; one refund retried five times needs the same identity every time.
Do not derive the key only by hashing tool arguments. Two people may intentionally receive the same template email, and the same customer may legitimately receive it twice months apart. The payload hash is still valuable, but for a different reason: it proves that a caller has not reused an existing operation key for changed arguments.
Persist a Small Operation State Machine
The first caller must claim the operation atomically. A primary key or unique constraint on operation_key is the simplest arbiter: one insert wins, and concurrent inserts take the conflict path. PostgreSQL's INSERT ... ON CONFLICT is one concrete implementation.
create table tool_operations (
operation_key text primary key,
payload_hash text not null,
status text not null check (
status in ('in_progress', 'completed', 'needs_reconciliation')
),
lease_until timestamptz not null,
result_json jsonb,
downstream_id text,
updated_at timestamptz not null default now()
);
A new key starts as in_progress with a short lease. A completed operation stores the response needed by future retries. A call that may have crossed an external boundary but did not finish local bookkeeping moves to needs_reconciliation. The important distinction is between "safe to try" and "unknown whether it happened". Collapsing both into failed is how duplicate effects are born.
- Operation key: stable across retries of one intended action.
- Payload hash: rejects the same key with changed arguments.
- Status and lease: coordinate concurrent attempts and abandoned workers.
- Stored result: lets duplicates receive the original answer.
- Downstream ID: gives reconciliation something concrete to query.
Make the Tool Contract Explicit
The model should not invent operation keys. The deterministic workflow layer creates the key before asking the model to call a tool, then injects it into the tool context. The model supplies business arguments; the runtime supplies identity and attempt metadata.
The following is contract-level pseudocode: claim must be implemented as an atomic storage operation, and reconcileBeforeRetry must query the real effect boundary rather than infer success from an exception.
async function invokeTool({ operationKey, attemptId, args }) {
const payloadHash = sha256(canonicalJson(args));
const claim = await operations.claim(operationKey, payloadHash);
if (claim.kind === 'payload_conflict') {
throw new Error('operation key reused with different arguments');
}
if (claim.kind === 'completed') return claim.result;
if (claim.kind === 'in_progress') throw new RetryLaterError();
if (claim.kind === 'needs_reconciliation') {
return reconcileBeforeRetry(claim);
}
return executeClaimedOperation({ operationKey, attemptId, args });
}
Make these outcomes part of the tool protocol. completed is a successful replay, not a fresh execution. in_progress tells the orchestrator to wait rather than spawn another worker. payload_conflict is a programming or workflow error and should never be retried. needs_reconciliation routes to deterministic code, not back to the model for a guess.
This also improves incident review. You can search one operation key and see the original business decision, every attempt ID, the downstream request identifier, and the response returned to each retry. Pair this with the correlation-ID practices in Zero Trust Architecture for Autonomous AI Agents without confusing trace identity with operation identity.
The External Side Effect Is the Hard Part
Reserving a database row and sending an email are two separate commits. The process can crash after the email provider accepts the message but before your database records completed. Expiring the lease and executing again would send the duplicate you were trying to prevent.
The strongest design passes the same operation key to a downstream API that provides its own idempotency contract. Stripe's API is a well-documented example: a client supplies an idempotency key, and a repeated matching request can return the first saved result rather than create another object. That is a provider-specific example, not a reason to copy its retention window into your system.
If the downstream system has no idempotency key, choose one of three honest alternatives. First, redesign the effect to converge, such as setting a deployment target to version v42 instead of issuing an unqualified "deploy again" command. Second, query the downstream system by a reference you control before retrying. Third, stop automatic retries and send the uncertain operation to human reconciliation. "At least once plus reconciliation" is safer than an unsupported claim of exactly-once delivery.
An outbox prevents a local transaction from losing work, but its worker can still crash after the external effect and before acknowledging the message. Keep the outbox; it solves a real problem. Then deduplicate at the effect boundary or reconcile the ambiguous outcome.
Handle Leases Without Replaying Blindly
A lease prevents one crashed worker from blocking an operation forever. It does not prove that the worker performed no effect. When a lease expires, inspect the operation's last durable milestone. If no downstream call began, another worker can claim it. If a downstream request ID was recorded, query that provider. If no query is possible, classify the operation as uncertain and require review.
AWS Powertools' idempotency utility demonstrates the value of persistent records, payload hashes, in-progress state, completed state, expiry, and concurrency handling. Use it as evidence that the pattern is established, not as a universal implementation. Your expiry must follow the business risk: a duplicate newsletter and a duplicate bank transfer do not deserve the same automatic replay policy.
Place the policy next to the tool definition. Document whether the downstream accepts a key, how long both sides retain it, how an uncertain result is reconciled, and which state allows an automatic retry. Otherwise the orchestrator will eventually treat every exception as retryable.
Four Tests That Prove the Contract
Run these tests below the model layer. A prompt-based evaluation can show that a model usually avoids duplicates; it cannot prove that two workers racing on the same key are serialized by storage.
Count provider invocations and business effects separately in the test double. A provider may receive two network requests yet create one resource because it deduplicates them. Your production metric should make the same distinction: attempts, duplicates served from cache, payload conflicts, operations awaiting reconciliation, and actual effects.
A Practical Release Gate
Before an agent receives a write-capable tool, answer five questions in the pull request: Who creates the operation key? Where is it persisted? What prevents changed arguments under the same key? Does the downstream deduplicate or support reconciliation? Which failure states may retry automatically?
Then run the four failure tests with network errors injected at both sides of the external call. If the test suite only covers a clean duplicate after a completed response, it proves the easiest case and misses the dangerous one. Combine this gate with rollback and approval gates for destructive actions, and with queueing and concurrency controls when several agents can reach the same resource.
Idempotency does not make an unsafe action safe. It makes retries of one approved action predictable. Least privilege limits what the agent may do, approval establishes that it should do it, and the operation record prevents uncertainty from turning one decision into two effects.
The definition of idempotent HTTP semantics comes from RFC 9110 section 9.2.2. Concrete implementation references are Stripe's idempotent request documentation, AWS Powertools for Lambda: Idempotency, and PostgreSQL's ON CONFLICT documentation. Sources checked Aug 10 2026.