Agent Trace Observability: What to Log So a Failed Run Is Debuggable

A Transcript Is Not a Trace

Most teams start with the thing that is easy to store: the message array. It is genuinely useful for reading what happened in the conversation, and genuinely useless for the questions you actually ask at 2 a.m. A transcript records the tool call the model requested. It does not record whether your executor ran it, how long it took, what the tool returned before your code reshaped the result, whether the call was retried after a timeout, or which of eleven near-identical calls consumed most of the budget.

This is a deliberately narrow implementation companion to the site's broader AI observability guide. That guide covers logging, metrics, dashboards and change markers. The acceptance artifact here is smaller: one failed agent run whose trace must answer four operational questions without a rerun.

The failure mode is specific and common: the transcript shows a tool_use block and then a plausible model response, so the run looks complete. What the transcript cannot show is that the tool returned a 502, your wrapper swallowed it into the string "no results", and the model reasoned confidently from an error message it had no way to identify as one. Nothing in the conversation log is wrong. The evidence you need was never written down.

One Span Per Operation, Named Consistently

OpenTelemetry publishes semantic conventions for generative AI, now maintained in their own semantic-conventions-genai repository. Adopting the naming costs nothing and buys you tooling that already understands your data. Two attributes are Required on a client span: gen_ai.operation.name and gen_ai.provider.name. The operation name is drawn from a defined set that covers the agent lifecycle, not just chat completions — invoke_agent, execute_tool, invoke_workflow and retrieval and planning operations among them. gen_ai.request.model is Conditionally Required when it is available, and the span name should be the operation name followed by the request model.

Four more attributes are Recommended and are the ones you will actually query: gen_ai.response.model, gen_ai.response.id, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. When an operation ends in an error, error.type becomes Conditionally Required. Record the response model separately from the request model even when you believe they are identical — an alias, a server-side fallback or a routing layer can serve a different model than you asked for, and a one-line difference in that field explains an otherwise inexplicable change in output.

Propagate One Trace Id Across Every Boundary

Spans are only useful if they join up. W3C Trace Context has been a W3C Recommendation since 23 November 2021, so this is settled ground: two headers, traceparent and tracestate. The trace id is 32 lowercase hex characters and the parent id is 16. An agent that calls your own services should pass traceparent outward, so the retrieval service's database query lands under the same trace as the model call that triggered it.

One detail catches people who hand-roll propagation: an all-zero value is explicitly invalid for both the trace id and the parent id, and a receiver that sees an invalid traceparent must ignore the header entirely. A middleware that defaults a missing id to zeros therefore does not produce a degraded trace — it produces a silently detached one, and the child spans surface as orphan roots with no obvious cause. Generate a real id or send no header at all.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^                                ^                ^
             |  trace-id (32 hex)                parent-id (16)   flags
             version

Record the Provider's Own Request Id

Your trace id is meaningful inside your system. It means nothing to the vendor whose API returned a 500. When a provider returns its own per-request identifier, capturing it is the difference between a support conversation that starts with an identifier and one that starts with an apology. Check each provider's response contract rather than inventing a field that may not exist.

With Anthropic's API the identifier arrives in the request-id response header and also appears as request_id in error response bodies. The Python and TypeScript SDKs expose it as a _request_id property on top-level response objects; the other SDKs expose it through their raw-response accessors. Store it alongside gen_ai.response.id rather than instead of it; they answer different questions, and the one you did not store is invariably the one you need.

Attribute Cost to the Run, Not to the Request

Token counts look like the simplest field on the span and hide the sharpest trap. In an Anthropic API response, usage.input_tokens is the uncached remainder only when prompt caching is in play — not the size of the prompt. The full prompt is the sum of three provider fields:

prompt_tokens = usage.input_tokens
              + usage.cache_creation_input_tokens
              + usage.cache_read_input_tokens

An agent that ran for an hour against a large cached system prompt can report a few thousand uncached input tokens per call and still be your most expensive workload. A dashboard summing only the provider's usage.input_tokens will show that agent as cheap and stay wrong indefinitely, because nothing about the number looks anomalous. Compute the total before mapping it to OpenTelemetry's gen_ai.usage.input_tokens, which is defined to include cached input tokens. Also retain the cache-read and cache-creation counts so you can derive hit rate; a hit rate that quietly drops to zero is one of the most useful cost alerts you can own, and it usually means something now varies inside your cached prefix.

Keep Content Capture Opt-In

The conventions classify gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions as Opt-In, with an explicit note that they are likely to contain sensitive information. Treat that default as the design, not an obstacle. Prompts carry whatever the user pasted, and tool results carry whatever your retrieval layer found — which is exactly the material you are least entitled to copy into a logging backend with a different retention policy and a wider access list than the source system.

A workable middle position: capture content on a sampled fraction of runs, on runs that ended in an error, and on runs a user explicitly flags — with field-level redaction applied before the span leaves the process, not in the collector. Redacting downstream means the raw value already crossed a process boundary and probably a network. If you need to group identical inputs, use a keyed HMAC computed inside the trust boundary instead of a plain prompt hash. The key prevents an observer from testing guessed prompts against the stored value; rotate it when long-term correlation is unnecessary.

Make Retries Visible, Not Invisible

SDK-level automatic retries are the single most common reason a trace disagrees with reality. If your instrumentation wraps the SDK call, three network attempts collapse into one span: you see one duration that is mysteriously three times the median and no indication why. If it wraps the transport instead, you get three spans that look like three separate operations unless something ties them together.

Give each attempt its own child span with an explicit attempt number, parented to the logical operation. Then a duplicate side effect becomes something you can see rather than deduce — a tool executed under attempt 1 and attempt 2 with the same arguments, which is precisely the failure that idempotent tool calls exist to prevent. The trace is how you find out whether your idempotency key is doing its job in production rather than only in the test suite.

The Four-Question Acceptance Test

Instrumentation is easy to add and hard to know you got right, because the day you need it is the day it is too late to improve it. Use a concrete drill instead of a checklist. Take a run that failed in staging, close the code, and answer these four questions using the trace alone:

If any answer requires opening the source or rerunning the agent, that is your next instrumentation task, and you now have a specific one rather than a vague intention to "add more logging". Run the drill again after the fix. Pair the result with rollback plans and approval gates so that what the trace reveals can also be undone, and with MCP authorization so that the calls you can now see are also the only ones permitted to happen.