AI Observability: Logging, Tracing, and Monitoring AI Features in Production

What Generic APM Misses

Standard application monitoring — request duration, status codes, error rates — treats an AI call like any other HTTP request. That's not wrong, but it's incomplete. An AI request has dimensions a normal API call doesn't:

• Model and version — the same endpoint can route to different models over time as you upgrade, fall back, or A/B test.
• Token counts — input, output, and cache tokens, which determine cost and correlate with latency.
• Prompt version — which system prompt or template produced this response, since prompts change more often than code deploys.
• Tool calls — how many tool invocations happened, which tools, and whether any failed or were retried.
• Non-determinism — the same input can produce a different output on the next call, so "it worked in staging" doesn't guarantee production behavior.

None of this shows up in a standard trace unless you put it there. AI Cost Modeling covers estimating spend before you ship; this guide covers watching it — and everything else — after you ship.

Structured Logging: What to Capture

Log AI calls as structured events, not free-text lines. A single JSON object per call, written to whatever log pipeline you already use, is enough to answer most incident questions:

{
  "event": "ai_call",
  "request_id": "req_8f2c1a",
  "timestamp": "2026-07-15T14:22:03Z",
  "model": "claude-sonnet-4-6",
  "prompt_version": "support-reply-v3",
  "input_tokens": 842,
  "output_tokens": 213,
  "cache_read_tokens": 620,
  "latency_ms": 1140,
  "tool_calls": ["lookup_order"],
  "status": "success",
  "user_id_hash": "u_9a31...",
  "stop_reason": "end_turn"
}

Note what's not in this event: the raw prompt text and the raw completion text. Logging full prompt/response content is the single most common way AI observability turns into a data-leak incident — customer PII, credentials pasted into a support ticket, or business logic embedded in a system prompt all end up sitting in your log aggregator with looser access controls than your database.

Correlating Logs with a Request ID

Generate one request ID at the edge of your system and thread it through every AI call, tool invocation, and retry that request triggers. Without it, a single user action that fans out into three AI calls and two tool calls produces five unrelated log lines with no way to reconstruct the sequence.

async function handleSupportReply(ticketId: string, requestId: string) {
  const context = await lookupTicket(ticketId, { requestId });
  const draft = await anthropic.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 1024,
    messages: [{ role: 'user', content: buildPrompt(context) }],
  });

  logAiCall({
    requestId,
    event: 'ai_call',
    model: draft.model,
    inputTokens: draft.usage.input_tokens,
    outputTokens: draft.usage.output_tokens,
    stopReason: draft.stop_reason,
  });

  return draft;
}

Tracing Multi-Step AI Pipelines

A single AI call is easy to log as one event. A multi-agent pipeline or a tool-calling loop is not — one user action can trigger an orchestrator call, three subagent calls, and five tool invocations, each with its own latency and failure mode. This is exactly what distributed tracing is for: one trace, many spans, each span showing where time and tokens actually went.

OpenTelemetry is the standard choice — vendor-neutral, with exporters for Datadog, Honeycomb, Jaeger, and most APM backends. Wrap each AI call and each tool call in its own span, nested under a parent span for the overall request:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('ai-pipeline');

async function runAgentTask(task: string, requestId: string) {
  return tracer.startActiveSpan('agent.orchestrate', async (span) => {
    span.setAttribute('request_id', requestId);

    try {
      const plan = await tracer.startActiveSpan('ai.call.plan', async (planSpan) => {
        const result = await anthropic.messages.create({
          model: 'claude-sonnet-4-6',
          max_tokens: 512,
          messages: [{ role: 'user', content: `Plan: ${task}` }],
        });
        planSpan.setAttribute('input_tokens', result.usage.input_tokens);
        planSpan.setAttribute('output_tokens', result.usage.output_tokens);
        planSpan.end();
        return result;
      });

      for (const step of parseSteps(plan)) {
        await tracer.startActiveSpan(`tool.${step.tool}`, async (toolSpan) => {
          toolSpan.setAttribute('tool_name', step.tool);
          await executeToolCall(step);
          toolSpan.end();
        });
      }

      span.end();
    } catch (err) {
      span.recordException(err as Error);
      span.end();
      throw err;
    }
  });
}

With this in place, a single trace ID shows the full shape of an agent run: how long planning took versus each tool call, which step failed, and how token usage split across the sub-calls. This is the difference between "the agent timed out" and "tool call #4 (inventory lookup) hung for 9 seconds while the model itself responded in 400ms." See Multi-Agent Systems and Tool Use for the orchestration patterns these spans are wrapping.

Metrics That Predict Incidents

Dashboards drown in metrics that don't change your response to an incident. For AI features, a small set actually matters:

Cost and quality metrics come from two different systems that should feed the same dashboard: AI Cost Modeling covers budgeting and spend alerts, and AI Evals in Production covers running eval suites against production traffic to catch quality regressions. Observability is the layer that correlates both against latency and errors in real time, instead of leaving them as separate weekly reports.

Building a Dashboard for Incident Response

A useful AI dashboard answers three questions in order, because that's the order an on-call engineer actually asks them during an incident:

1. Is it happening now, and how widely? Error rate and latency, sliced by model and prompt version, over the last hour.
2. What changed? A timeline of prompt deploys, model version changes, and config changes overlaid on the error/latency graphs — most AI incidents follow a prompt or model change, not a code change.
3. Which specific requests were affected? A way to jump from "errors spiked at 14:22" to the actual request IDs, traces, and (if sampled) prompts from that window.

Copy-Paste Prompt: Instrument an Existing AI Integration

Use this to add logging and tracing to an AI call that currently has none:

I have an AI integration in [file/function] that calls [model/provider]
with no observability. Add:

1. A structured log event per call: request ID, model, prompt version,
   input/output/cache token counts, latency, tool calls made, status,
   and stop reason. Do not log raw prompt or completion text by default.
2. An OpenTelemetry span wrapping the call, nested under the current
   request's parent span if one exists, with token counts and model
   as span attributes.
3. If this call is part of a multi-step pipeline (tool calls, subagent
   calls), wrap each step in its own child span so the full pipeline
   shows up as one trace.
4. A feature-flagged path to log the actual prompt/completion text for
   a sampled percentage of requests, using the redaction rules from
   [reference to your data-sanitization approach].

Show me the diff. Do not change the AI call's actual behavior or
retry logic — this is instrumentation only.

Related Guides