AI Observability: Logging, Tracing, and Monitoring AI Features in Production

What Generic APM Misses

Standard application monitoring — request duration, status codes, error rates — treats an AI call like any other HTTP request. That's not wrong, but it's incomplete. An AI request has dimensions a normal API call doesn't:

• Model and version — the same endpoint can route to different models over time as you upgrade, fall back, or A/B test.
• Token counts — input, output, and cache tokens, which determine cost and correlate with latency.
• Prompt version — which system prompt or template produced this response, so you can correlate a prompt change with request behavior.
• Tool calls — how many tool invocations happened, which tools, and whether any failed or were retried.
• Non-determinism — the same input can produce a different output on the next call, so "it worked in staging" doesn't guarantee production behavior.

None of this shows up in a standard trace unless you put it there. AI Cost Modeling covers estimating spend before you ship; this guide covers watching it — and everything else — after you ship.

Structured Logging: What to Capture

Log AI calls as structured events, not free-text lines. Use an explicit field allowlist that covers successful and failed calls. This illustrative event records an Anthropic response; receiving a response does not establish that the application accepted it or that the answer was correct:

{
  "event": "ai_call",
  "request_id": "68fd66be-faa1-46e7-bbe2-544ee602a961",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "model": "claude-sonnet-5-5",
  "prompt_version": "v1",
  "uncached_input_tokens": 842,
  "cache_read_input_tokens": 620,
  "cache_write_input_tokens": 64,
  "input_tokens_total": 1526,
  "output_tokens": 213,
  "latency_ms": 1140,
  "status": "response_received",
  "stop_reason": "end_turn"
}

Note what's not in this event: the raw prompt text and the raw completion text. Logging full prompt/response content can expose sensitive data — customer PII, credentials pasted into a support ticket, or business logic embedded in a system prompt all end up sitting in your log aggregator with looser access controls than your database.

For Anthropic, usage.input_tokens counts uncached input. Add cache_read_input_tokens and cache_creation_input_tokens for OpenTelemetry's total gen_ai.usage.input_tokens. Map the API's cache-creation field to gen_ai.usage.cache_write.input_tokens. Cache counts are subsets of the total: do not add them a second time. For cost, apply the appropriate rates to the separate categories rather than multiplying the total by the uncached rate.

Correlating Logs with a Request ID

Generate one request ID at the edge of your system and thread it through every AI call, tool invocation, and retry that request triggers. Without it, a single user action that fans out into three AI calls and two tool calls produces five unrelated log lines with no way to reconstruct the sequence.

// Integration fragment: application adapters are supplied by your service.
// Import tracing.mjs once during startup; save the main implementation below as observability.mjs.
import { tracer } from './tracing.mjs';
import { createAiObserver } from './observability.mjs';

const observeModelCall = createAiObserver(tracer, event => logAiCall(event));

async function handleSupportReply(ticketId, requestId) {
  // requestId is a fresh UUID created at the request boundary, not a customer identifier.
  const context = await lookupTicket(ticketId, { requestId });
  return observeModelCall(() => anthropic.messages.create({
    model: 'claude-sonnet-5-5', max_tokens: 1024,
    messages: [{ role: 'user', content: buildPrompt(context) }]
  }), requestId);
}

Tracing Multi-Step AI Pipelines

A single AI call is easy to log as one event. A multi-agent pipeline or a tool-calling loop is not — one user action can trigger an orchestrator call, three subagent calls, and five tool invocations, each with its own latency and failure mode. This is exactly what distributed tracing is for: one trace, many spans, each span showing where time and tokens actually went.

OpenTelemetry supports tracing across vendors. Use a parent application span, a CLIENT span for a model operation, and INTERNAL spans for local tool calls. Register a tracer provider, async context manager, and propagation before instrumented work begins. The following example implements a subset of the Anthropic conventions and tool conventions pinned to commit 51a0f580f8a6. These GenAI fields are Development conventions, so review the pin when updating.

Save these modules together in a scratch ESM project. The documented baseline is Node.js 24, @opentelemetry/api@1.10.0, and @opentelemetry/sdk-trace@3.0.0, @opentelemetry/context-async-hooks@3.0.0, @opentelemetry/core@3.0.0. Install those exact package versions with npm. SDK 3.0 replaces the older trace-base/node/web packages; do not mix its API version with older SDK peer requirements.

tracing.mjs is a local demonstration with an in-memory exporter. It produces no dashboard and retains data only in the process. In a service, use one startup configuration with your approved exporter and retention/sampling policy, flush before shutdown, and propagate extracted/injected trace context at HTTP or messaging boundaries. Do not register a second provider inside a request handler.

// Node.js 24; initialize once, before loading instrumented application code.
import { context, propagation, trace } from '@opentelemetry/api';
import { TracerProvider, SimpleSpanProcessor, InMemorySpanExporter } from '@opentelemetry/sdk-trace';
import { AsyncLocalStorageContextManager } from '@opentelemetry/context-async-hooks';
import { W3CTraceContextPropagator } from '@opentelemetry/core';

export const exporter = new InMemorySpanExporter(); // Local demonstration, not durable storage
export const provider = new TracerProvider({
  spanProcessors: [new SimpleSpanProcessor({ exporter })]
});
trace.setGlobalTracerProvider(provider);
context.setGlobalContextManager(new AsyncLocalStorageContextManager().enable());
propagation.setGlobalPropagator(new W3CTraceContextPropagator());
export const tracer = trace.getTracer('ai-pipeline', '1.0.0');

observability.mjs targets non-streaming Anthropic Messages responses and takes application-owned adapters. createMessage wraps your configured Anthropic client and owns retries. validateAndAuthorizePlan parses typed text blocks, validates argument schemas and the caller's permissions, and returns at most ten allowed steps; it must not execute model text. executeTool dispatches those validated steps. The fixed registry here is lookup_order and search_policy. These adapters and the business service are not supplied by OpenTelemetry. Other providers, streaming, and server-side tool iterations require their own mappings. Avoid duplicating spans if automatic instrumentation already covers the same call.

createAiObserver returns the original response or rethrows the original error. It does not retry or classify a token-limit stop as a transport failure. The separate runPipeline example has an explicit application policy: accept only end_turn before parsing or executing a plan. Keep your own application's acceptance policy when adding instrumentation. Raw task text, model output, tool arguments/results, exception messages, and stack traces are absent from the telemetry allowlist.

import { SpanKind, SpanStatusCode } from '@opentelemetry/api';
import { randomUUID } from 'node:crypto';

const MODEL = 'claude-sonnet-5-5';
const PROMPT = 'support-plan';
const VERSION = 'v1';
const STOPS = new Set(['end_turn', 'max_tokens', 'stop_sequence', 'tool_use',
  'pause_turn', 'refusal', 'model_context_window_exceeded']);
const TOOLS = new Set(['lookup_order', 'search_policy']);
const count = value => Number.isSafeInteger(value) && value >= 0 ? value : undefined;

function errorType(error, fallback) {
  // Bounded status code or application-owned label; no message, stack, or arbitrary name.
  try {
    if (Number.isInteger(error?.status) && error.status >= 400 && error.status <= 599) return String(error.status);
  } catch { /* Some thrown values are not ordinary Error objects. */ }
  return fallback;
}
function failed(span, type) {
  span.setAttribute('error.type', type);
  span.setStatus({ code: SpanStatusCode.ERROR });
}
function requestId(value) {
  if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value)) {
    throw new Error('Expected an application-generated UUID');
  }
  return value;
}

export function createAiObserver(tracer, emitLog) {
  // call() owns provider invocation and retries. Return its result or error unchanged.
  return async function observeModelCall(call, id) {
    requestId(id);
    return tracer.startActiveSpan(`chat ${MODEL}`, {
      kind: SpanKind.CLIENT,
      attributes: {
        'app.request_id': id,
        'gen_ai.operation.name': 'chat',
        'gen_ai.provider.name': 'anthropic',
        'gen_ai.request.model': MODEL,
        'gen_ai.prompt.name': PROMPT,
        'gen_ai.prompt.version': VERSION
      }
    }, async span => {
      const start = performance.now();
      const event = { event: 'ai_call', request_id: id, trace_id: span.spanContext().traceId,
        model: MODEL, prompt_version: VERSION };
      try {
        const result = await call();
        const usage = result?.usage ?? {};
        const uncached = count(usage.input_tokens);
        const read = count(usage.cache_read_input_tokens ?? 0);
        // Anthropic's API field is cache_creation_input_tokens; OTel calls it cache_write.
        const write = count(usage.cache_creation_input_tokens ?? 0);
        const output = count(usage.output_tokens);
        const total = [uncached, read, write].every(x => x !== undefined)
          ? count(uncached + read + write) : undefined;
        for (const [key, value] of Object.entries({
          'gen_ai.usage.input_tokens': total, 'gen_ai.usage.output_tokens': output,
          'gen_ai.usage.cache_read.input_tokens': read, 'gen_ai.usage.cache_write.input_tokens': write
        })) if (value !== undefined) span.setAttribute(key, value);
        const stop = STOPS.has(result?.stop_reason) ? result.stop_reason : 'other';
        span.setAttribute('gen_ai.response.finish_reasons', [stop]);
        if (/^claude-sonnet-5-5(?:-\d{8})?$/.test(result?.model ?? '')) {
          span.setAttribute('gen_ai.response.model', result.model);
        }
        Object.assign(event, { status: 'response_received', stop_reason: stop,
          uncached_input_tokens: uncached, cache_read_input_tokens: read,
          cache_write_input_tokens: write, input_tokens_total: total, output_tokens: output });
        return result;
      } catch (error) {
        const type = errorType(error, 'model_request_failed');
        failed(span, type);
        Object.assign(event, { status: 'transport_error', error_type: type });
        throw error;
      } finally {
        // emitLog receives only this allowlist. Log sink failure must not change the API result.
        try {
          void Promise.resolve(emitLog({ ...event, latency_ms: performance.now() - start }))
            .catch(() => {}); // Delivery is not awaited; monitor/flush the sink separately.
        } catch { /* Monitor sink health separately. */ }
        span.end();
      }
    });
  };
}

export async function runPipeline(task, deps, tracer, emitLog, id = randomUUID()) {
  requestId(id);
  const observe = createAiObserver(tracer, emitLog);
  return tracer.startActiveSpan('support.pipeline', {
    kind: SpanKind.INTERNAL, attributes: { 'app.request_id': id }
  }, async span => {
    let stage = 'model';
    try {
      const plan = await observe(() => deps.createMessage({
        model: MODEL, max_tokens: 1024,
        messages: [{ role: 'user', content: `Plan: ${task}` }]
      }), id);
      stage = 'plan';
      // Explicit application policy, not an automatic side effect of instrumentation.
      if (plan?.stop_reason !== 'end_turn') throw new Error('Plan incomplete');
      const steps = await deps.validateAndAuthorizePlan(plan);
      if (!Array.isArray(steps) || steps.length > 10 || steps.some(s => !TOOLS.has(s?.tool))) {
        throw new Error('Invalid plan');
      }
      stage = 'tool';
      const results = [];
      for (const step of steps) {
        results.push(await tracer.startActiveSpan(`execute_tool ${step.tool}`, {
          kind: SpanKind.INTERNAL,
          attributes: { 'gen_ai.operation.name': 'execute_tool',
            'gen_ai.tool.name': step.tool, 'gen_ai.tool.type': 'function' }
        }, async toolSpan => {
          try { return await deps.executeTool(step, { requestId: id }); }
          catch (error) { failed(toolSpan, errorType(error, 'tool_failed')); throw error; }
          finally { toolSpan.end(); }
        }));
      }
      return results;
    } catch (error) {
      failed(span, errorType(error, 'pipeline_failed'));
      span.setAttribute('app.failure.stage', stage);
      throw error;
    } finally { span.end(); }
  });
}

To check the SDK setup without an API account, save the following as demo.mjs next to the modules and run node demo.mjs. It prints one metadata event and the model/parent span names and IDs. The provider response and empty plan are fixtures; this checks local tracing, not AI quality or business authorization.

// Local smoke test: provider and plan adapters are fixtures, not a real AI service.
import { tracer, provider, exporter } from './tracing.mjs';
import { runPipeline } from './observability.mjs';

try {
  await runPipeline('Local demonstration', {
    createMessage: async () => ({ model: 'claude-sonnet-5-5', stop_reason: 'end_turn',
      usage: { input_tokens: 10, output_tokens: 2 }, content: [] }),
    validateAndAuthorizePlan: async () => [], // No tools in this fixture
    executeTool: async () => { throw new Error('No tool should run in this fixture'); }
  }, tracer, event => console.log(JSON.stringify(event)));
  await provider.forceFlush();
  console.log(JSON.stringify(exporter.getFinishedSpans().map(span => ({
    name: span.name, trace_id: span.spanContext().traceId,
    parent_span_id: span.parentSpanContext?.spanId
  }))));
} finally { await provider.shutdown(); }

With this in place, a single trace ID shows the full shape of an agent run: how long planning took versus each tool call, which step failed, and how token usage split across the sub-calls. This is the difference between "the agent timed out" and "tool call #4 (inventory lookup) hung for 9 seconds while the model itself responded in 400ms." See Multi-Agent Systems and Tool Use for the orchestration patterns these spans are wrapping.

Metrics for Incident Response

Dashboards drown in metrics that don't change your response to an incident. For AI features, a small set actually matters:

The following are example alert policies. Choose windows, traffic floors, and thresholds from your service objectives; a single rate-limit response is not automatically a paging incident.

Cost and quality metrics come from two different systems that should feed the same dashboard: AI Cost Modeling covers budgeting and spend alerts, and AI Evals in Production covers running eval suites against production traffic to catch quality regressions. Observability is the layer that correlates both against latency and errors in real time, instead of leaving them as separate weekly reports.

Building a Dashboard for Incident Response

A useful AI dashboard answers three questions in order, because that's the order an on-call engineer actually asks them during an incident:

1. Is it happening now, and how widely? Error rate and latency, sliced by model and prompt version, over the last hour.
2. What changed? A timeline of prompt deploys, model version changes, and config changes overlaid on the error/latency graphs — in AI features, a prompt or model change is often the trigger rather than a code change.
3. Which specific requests were affected? A way to jump from "errors spiked at 14:22" to the actual request IDs, traces, and (if sampled) prompts from that window.

Copy-Paste Prompt: Instrument an Existing AI Integration

Use this to add logging and tracing to an AI call that currently has none:

I have an AI integration in [file/function] that calls [model/provider]
with no observability. Add:

1. A structured log event per call: request ID, model, prompt version,
   input/output/cache token counts, latency, tool calls made, status,
   and stop reason when available. Separate uncached, cache-read, cache-write,
   and total input counts. Distinguish response received from app success.
2. An OpenTelemetry span wrapping the call, nested under the current
   request's parent span if one exists, with token counts and model
   as span attributes.
3. If this call is part of a multi-step pipeline (tool calls, subagent
   calls), wrap each step in its own child span so the full pipeline
   shows up as one trace.
4. No raw prompt, output, tool arguments/results, or exception text in
   general telemetry. If content capture is required, describe its explicit
   authorization, pre-export redaction, access controls, retention, and
   disable switch separately. A sample flag alone is not permission.

Show me the diff. Do not change the AI call's actual behavior or
retry logic — this is instrumentation only.

Sources and review scope

Checked October 27, 2026: pinned Anthropic conventions, span conventions, and exception-content warning; OpenTelemetry JS SDK 3.x migration; and Anthropic usage fields. The explicit model ID remains an example, not a claim about the best current model. The tracing SDK was exercised locally with simulated model and tool adapters; no paid API call or production exporter was run. Recheck by November 26, 2026, or earlier when conventions, SDKs, provider usage fields, or models change.

Related Guides