Treat Export as a Data Boundary
GitHub announced OpenTelemetry support in the Copilot app on September 22, 2026. The app can receive OTel configuration through enterprise-managed settings and send agent activity to a compatible monitoring tool. That extends the observable path across agent sessions, model requests, and tool use, but it also creates a new outbound data route that deserves an acceptance test before broad enablement.
The release says prompt and response content is excluded by default. Make that boundary explicit anyway. Defaults can be obscured by an old user setting, an environment variable, or an incomplete rollout. The enterprise settings reference provides both captureContent and lockCaptureContent; set the first to false and the second to true. Then inspect what the collector actually receives.
This procedure is narrower than the vendor-neutral design in the agent trace observability guide and the production program in AI observability. It answers one product-specific question: can the current Copilot clients export useful metadata through the managed-settings contract without sending synthetic prompt, response, or tool content?
Start with an Isolated Collector
Create a pilot endpoint with a separate dataset, a short retention period, and access limited to the test operators. Do not point the first run at the production observability tenant. The test needs raw-payload inspection and controlled deletion, while a production pipeline may transform, sample, or redact fields before you can examine them.
Use placeholder credentials in the configuration repository and inject the real collector token through the approved managed-settings delivery mechanism. GitHub documents http/json and http/protobuf as accepted managed protocol values. This fixture chooses protobuf only to make the test concrete:
{
"telemetry": {
"enabled": true,
"endpoint": "https://otel-pilot.example.internal",
"protocol": "http/protobuf",
"captureContent": false,
"lockCaptureContent": true,
"serviceName": "copilot-pilot",
"resourceAttributes": {
"deployment.environment": "telemetry-pilot",
"rollout.wave": "wave-0"
},
"headers": {
"Authorization": "Bearer MANAGED_SECRET"
}
}
}The host, service name, attributes, and secret are examples, not GitHub-required values. Use a unique rollout.wave value so the collector query can distinguish this test from other Copilot traffic. Record the managed-policy revision, delivery channel, test identity, client name and version, operating system, UTC start time, collector dataset, and query link before starting a session.
Use Canaries That Are Safe to Search
Create a disposable repository containing telemetry-fixture.txt with the value TOOL-CANARY-7F3A. Start a Copilot session with the prompt: “Read telemetry-fixture.txt and answer with PROMPT-CANARY-4D91 followed by the file value.” These are synthetic markers, not secrets. Their only purpose is to make an accidental content export detectable.
Ask the agent to read the file once and produce the response. Preserve the session start and end times and the collector's trace identifier. Repeat with a second session that does not call a tool, using a different prompt canary. The pair separates a basic model exchange from a tool-bearing interaction and prevents one missing tool span from being mistaken for total export failure.
A content-boundary test deliberately searches the telemetry pipeline. Use meaningless unique strings. If content capture is unexpectedly active, a real credential would already have crossed the boundary you are trying to validate.
Prove the Trace Shape
For Copilot CLI, GitHub documents a hierarchical tree with one invoke_agent root span and chat and execute_tool children. The root covers the agent work for one user message; a chat span represents an LLM request, and an execute_tool span represents a tool call. Use that exact hierarchy as the CLI acceptance case. For the Copilot app or VS Code, record the observed signal shape and compare it with the client documentation available for the tested version rather than assuming every client emits an identical field set.
| Check | Expected evidence | Failure meaning |
|---|---|---|
| Routing | service.name=copilot-pilot and rollout.wave=wave-0 | Wrong policy, endpoint, or resource attributes |
| Agent session | An agent root correlated to the recorded test window | No usable session trace arrived |
| Model call | At least one model-request child with model and duration metadata | Partial trace or unexpected client support |
| Tool call | A tool child for the file read, including tool name but not arguments or result | Tool activity is missing, or content crossed the boundary |
| Metrics | Token and duration measurements for the pilot service where the client emits them | Metrics path or query needs investigation |
Do not turn every field into a mandatory cross-client contract. The CLI reference is specific about CLI spans and metrics. The September 22 app announcement promises visibility into session flow, model requests, and tools, but does not enumerate an identical schema for every app build. Gate each client on documented and observed behavior, and store the raw field inventory with the client version.
Test What Must Not Be Present
Search the raw accepted payloads and the retained dataset for both prompt canaries, the tool-file canary, and the response string. Also query for content-only attribute names documented by the CLI reference: gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions, gen_ai.tool.call.arguments, and gen_ai.tool.call.result. With capture disabled, the fixture should not populate those values with the test content.
A negative query is meaningful only when paired with positive control evidence. First prove the same session produced the expected metadata spans; otherwise “zero canary matches” might mean nothing was exported. Keep the collector's ingestion count, trace ID, raw-payload sample, and canary-search query together in one evidence record.
The CLI documentation still permits useful metadata such as model names, token counts, timing, finish reasons, session identifiers, tool names, and error types. Decide whether your organization treats any of those identifiers as sensitive before rollout. “No prompt content” does not mean “no governed data.” Apply access, retention, and deletion rules to the telemetry dataset itself.
Check the Local Override Boundary
On the test device, inspect relevant Copilot OTel environment variables before and during the run. The CLI reference says export can activate when COPILOT_OTEL_ENABLED=true, OTEL_EXPORTER_OTLP_ENDPOINT is set, or COPILOT_OTEL_FILE_EXPORTER_PATH is set. Record any existing values so a second exporter or local file does not confuse the managed-path result.
Attempt to enable content capture through the client control available to the test user, then repeat the synthetic session. The managed setting uses lockCaptureContent: true specifically to prevent users from changing captureContent. The acceptance evidence is not merely a disabled toggle: it is another positive metadata trace plus zero canary matches and no populated content-only fields.
Keep telemetry policy testing separate from tool-permission testing. The managed-permissions acceptance fixture covers denied, approval-required, and allowed operations. Here, the file read is intentionally harmless and exists only to produce a recognizable tool span and a content canary.
Prove the Rollback
Change only telemetry.enabled to false in the managed policy, increment the policy revision, and wait for the normal delivery and client-refresh path. Record when the revised policy becomes effective. Then start a third session with a new marker, ROLLBACK-CANARY-82BE, and perform the same harmless read.
The rollback passes when the client still functions but no new pilot-service spans or metrics arrive after the documented propagation and collector-flush window. Query by the unique rollback canary, test identity metadata, service name, and UTC interval. Preserve the last pre-disable signal time and the query result after the waiting window. If signals continue, stop the rollout and investigate policy delivery, a still-running session, queued exporter data, or an unmanaged local exporter.
Re-enable only after the disable path has been observed. A configuration diff that says false is not a rollback test; the external effect is the stopped stream. Keep the collector endpoint available until the stop has been verified, then revoke the pilot token and delete the isolated dataset under its retention procedure.
Use an Evidence Gate
Approve the next rollout wave only when every in-scope client has: a recorded managed-policy revision, a positive metadata trace, the expected model and tool visibility, zero matches for all synthetic content canaries, an unsuccessful local content-capture override, and a tested disable path. A client that does not export, exports unexpected content, or keeps exporting after disable is a stop condition.
Build dashboards only after that gate. In CLI data, read github.copilot.nano_aiu from the root invoke_agent span; GitHub warns that the value also appears on child chat spans and will be double-counted if summed across the tree. Treat github.copilot.cost as a model multiplier, not money. Those two details prevent an observability rollout from creating misleading consumption reports.
The durable deliverable is a small evidence bundle: policy revisions for enable and disable, client inventory, collector configuration, two positive trace samples, canary queries, override result, rollback query, and named owners for the exporter, collector, and data retention. Repeat the content-boundary and rollback cases after client upgrades or changes to the managed telemetry block.