VS Code Agent Usage Metrics Are Not Productivity Metrics: Build a Dashboard That Says What It Measures

What GitHub Added on September 11

GitHub's September 11, 2026 changelog says enterprise and organization reports now include generally available metrics for the dedicated VS Code Agents window. The additions appear in both 1-day and 28-day aggregate reports and their user-level counterparts.

The product boundary belongs in the dashboard title, not a footnote: these fields cover the dedicated VS Code Agents window only. GitHub says they remain separate from editor-window Agent Mode and generic usage rollups. “VS Code Agent adoption” is therefore too broad unless your label also names the measured window.

Start With a Metric Dictionary

Give every tile five pieces of metadata: source report, grain, window, null rule, and interpretation boundary. For example, a daily-active-users tile should say “organization aggregate; one row per report day; dedicated VS Code Agents window; unavailable values excluded; adoption signal only.” That sentence prevents four common errors before anyone opens a spreadsheet.

• Do not add daily actives across days. The same person may be active on several days. A sum is user-days, not unique people over the period.
• Do not divide messages by active users without labeling the ratio. It is messages per active user, not tasks completed per developer.
• Do not rank individuals by session or message counts. A high count might mean sustained use, a difficult task, repeated corrections, or a workflow that fragments conversations.
• Do not merge surfaces silently. A team that works mainly in editor-window Agent Mode can look inactive in this dataset while using agents heavily elsewhere.

The dictionary is a measurement contract. It makes later analysis reviewable in the same way that the AI Verification Ladder makes evidence proportional to risk.

Preserve Absent, Null, and Zero

GitHub marks the new properties optional and says they may be absent or null when corresponding data is unavailable. That is not equivalent to a measured zero. If an old parser fills missing values with zero, the launch date can appear as an adoption collapse. If a chart drops missing days without marking them, the trend can appear smoother than the source.

Keep both the parsed value and a presence state:

function optionalCount(row, key) {
  if (!Object.prototype.hasOwnProperty.call(row, key)) {
    return { value: null, state: "absent" };
  }
  if (row[key] === null) {
    return { value: null, state: "null" };
  }
  return { value: Number(row[key]), state: "present" };
}

For the nested totals object, test the parent before reading either child. Store the raw report beside normalized rows, reject unexpected types, and expose an “availability” strip above the chart. A useful daily table has report_day, scope_id, field_state, active_users, sessions, and user_messages. Only rows with field_state = present belong in a numeric trend.

Use 1-Day and 28-Day Reports for Different Questions

The Copilot usage metrics API reference describes specific-day endpoints and latest 28-day endpoints for enterprise and organization scopes. A specific-day report covers one complete processed day. The 28-day response supplies its actual start and end dates and ends on the latest processed day. Use those returned dates; do not label the tile “last 28 days” based on the viewer's clock.

The daily report is suitable for ingestion checks, rollout events, and gaps. The 28-day report is suitable for a stable adoption snapshot and user cohort analysis. Do not splice a daily point and a 28-day total into one line: their grains and denominators differ. Keep two panels:

The second panel does not come from these new fields. It needs separately governed engineering data, agreed definitions, and privacy review. The first panel can be useful without it, but it must stay labeled “adoption and activity.”

Record the Scope and Access Contract

The REST call returns signed download links, not the metric rows themselves. Record the endpoint, requested day or returned range, report scope, retrieval time, API version, and downloaded object checksum. A dashboard that loses that lineage cannot explain why two exports disagree.

Access also differs by scope. The docs name enterprise owners, billing managers, and users with the relevant enterprise Copilot-metrics permission for enterprise reports. Organization reports name organization owners and users with the corresponding organization permission. For classic tokens, enterprise endpoints accept manage_billing:copilot or read:enterprise; organization endpoints require read:org. The Copilot usage metrics policy must be enabled before these endpoints work.

Put that preflight in the runbook: policy enabled, authorized identity, expected scope, successful download, and schema check. A 403 is an access failure; an absent optional field in a successfully downloaded report is a data-availability state. Treating both as zero hides the difference.

Join Outcomes Without Claiming Causation

Suppose a team's 28-day report shows more users, sessions, and messages while median review time falls. That is a useful observation, not proof that the agent caused the change. Team composition, project phase, staffing, release freezes, and task mix can all move the outcome.

Run one explicit confounder check before presenting the comparison:

• Task mix: split maintenance, feature, and incident work instead of comparing one blended average.
• Team exposure: record who had access to the dedicated Agents window throughout the period.
• Concurrent changes: mark staffing changes, release freezes, CI upgrades, and review-policy changes on the same timeline.

Define a cohort before looking at results, keep calendar windows aligned, and report denominators. Compare at team or repository level where practical, not by scoring individuals. Add a pre-rollout baseline and annotate other process changes. For a controlled rollout, pair the activity panel with the outcome framework in the Microsoft CLI-agent rollout analysis: retention, review load, rework, quality, and cost all matter alongside throughput.

A defensible dashboard conclusion sounds like this: “Dedicated VS Code Agents-window adoption increased in the returned 28-day period; delivery and quality panels show the separately defined outcomes for the same cohort.” It does not say “messages increased productivity.” The first statement matches the contract. The second invents a relationship the API does not measure.

The Acceptance Checklist

1. Title every usage tile with “dedicated VS Code Agents window.”
2. Show the exact report scope and returned day or 28-day range.
3. Preserve absent, explicit null, present zero, and present nonzero states.
4. Keep daily and 28-day grains in separate views.
5. Describe sessions and messages as activity, never completed work.
6. Place delivery and quality outcomes in a separate panel with independent sources.
7. Suppress individual rankings and review the user-level use case with privacy stakeholders.
8. Retain endpoint, API version, retrieval time, checksum, and access identity class.

If the dashboard can pass those eight checks, it can answer a useful question without pretending to answer a larger one. GitHub has added observability for adoption of a particular agent surface. The engineering work is to preserve that precision all the way from optional JSON fields to the sentence a leader reads.