What the Study Actually Measured
The paper, Adoption and Impact of Command-Line AI Coding Agents, was posted to arXiv on July 1, 2026 by Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva. It studies Microsoft's early-2026 rollout of two sanctioned agentic command-line tools: Anthropic's Claude Code and GitHub Copilot CLI.
The authors separate two questions that often get mashed together in AI-tool rollouts. First: who tries the tool, and who keeps using it? Second: what changes in output once engineers adopt it? That distinction matters because a company can get a huge launch-week spike without durable use, or plenty of enthusiastic anecdotes without measurable workflow change.
The adoption study focuses on Copilot CLI, where the eligible population was well-defined. The outcomes study covers both Claude Code and Copilot CLI among engineers who adopted. The observation window ends on April 29, 2026, before an internal shift away from most Claude Code licenses would have distorted the results.
The Headline Number Is 24%
The result most teams will quote is the merged-PR lift: adopters merged roughly 24% more pull requests than they would have otherwise, and the paper says that lift persisted across the four-month window. That is unusually concrete for the AI-coding-tool debate, where evidence is often a mix of self-reported productivity, demos, and benchmark tasks that do not look like a real team's queue.
But the paper is careful about what the number means. Merged pull requests are not the same thing as shipped product value. They can hide smaller PRs, review load, rework, churn, bug risk, and whether the merged code solved the right problem. In other words: merged PRs are a useful output metric, not a full ROI model.
That caveat is not a weakness. It is the reason the study is useful. It gives teams a measurable signal while making the measurement boundary visible. If your company is rolling out CLI agents, the right takeaway is not "expect 24% everywhere." It is "build an instrumentation plan before you buy a broad license."
Adoption Was Social
The adoption findings may be more actionable than the productivity number. Initial use spread through visible peer use: reviewer peers, skip-level peers, and direct managers using Copilot CLI all became signals that another engineer might try it. That fits the lived reality of developer tools. Engineers trust tools faster when the person reviewing their PR, pairing with them, or managing their team visibly uses the same tool in real work.
Retention looked different. The paper reports that sustained use was associated more with the engineer's coding activity than with demographics. That means a rollout plan built only around training sessions, blanket enablement, or job level is likely to miss the point. The best early candidates are not necessarily the loudest AI enthusiasts. They are people with enough active coding work for the tool to become part of a loop.
For teams, this suggests a practical rollout sequence: start with active contributors in real repositories, make successful workflows visible to their reviewers and neighboring teams, then measure whether use survives beyond the first two weeks.
What to Measure Before Expanding
- Initial use and retention separately. A first launch tells you curiosity. Sustained use tells you whether the tool found a real workflow.
- PR throughput plus review burden. More merged PRs are not free if reviewers are absorbing more uncertainty, larger diffs, or lower-context changes.
- Rework and rollback. If agents increase output but also increase revert rate, bug fixes, or follow-up cleanup, throughput alone will lie to you.
- Cost per accepted change. CLI agents can burn tokens quickly. Track cost by task class, not just total spend.
- Where the tool helps. Separate scaffolding, test generation, codebase search, migration work, review prep, and incident fixes. One blended productivity number is too blunt.
This is the same discipline behind the AI Change Risk Matrix and the AI Verification Ladder: match the tool to the work, then match the proof to the risk.
The Practical Read
The study makes CLI agents look real, not magical. They can move output in a large engineering organization, but their value depends on adoption mechanics, review capacity, and whether the organization measures the right downstream effects.
If you are planning a rollout, copy the paper's shape before copying its number. Define the eligible population. Separate trial from retention. Track output against a baseline. Add quality and cost measures. Then decide whether broad access is a productivity investment, a training investment, or just a very expensive way to make the PR queue busier.