AI Coding Assistants and Productivity: What the Research Actually Says

The research on AI coding assistants does not give one number. Controlled experiments on well-defined tasks show large speedups (55% faster in GitHub's 2022 study), field experiments in companies show a smaller but real gain (about 26% more completed tasks), and the best-known randomised trial on experienced developers working in their own large codebases found they were 19% slower with AI. At team level, Google's DORA research finds more throughput but less stability.

Here is what each major study measured, why they disagree, and how to use the evidence when you plan a roadmap or judge a developer's estimates.

The main studies, side by side

GitHub's controlled experiment (2022)

GitHub asked 95 professional developers to write an HTTP server in JavaScript. Those using Copilot finished in 1 hour 11 minutes on average, against 2 hours 41 minutes without it: 55% faster. 78% of the Copilot group completed the task, against 70% without. This is the source of the often-quoted "55%" figure. It measured one greenfield, clearly specified task, which is close to the best case for an assistant.

Field experiments at Microsoft, Accenture and a Fortune 100 company

A team of economists (Cui, Demirer, Jaffe, Musolff, Peng and Salz) randomly gave some developers access to an AI coding assistant in their normal jobs. Across the three experiments and 4,867 developers, the tool led to a 26.08% increase (standard error 10.3%) in completed tasks. Less experienced developers adopted the tool more and gained more.

METR's randomised trial with experienced open-source developers (2025)

METR recruited 16 experienced developers from large open-source repositories they already maintained. Each of 246 real issues (about two hours each) was randomly assigned to "AI allowed" or "AI not allowed". The main tool was Cursor Pro with Claude 3.5/3.7 Sonnet. With AI, tasks took 19% longer. Before starting, the developers expected a 24% speedup; after finishing, they still believed AI had sped them up by 20%. METR is careful to say the result does not show AI fails everywhere, only that it slowed these developers on this kind of work at that time.

METR's follow-up (February 2026)

METR ran a larger study from August 2025 with 57 developers, 143 repositories and more than 800 tasks. The estimated speedup was 18% for returning developers (confidence interval from a 38% speedup to a 9% slowdown) and 4% for newly recruited ones (from 15% faster to 9% slower). METR itself calls this an unreliable signal: many developers refused to take part because they did not want to work without AI, 30% to 50% avoided submitting tasks they expected AI to speed up a lot, and pay dropped from $150 to $50 an hour. METR is changing its study design rather than publishing a headline number.

Google's DORA report (2025)

The 2025 DORA report surveyed nearly 5,000 technology professionals. AI adoption reached 90%, with a median of two hours a day spent working with AI. Over 80% said AI improved their productivity and 59% said it improved code quality. On delivery metrics, Google's DORA team found a positive relationship between AI adoption and software delivery throughput, but AI adoption "does continue to have a negative relationship with software delivery stability."

Where the gains show up

• Self-contained, well-specified tasks, like the HTTP server in the GitHub experiment.
• Less experienced developers, who gained the most in the field experiments.
• Team throughput: DORA links AI adoption to teams shipping more.
• Developer sentiment: in the 2025 Stack Overflow survey, 84% of respondents were using or planning to use AI tools, and 60.7% viewed them favourably.

Where they don't

• Experts in large codebases they know well, the METR 2025 setting. Context, conventions and quality standards that live in a maintainer's head are hard to hand to a model.
• Stability. DORA's explanation is that AI increases change volume, and "without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability."
• "Almost right" code. 66% of Stack Overflow respondents named "AI solutions that are almost right, but not quite" as their top frustration, and 45.7% distrust the accuracy of AI output against 32.3% who trust it.
• Self-assessment. METR's developers felt 20% faster while measuring 19% slower. Opinions about AI speed are not evidence.

Why the studies disagree

They measure different things. GitHub timed one task. The field experiments counted completed tasks in developers' day-to-day work. METR timed real issues in mature projects, with screen recordings. DORA relies on surveys linked to delivery metrics. The populations differ too (professionals on a test task, employees at large companies, open-source maintainers), as do the tools, which have improved quickly between 2022 and 2026. None of these studies is wrong; they answer different questions.

What this means for your product team

1. Measure delivery, not impressions. Track how long changes take to reach production, how often you deploy, how often a release breaks something, and how much work gets redone. If you work with a remote developer, a weekly release rhythm makes these numbers visible.
2. Put tests and CI in place before pushing more AI-generated code. DORA's finding on stability is a direct warning. Automated end-to-end tests, such as the Playwright setup we describe here, are the kind of control system it refers to.
3. Don't cut estimates in half because of a vendor figure. A realistic planning assumption sits somewhere between the field-experiment gain and no gain at all, depending on the codebase and the developer.
4. Budget review time. More code means more to review, and AI-generated code has its own security failure patterns.
5. Watch for debt. Code that is "almost right" and merged anyway becomes technical debt.

Frequently asked questions

Do AI coding assistants make developers faster?

Often, but not always. Studies range from 55% faster on a single controlled task to 26% more completed tasks in company field experiments, while METR's 2025 trial found experienced developers 19% slower in their own large codebases.

Did the METR study show AI slows developers down?

For 16 experienced open-source maintainers working on their own repositories in early 2025, yes: tasks took 19% longer. METR does not generalise this to all developers, and its 2026 follow-up was too affected by selection bias to give a reliable new estimate.

How much productivity gain should a startup plan for?

Plan conservatively and measure. The company field experiments found about 26% more completed tasks on average, with bigger gains for less experienced developers. Your result depends on task type, codebase and testing practices.

Should I require my developers to use AI tools?

Adoption is already near universal (90% in DORA 2025). The more useful requirement is on outcomes: tests, code review and stable releases, whatever tools are used.

Sources

1. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, 2025-07
2. We are Changing our Developer Productivity Experiment Design, METR, 2026-02
3. Research: quantifying GitHub Copilot's impact on developer productivity and happiness, GitHub, 2022-09 (updated 2024-05)
4. The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers, Microsoft Research (Cui, Demirer, Jaffe, Musolff, Peng, Salz), 2025-06
5. Announcing the 2025 DORA Report, Google Cloud, 2025-09
6. How are developers using AI? Inside our 2025 DORA report, Google, 2025-09
7. 2025 Developer Survey: AI, Stack Overflow, 2025