Quick answer: a randomized trial by METR found that experienced developers using AI tools in early 2025 took 19 percent longer on real tasks, even though they believed AI had made them about 20 percent faster (METR).
METR has since marked that result as out of date, and its newer study points to likely speedups but is too unreliable to quantify (METR’s February 2026 update). The lasting lesson is the gap between feeling fast and being fast, and it gives you a habit to try: measure your own time.
Contents
The Study Behind the Headline
According to METR, the trial involved 16 experienced developers who had contributed for multiple years to large open-source repositories, averaging more than 22,000 stars and over a million lines of code. They worked on 246 real issues they had listed as valuable to their repositories: bug fixes, features and refactors, averaging about two hours each.
Each issue was randomly assigned to allow or disallow AI use. When AI was allowed, developers could use any tools they chose, primarily Cursor Pro with Claude 3.5 and 3.7 Sonnet. Developers recorded their screens and self-reported total implementation time.
With AI allowed, developers took 19% longer to complete issues. Before the study they forecast AI would speed them up by 24%, and after it they estimated AI had sped them up by 20%.
What the Authors Ruled Out, and What They Cautioned
METR reports that it ruled out several experimental artifacts: developers used frontier models, complied with their treatment assignments, did not differentially drop issues, and submitted pull requests of similar quality with and without AI. The slowdown persisted across different outcome measures and subsets of the data. The authors investigated 20 potential factors and found evidence that five likely contribute to the slowdown.
They also list limits. They do not claim AI slows most developers or fails in other domains, and they say existing AI could be used more effectively in this setting than the study’s tooling allowed. Developers had used Cursor for only a few dozen hours before and during the study, so learning effects cannot be ruled out. Self-reported time can be inaccurate and overoptimistic, which is part of why the study’s gap between perception and measurement is interesting.
What METR Reported Next
METR’s February 2026 update says its second study, using late-2025 tools with 57 developers across 143 repositories and more than 800 tasks, is unreliable because of selection effects. More developers said they would not work without AI, so the study is missing those with the most optimistic views of AI, and 30% to 50% of surveyed developers said they withheld some tasks because they did not want to do them without AI. Time tracking was also unreliable for developers running multiple agents at once.
The reported estimates were -18% for the 10 returning developers (95% interval from -38% to +9%) and -4% for newly recruited developers (-15% to +9%), on the same scale as the original study, where a positive number means longer task time and a negative one means shorter. Both intervals include zero. METR says it believes developers are likely more sped up in early 2026 than in early 2025, but that its data is “only very weak evidence” for that, and it is redesigning the experiment.
For you, the practical reading is: tools change quickly, any single study is a snapshot, and your own measurement is the evidence that applies to your work.
Turning the Gap Into a Habit You Can Test
The study’s most practical message is that felt speed and measured speed can point in opposite directions. These steps are my suggestions, not findings from the study.
- Log your own time for two weeks. Write one line per task: what it was, whether you used AI, and how long it actually took. Compare the averages at the end.
- Write the test or expected result first. That gives the AI’s output a concrete check instead of relying on how plausible the code looks.
- Read each suggestion before accepting it. Treat it the way you would treat a first draft from a colleague.
- Compare task types. If the log shows AI helps on small, well-defined tasks and costs time on complex ones, use it where it helps.
- Re-measure when your tools change. METR’s own results moved between 2025 and 2026, so a log from last year may not describe this year.
A Quick Calculation for Your Own Log
Here is the arithmetic with assumed numbers. Suppose a task takes 10 hours without AI. If AI made it 19 percent longer, it takes 10 × 1.19 = 11.9 hours. If you believed AI had made you 20 percent faster, you would have expected about 10 ÷ 1.2 = 8.3 hours. The gap between 8.3 and 11.9 hours is 3.6 hours, which is the kind of difference a personal time log can catch. The 10 hours is an assumption for illustration.
Below is my work log for two weeks. Each row has the task, whether I used AI (YES or NO), and the minutes it took.
[paste the log]
Please:
1. Count the tasks and add up the minutes for the YES group and the NO group, showing each sum.
2. Calculate the average minutes per task for each group, showing the division.
3. Calculate the percent difference as (YES average minus NO average) divided by the NO average, showing the steps. A positive number means tasks with AI took longer.
4. Recalculate step 3 after removing my single longest YES task, so I can see how much one task moves the result.
Rules:
- Use only the log.
- Do not say whether AI helps or hurts me.
- If either group has fewer than 10 tasks, write "SMALL SAMPLE."An invented example. Say the NO tasks took 90, 110, 120, 130 and 150 minutes, which sum to 600, so the average is 600 divided by 5, or 120. The YES tasks took 200, 100, 110 and 130, which sum to 540, so the average is 540 divided by 4, or 135. The difference is 135 minus 120, or 15 minutes, and 15 divided by 120 is 12.5%, so AI tasks took 12.5% longer. Now remove the longest YES task, 200 minutes.
The remaining three sum to 340, the average is about 113.3, and (113.3 minus 120) divided by 120 is about -5.6%, so the sign flips. Both groups are far under 10 tasks, so the output should say SMALL SAMPLE. The 10-task cutoff is my rule of thumb, not a statistical standard, and the log is invented. The takeaway is the same as the trial’s: measure over enough tasks before trusting either feeling or a single number.
The Five Factors Behind the Slowdown, From the Paper
The blog post says METR examined 20 potential factors and found evidence that five likely contribute to the slowdown, but the details sit in a table. The full paper on arXiv (Section 3.3 and Appendix C) lists them:
- Over-optimism about AI usefulness. Developers forecast a 24% time reduction and afterward estimated 20%, while observed time rose 19%. The authors suggest this optimism may lead developers to keep using AI despite its cost.
- High familiarity with the repositories. Developers were slowed more on issues where they reported high prior exposure and low need for outside resources. They averaged 5 years of experience and 1,500 commits on their repositories, which the authors suggest makes AI less helpful than in earlier studies.
- Large, complex repositories. The repositories averaged about 10 years old and over 1,100,000 lines of code. Developers said AI performed worse there, sometimes making unwanted edits elsewhere in files.
- Low AI reliability. Developers accepted fewer than 44% of Cursor’s generations; 75% read every line of AI-generated code, and 56% often made major changes to clean it up. In a labeled subset of 44 issues, they spent about 9% of their time reviewing and cleaning AI outputs.
- Implicit repository context. This one rests on qualitative evidence only: developers said AI lacked tacit knowledge of the codebase, such as undocumented edge cases and backward-compatibility needs.
The paper reports the shift in how time was spent in a different section: with AI allowed, developers spent less time actively coding and searching, and more time prompting, waiting, reviewing and idling.
Turn Each Factor Into a Column in Your Log
The factors suggest columns worth adding to the log from the habit section above. These mappings are my suggestions, and the numbers they refer to come from the paper.
- Over-optimism: write your guess of the time before you start, then the real time. The paper’s 24% forecast versus a 19% increase shows why the guess alone is not enough.
- Reliability: note whether you kept the suggestion, edited it heavily or threw it away. The paper’s under-44% acceptance figure gives you something to compare against.
- Review cost: note the minutes spent reading and fixing AI output on each task. The paper put this at about 9% of time in its subset. For scale, 9% of a two-hour task is 0.09 × 120 = 10.8 minutes.
- Repository size and familiarity: tag each task as “know this code well” or “new to me.” The paper found larger slowdowns on well-known code, so your log may show a different pattern for unfamiliar code.
This prompt computes the review share from your log (my suggestion):
Below is my work log for two weeks. Each row has the task, whether I used AI (YES or NO), the total minutes, and for YES tasks the minutes I spent reviewing or fixing AI output.
[paste the log]
Please:
1. Add up the total minutes and the review minutes for the YES tasks, showing each sum.
2. Calculate review minutes divided by total minutes as a percentage, showing the division.
3. List the YES tasks where the review share is above 20%.
Rules:
- Use only the log.
- Do not say whether AI helps or hurts me.
- If I have fewer than 10 YES tasks, write "SMALL SAMPLE."An invented example. Say the YES tasks in your log took 540 minutes in total, and 54 of those were spent reviewing and fixing AI output. The review share is 54 ÷ 540 = 10%, a bit above the paper’s about 9%.
One task at 120 minutes with 36 spent reviewing has a share of 36 ÷ 120 = 30%, so it appears on the list of tasks above 20%. Looking at that task tells you what kind of work makes the output expensive to check. The log is made up; the 9% figure is from the paper, and the 10-task cutoff is my rule of thumb.
The encouraging part is that the paper also says what could change the picture: more experience with the tool (the developers had only a few dozen hours of Cursor use), better setups, and different kinds of work. Your own log tells you which of those applies to you.
My View on AI Coding Assistants
This closing part is my opinion. I think using AI well is a skill, and the log is the quickest way I know to learn where it helps you and where it does not. Your results may differ from the trial’s, and that is the point of measuring them.
AI Coding Assistants: Sources
- METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- METR: We are Changing our Developer Productivity Experiment Design (Feb. 24, 2026)
- arXiv:2507.09089: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
For another look at using AI without handing over the thinking, see learning to code with AI.
Leave a Reply