In early 2025, one of the most carefully designed studies of AI coding tools reached a result few people expected. Experienced developers were 19% slower when they were allowed to use AI. A year later, the same researchers tried to run the study again and could not get a reliable answer, because many developers refused to work without AI at all.
Both results are true. Together, they reveal something more useful than either headline: the way we measure AI developer productivity has fallen behind the way software is now built.
Key takeaways
In a 2025 randomised controlled trial, 16 experienced open-source developers completed 246 real tasks. With AI allowed, tasks took 19% longer.
The developers expected AI to make them 24% faster, and still believed it had made them 20% faster after the slowdown.
A larger follow-up with 57 developers and more than 800 tasks estimated that AI cut task time by 18% for returning developers and 4% for new ones, but the researchers called the data an unreliable signal.
The main reason: developers who rely on AI declined to take part, and 30% to 50% of participants held back tasks they did not want to do without AI.
For any business that builds software, the lesson is to measure delivery outcomes, not how quickly code appears.
The study that found AI made developers slower
In 2025, an independent AI research non-profit set out to answer a simple question with a rigorous method. It recruited 16 experienced developers from large open-source projects, averaging more than 22,000 GitHub stars and over a million lines of code, that each developer had contributed to for years.
The developers listed 246 real issues from their normal work, including bug fixes, new features and refactors. Each issue was then randomly assigned to one of two conditions: AI allowed or AI disallowed. When AI was allowed, developers mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, the leading models at the time. Tasks averaged about two hours, every session was screen-recorded, and developers were paid $150 an hour.
With AI allowed, tasks took 19% longer to finish, with a confidence interval of 2% to 39% longer.
The researchers examined 20 possible explanations and found evidence that five likely contributed. They also ruled out the obvious objections. The developers used frontier models, followed their assigned conditions and submitted pull requests of similar quality with and without AI. The researchers were careful about scope, too. The study described experienced developers working in mature codebases they knew intimately, not software development as a whole.
The perception gap
The slowdown was not the most striking result. Before starting, the developers expected AI to make them 24% faster. After finishing, they still believed it had made them 20% faster. People who had just been measured as slower felt faster.
That gap of around 40 percentage points matters to every engineering leader. If experienced developers can misjudge AI's effect on their own work by that much, then developer sentiment, internal surveys and vendor claims are weak evidence on their own.
One plausible explanation is technical. Writing code is only one part of a developer's day, and working with an AI assistant adds new steps: describing the task, waiting for output, reading it, correcting it and checking it against conventions the model does not know. In a large codebase that a developer already understands deeply, those steps can cost more time than the typing they replace, even when the work feels lighter.
What happened when the study ran again
In August 2025, the researchers began a larger follow-up. By then, development had changed. Open-source developers had moved quickly towards agentic coding tools such as Claude Code and Codex, which can plan a change, edit several files, run commands and iterate on the result with limited supervision.
The new study covered 57 developers across 143 repositories and more than 800 tasks. Ten developers returned from the original study, and 47 were recruited from a wider range of projects, including smaller and newer codebases. Participants had a median of 10 years' experience and were paid $50 an hour.
The results, published in February 2026, pointed the other way:
Returning developers: tasks took an estimated 18% less time with AI, with a confidence interval from 38% less time to 9% more.
New developers: tasks took an estimated 4% less time with AI, with a confidence interval from 15% less time to 9% more.
Both intervals include zero, so neither result is statistically conclusive. The researchers went further and said the data gives an unreliable signal of AI's real effect. They are now redesigning the experiment and have marked their 2025 findings as out of date.
Why the experiment broke
The reasons the follow-up failed are the most valuable part of the story, because each one describes how engineering work has changed.
Developers opted out. A growing share of developers declined to take part because they did not want to do half of their work without AI, even when paid to work on tasks of their own choosing. The study therefore lost the developers with the highest expectations of AI.
Tasks were filtered. When surveyed, 30% to 50% of participants said they had chosen not to submit some tasks because they did not want to do them without AI. Those are likely the tasks where AI helps most. One participant compared going back to manual work with walking across a city after getting used to taking an Uber.
Time stopped being a clean measure. Some developers ran several AI agents at once and worked on unrelated tasks while an agent finished its work. When one engineer supervises parallel agents, the time spent on a single task no longer describes the work.
Pay fell. The hourly rate dropped from $150 to $50, which the researchers believe also contributed to the problem.
The result is a textbook selection effect. A randomised controlled trial is reliable because researchers, not participants, decide which work receives the treatment. Once participants choose which tasks enter the study, and the most AI-dependent developers stay away, randomisation can no longer protect the result. The researchers concluded that their estimate is likely a lower bound, and that developers are probably more sped up by AI now than in early 2025. They also stressed that their data is only weak evidence of how large that increase is.
Put simply, AI became so embedded in daily development that it could no longer be removed for an experiment. That is a finding in its own right.
Speed is not the same as value
In May 2026, the same research group surveyed 349 technical workers, including software engineers, researchers, academics and founders. Respondents reported a median 1.4x to 2x increase in the value of their work because of AI, and a median 3x increase in speed.
The gap between those two figures is important. AI makes some low-priority work so cheap that people start doing it, such as building an interactive dashboard that would never have justified the effort before. Output and speed go up, but value does not necessarily follow. The researchers also warn that self-reported gains need caution, given how far their 2025 participants misjudged their own speed.
For engineering leaders, the core point is simple: how fast code appears is not the same as how much useful software a team delivers.
The bottleneck moves, it does not disappear
Amdahl's law, a principle from computer architecture, is a useful lens here. The overall speedup of a system is limited by the parts you do not speed up. If writing code is only part of delivering a feature, making code generation much faster cannot make delivery equally faster.
For illustration, if writing code takes 30% of the time needed to ship a feature and AI makes that part twice as fast, the feature ships only about 15% sooner. Even if AI wrote the code instantly, it could ship at most 30% sooner. The constraint moves to requirements, code review, testing, integration and release.
Industry data points the same way. Google's 2025 DORA report, based on nearly 5,000 technology professionals, found that 90% use AI at work and more than 80% believe it has increased their productivity. It also found a positive relationship between AI adoption and software delivery throughput, alongside a continued negative relationship with delivery stability. In DORA's words, AI "amplifies what's already there".
Stack Overflow's 2026 Developer Survey, with 30,903 responses from 169 countries, shows how far adoption has gone. 80% of respondents used AI for at least an hour a day, and Claude Code (66%) and GitHub Copilot (59%) are the most used coding agents. Trust remains conditional. 48% trust AI only when they can easily verify the output, and just 6.6% trust it for many tasks including important work decisions.
We explored the risk side of this trade-off in our earlier article, Faster Code, Greater Risks. When a team produces more code, faster, review, testing and security become the limiting factors.
How to measure AI developer productivity
Neither developer opinion nor a vendor benchmark can tell a business whether AI is working. A reliable approach combines delivery metrics, review data and outcome measures.
Set a baseline before rollout. Record delivery metrics for at least one release cycle before introducing new AI tools, so you compare like with like.
Track the five DORA metrics. Change lead time, deployment frequency and failed deployment recovery time measure throughput. Change fail rate and deployment rework rate measure instability. If throughput and instability rise together, part of the gain is borrowed from the future.
Watch the review queue. AI shifts effort from writing code to reading it. Monitor pull request size, time to first review and time to merge. Large AI-generated pull requests that wait days for review are not a productivity gain.
Measure value, not just speed. Ask whether completed work maps to roadmap goals and customer outcomes, not only how many tickets were closed.
Treat self-reported gains as a signal, not proof. Surveys help you understand the developer experience, but the perception gap shows they can be wrong by a wide margin.
How to make agentic AI tools pay off
Measurement tells you whether AI is helping. Engineering practice decides whether it can. In our view, these practices separate useful AI adoption from expensive noise:
Give agents context, not just prompts. Repository instruction files such as AGENTS.md or CLAUDE.md, clear specifications and architecture notes help an agent follow project conventions instead of guessing.
Make tests the contract. A strong automated test suite lets an agent check its own work and lets reviewers trust what passes. Without tests, agentic speed mostly creates review debt.
Keep changes small. Ask agents for focused, reviewable pull requests rather than sweeping rewrites.
Automate the safety net. Run static analysis, dependency scanning and secret detection in CI on every change, whether a person or an agent wrote it.
Keep humans accountable. Engineers remain responsible for architecture, security, data handling and production decisions. Agents can draft. People must own the result.
What this means for businesses commissioning software
For a business choosing a software partner, the question is no longer whether a team uses AI. Almost every team does. The better questions are these:
How do you verify AI-generated code before it reaches production?
Which delivery metrics show that AI has improved your work?
Who is accountable when something fails?
As we argued in AI vs Software Development Companies: Can One Founder Beat a Team of 20 in 2027?, headcount and access to AI tools do not, on their own, decide who delivers the better product. Engineering discipline does.
At DibHub, we believe the past year holds a bigger lesson than any single productivity figure. AI is now so embedded in software development that measuring its effect has become harder than adopting it. The 2025 study showed that confidence and reality can diverge. The follow-up showed that many engineers will no longer work without AI. Both lessons lead to the same conclusion: adopt the tools, and measure the outcomes.
So here is a question worth asking your own team. If your developers had to give up AI tools for a month, would delivery slow down, speed up, or simply become easier to measure?
Faique Ali Raza
Business Development Lead
DibHub
FAQs
Does AI make software developers faster?
It depends on the developer, the codebase and the tools. A 2025 randomised trial found experienced developers were 19% slower with early-2025 AI tools on mature projects. A follow-up with agentic tools estimated time savings of 18% for returning developers and 4% for new ones, but the researchers consider that data unreliable and likely an underestimate. The safest answer is to measure delivery outcomes in your own team.
Why did the follow-up study fail to measure AI's effect?
Because of selection effects. Many developers declined to take part because they did not want to work without AI, and 30% to 50% of participants held back tasks they preferred to do with AI. Time tracking also broke down when developers ran several AI agents at once. The researchers believe the true effect is likely larger than their estimate.
How should a company measure AI developer productivity?
Start with a baseline, then track the five DORA metrics: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. Add review data such as pull request size and time to merge, and check whether shipped work delivers business value. Avoid relying on self-reported speedups alone.







Comments
Loading comments…