Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford

Channel:AI Engineer · Length:18:12 · Published:2025-07-23 · Views:313276

Watch on YouTube

Summary

Yegor Denisov-Blanch of Stanford presents a study of nearly 100,000 developers at hundreds of companies, measuring how AI affects productivity from commit data. It finds an average boost of about 20% with wide variance by codebase, task complexity and stack.

Why HN recommends it

Cited in AI-coding debates as one of the largest data-driven studies, notably for finding no correlation between self-assessed and measured productivity.

This is a study that tries to be scientific, unlike developer self reports and CEO promises of 10x. — HN, 2025

Software practice, open source & careers · LLMs & generative AI

Subtitles (excerpt)

[Music] In January of this year, Mark Zuckerberg said that he was going to replace all of the mid-level engineers at Meta with AI by the end of the year. I think Mark was a bit optimistic and he was probably acting like a good CEO would to inspire a vision and also probably to keep the Facebook stock price up. But what Mark also did was create a lot of trouble for CTOs worldwide. Why? Because after Mark said that, every single CEO in the world almost turned to their CTO and said, "Hey, Marcus says he's going to replace all of his developers with AI. Where are we in that journey?" And the answer probably was honestly not very far and we're not sure we're going to do that. And so I personally think hopefully this is not um you know going to going to change, but I don't think AI is going to replace developers entirely. at least at least this year, let alone at at at Meta, right? But um and I do think that AI increases developer productivity, but there's also cases in which it decreases developer productivity. So AI or using AI for coding is not a one-sizefits-all solution, and there are cases in which it shouldn't be used. And so for the past three years we've uh been running one of the larger uh studies

On Hacker News

This Stanford study on developer productivity found 0 correlation between developers assessment of their own productivity and independent measures of their productivity. Any anecdotal evidence from developers on how AI has made them more or less productive is worthless. [VIDEO] — Ianjit, 2026-01
As mentioned in the thread I linked, they acknowlege the productivity puzzle and try to control for it in their studies. It's worth reading them in detail, I feel like many of them did a decent job controlling for many factors. For instance, when measure the number of PRs they ensure that each one goes through the same review process whether AI-assisted or not, ensuring these PRs meet the same qu… — keeda, 2025-12
The folks at Stanford in this video have a somewhat similar dataset, and they account for "code churn" i.e. reworking AI output: [VIDEO] -- I think they do so by tracking if the same lines of code are changed in subsequent commits. Maybe something to consider. — keeda, 2025-12
"That study design has some issues. " This is a study that tries to be scientific, unlike developer self reports and CEO promises of 10x. Can you point to a better study on the impact of AI on developer productivity? The only other one I can think of finds a 20% uplift in productivity. [VIDEO] — Ianjit, 2025-10
This contains some specific data with pretty graphs: [VIDEO] But if you do professional development and use something like Claude Code (the current standard, IMO) you'll quickly get a handle on what it's good at and what it isn't. I think it took me about 3-4 weeks of working with it at an overall 0x gain to realize what it's going to help me with and what it will make take longer. — stitched2gethr, 2025-09

Related videos