The dashboard turned green about a month after your team switched on the AI coding tools, and everyone exhaled. Lines of code accepted climbed, suggestions used climbed, and story points per sprint climbed. Then you walked over and talked to the actual engineers, and the story got a little fuzzier.
Some of them swear they are flying. A couple is quietly spending their afternoons cleaning up code the AI wrote that morning. Nobody in the room can point to a single number that captures both of those things at once, and yet a number is exactly what leadership asked for. That gap between the confident chart and the murky reality beneath it is the whole problem with measuring AI developer productivity right now.
If that scene feels familiar, you are not measuring badly. You are measuring the wrong thing with tools that were shaky before AI and got shakier after. Let's get into why.
The dashboard got louder, not smarter
The comfortable lie is that a green productivity chart tells you whether AI is working. The uncomfortable reality is that most of those charts measure activity rather than value. They tell you how much AI-flavored work is happening, not whether any of it made your product better or your customers happier.
Here is the part I only say to people I trust: we spent years confusing motion for progress long before anyone typed a prompt, and AI just cranked the motion up until the confusion became impossible to ignore. The dashboard did not get smarter. It got more confident. That is the same amplifier effect we covered when AI made everything louder, applied to your metrics instead of your meetings.
Is AI making developers faster? We honestly can't cleanly say
The best evidence we have for how hard this is to measure comes from people who tried to measure it carefully. In early 2025, the research group METR ran a proper randomized trial with experienced open-source developers working on their own repositories.
Before starting, those developers predicted AI would make them about 24 percent faster. Afterward, they felt they had been about 20 percent faster. When METR measured the actual clock time, the AI-assisted tasks took about 19 percent longer in that snapshot. Please read that carefully, because the point is not the slowdown.
The slowdown number was one snapshot, one setting, early-2025 tools, and it is not a law of nature. The durable finding is the gap between felt and measured. Smart, experienced people were sure they were faster while the stopwatch disagreed, and expert forecasters overshot too. Then METR went back in 2025 to measure again, but could not get a clean read because AI adoption had broken their control group. In February 2026, they announced they were redesigning the whole experiment.
Source: METR measurement update, February 2026 – https://metr.org/blog/2026-02-24-uplift-update/
Say what? The measurement got harder precisely because the thing being measured became normal. Their honest read today is that developers are probably sped up, but the size cannot be pinned down, and that is the answer to carry into your next budget conversation.
Why your AI productivity metrics break
Once you accept that the magnitude is unknowable for now, the popular proxies start to look flimsy for a reason. Each one breaks in its own way, and it helps to see the pattern rather than the individual number.
Lines of code accepted breaks because volume is not value; a high number can hide duplication and rework that surfaces weeks later. GitClear's analysis of 211 million changed lines found code churn, meaning code reworked within about two weeks, has roughly doubled since AI tools went mainstream. Treat that as a method-specific signal from one dataset, not a universal law, but it makes the point: more accepted code can quietly mean more code you will pay to fix.
Source: GitClear AI Copilot Code Quality research, 2025 – https://www.gitclear.com/ai_assistant_code_quality_2025_research
Suggestion acceptance rate breaks because it rarely changes a decision. Would you act differently at 40 percent versus 70 percent? A high rate can mean fluency, or it can mean people waving through code they do not fully understand, and the two look identical on a chart. Story points per sprint breaks because points were always a planning aid, not a productivity score; when AI makes some tasks feel trivial, teams quietly re-baseline, and the number climbs with no extra value delivered.
Cycle time is the one worth keeping, if you scope it honestly, because a faster coding cycle can still mask a growing queue downstream in review and deploy. This is the same trap behind the pattern in which delivery metrics start lying to you: output rises while stability erodes, and the number that catches it is not velocity; it is the ratio of new work to rework. Running each metric through a couple of hard questions is the heart of the AI Velocity Reality Check I sent subscribers this week, and it is worth building that habit with your own dashboard.
What actually holds up: flow, stability, and rework
The research that survives scrutiny points to a systems view, not a single-hero number. DORA's 2025 report, drawing on nearly 5,000 professionals, found that AI adoption is now associated with higher throughput, so teams are really shipping more. It also found that the same adoption is associated with greater delivery instability. Faster output, shakier stability, and a real speedup that no one can yet size.
Source: DORA 2025, State of AI-assisted Software Development – https://dora.dev/dora-report-2025/
So watch three things together instead of one alone.
Flow: Are you shipping valuable change faster, measured as deployment frequency and lead time from commit to production?
Stability: Are you staying healthy while you move, measured as the change failure rate and time to restore?
Rework: how much of this sprint went to fixing last sprint, which is where an AI speedup quietly turns into an AI tax.
None of those is a magic number, and that is the point. A pair of honest signals, one for speed and one for health, beats a single confident one every time. And when one of them moves in a way you did not expect, the smartest move is not to grade the tool but to read the surprise as a signal about your system.
Common traps when measuring AI developer productivity
The first trap is swapping one shiny number for another and calling it rigor. Pair every speed signal with a health signal so you cannot optimize one into the ground. The real-talk version: if a metric cannot change a decision, it is not a metric, it is decoration, and decoration belongs off the board slide.
The second trap is demanding a precise magnitude that the measurement cannot give you. When you insist on a single "AI made us X percent faster" figure, you pressure your team to invent one, and now you are managing fiction. Ask for direction and confidence instead of false precision, and reward the person who says "we do not know yet" over the one who guesses.
The third trap is watching the dashboard and never the humans. The engineers cleaning up AI output every afternoon are your earliest signal, and they will tell you things no chart can, if you ask before you accuse.
Try this next week
Pick the one AI metric your leadership trusts most, the number that makes it into the board deck. Write a single sentence next to it: "This number is supposed to help us decide _____." Fill in the blank honestly.
If you can name a real decision, keep the metric; it earns its place. If you cannot finish the sentence, you have found a green chart that decorates the slide instead of informing it. Do this with one other person, a tech lead or a product owner, so it is a conversation and not a solo audit, and you will learn more about your real risk posture in ten minutes than most leadership teams learn in a quarter.
The next step, once you have a metric worth keeping, is to pair it with a health signal and a rework signal, then bring all three to your next review instead of the single number you walked in with. If you want a structured way to build this habit in a product role, our AI for Product Owners course focuses on exactly this, using real artifacts rather than clean hypotheticals.
Stop measuring how fast your team types. Start measuring how fast it learns.