Let me guess. Somebody in your org just signed off on an AI budget this quarter. And the justification was some kind of productivity chart — green line, big number, "productivity is up 30%." If I'd stopped that meeting and asked "30% compared to what? Measured how?" the room would go quiet.
That's not because nobody in the room is smart. Everybody in that room is smart. It's because the number was never built to survive the question.
I'm Lance Dacy with Big Agile, and here's why this matters right now. Budgets are getting approved this quarter on AI productivity numbers that fall apart the second you press on them. I'm a data guy. I love metrics. But I keep meeting leaders who can tell something feels off about these numbers. They just don't have the language for it yet. This is all new to every one of us. So this week I want to give you that language as best I can.
The Green Dashboard That Isn't Measuring What You Think
Here's the pattern. A team adopts AI coding tools. Within a month, the dashboards light up: lines of code, accepted suggestions, story points per sprint. They're all going up and to the right. Leadership sees green. The VP tells the board AI is paying off. Everybody relaxes.
Then you go talk to the actual engineers, and the story gets fuzzy. Some of them feel like they're flying. Others are quietly spending their afternoons cleaning up code that the AI wrote that morning. And nobody can point to a single number that captures both of those things at once.
So that green dashboard isn't measuring whether AI made your team better. It's measuring how much AI-flavored activity is happening. Those are not the same thing. They were never the same thing. We just spent years confusing motion for progress, and AI cranked the motion up loud enough that the confusion is getting impossible to ignore. I'm used to this. I've been in agile development a long time, and it happens all the time. AI just tweaked the amplitude up.
The People Who Tried to Measure This Carefully
There's a research group called METR. In early 2025 they ran a proper randomized controlled trial with experienced open-source developers working on real repositories they knew well.
Before they started, those developers predicted AI would make them about 24% faster. After doing the work, they felt they'd been about 20% faster. But when METR measured the actual time, the AI-assisted tasks took about 19% longer.
Now, that slowdown isn't a law of nature. It's one snapshot, one setting, based on early-2025 tools. To me the durable finding isn't the slowdown. It's the gap in those numbers. People were sure they were faster while the clock said otherwise. And it wasn't just the developers. Expert forecasters in machine learning and economics guessed even bigger speedups, around 40%, and they were wrong too.
That's the real headline: smart, experienced people could not feel their own productivity accurately. That's a warning sign.
Here's where it gets interesting. METR went back in late 2025 to measure again with better tools, and they couldn't get a clean read. So they announced they're redesigning the whole experiment, and I was glad to see that. Why couldn't they get a clean read? Because AI adoption broke their control group. So many developers refused to work without AI that the comparison started to fall apart.
Think about what that means. The act of measuring got harder precisely because the thing you're measuring became normal. Their honest read today is that developers are probably sped up now, but the size of that speedup can't be measured cleanly. That's the grown-up answer.
What Actually Holds Up
So where does that leave you? With the research that does hold up. We talk about DORA all the time. It looks at thousands of professionals every year. In 2025, AI adoption finally lined up with higher throughput. Teams are shipping more. But that same report found AI adoption still lines up with more instability. More changes are going out, and more things are breaking downstream, sometimes we don't find out until it's way too late.
There's also code-quality tracking from a group called GitClear showing more churn and duplicated code in AI-heavy work. That's a useful signal, but it's specific to their method, so I won't stretch it into a universal law.
Put it all together and the picture is honest, not hopeless: faster output, shakier stability, and a real speedup we can't size yet.
The Gut Check: Three Questions Every AI Metric Must Survive
So what do we do about it? Stop trusting any single AI metric until it survives three questions. I call it the gut check, because we humans are the only ones who have a gut.
Question one: What decision does this metric actually change?
If lines-of-code-accepted goes up, what will you do differently? If the honest answer is "nothing, we just like to see it go up," that's not a metric. That's a decoration.
Question two: What would a high number here hide?
This is the one people skip. A high suggestions-accepted rate could mean your team is flying. It could also mean they're accepting code they don't fully understand, and the bill comes due in code review three weeks later, or in production nine months later. Remember the pattern from the research: the time you save writing code gets quietly re-spent auditing it. A high acceptance number can be hiding exactly that tax.
Question three: What confounder haven't you controlled for?
The METR story is the warning here. Their developers got more selective about which tasks they even attempted with AI. Your story points per sprint might be climbing because the work itself shifted, not because anyone got faster.
Run your AI metrics through those three. Some survive. Throughput survives when you pair it with stability: are you shipping more without breaking more? Rework survives: how much of last sprint's work was fixing the sprint before it? AI can quietly inflate that. And the best one, learning speed: how fast does your team turn a question into a validated answer? That's the actual job. Not typing speed. Not how much you pushed out.
Your One Experiment This Week
Here's your experiment, and it takes ten minutes. It's a little uncomfortable, so be okay with that.
Open whatever dashboard your leadership uses. Pick the one metric they trust the most, the number that shows up in the board deck. Write one sentence next to it: "This number is supposed to help us decide ______."
If you can finish that sentence with a real decision, great, you've got a real metric. Keep it. If you can't finish it, you've found a green chart that's decorating a bunch of slides, and that may cause danger later. Do this with one other person in the room, a tech lead or a product owner, so it's a conversation, not a solo audit.
One metric, one sentence, one honest verdict. That's the start of measuring what matters, and I think we've lost sight of that in the age of chasing efficiency.
So here's the idea to carry out of here: stop measuring how fast you type, and start measuring how fast you learn. And the honest answer about AI and velocity right now is that we don't know the exact magnitude. Be honest with yourself about that. That's not a weakness. To me, it's the most credible thing a leader can say. "I don't know" is a defensible position. A green chart nobody can explain is not.
We're all in this together. It's all new. I don't believe there are really any experts out there yet. We're all just doing the best we can with the day we have.
Keep Going
If this hit a nerve, the best next step is to put the gut check to work on your own dashboard this week. When you're ready to go deeper on measuring what actually matters, throughput paired with stability, rework, and learning speed, explore our upcoming classes and coaching. That's where we help leaders build metrics that survive the hard questions.