I see this question in a leadership review sometimes comes across like a bad joke.
"So what is our AI productivity number?"
Every head turns toward whoever owns the dashboard, and that person starts assembling an answer in real time out of whatever is handy: suggestions accepted, active licenses, lines generated, or worse, nothing.
A number gets produced. Everyone nods and moves on. And almost no one notices that the number measures how much AI is being used, not whether any of it made the product better. If you have felt that quiet gap between the number you reported and the thing you were actually asked, this piece is about closing it.
The good news is that measuring AI-assisted work honestly does not take a data science team. It takes a comparison and two signals that most dashboards skip. Here is how it works.
Usage equals impact, right?
The team is tracking AI usage, which most think tells you AI's impact. We can see the acceptance rate climbing, seats filling, and tokens flowing, so we know it is working. It feels like a measurement.
Unfortunately, usage and acceptance only really measure adoption, not effect. They tell you people are using the tool, which you already knew. To claim an effect, you need something almost no dashboard has: a comparison. Without a baseline or a holdout, "productivity is up" is a feeling with a chart attached, and the same amplifier dynamic that drives AI pilots to fail is at work here: the tool is fine; the measurement around it is the weak link.
Why "felt faster" is the least reliable signal you have
If you are tempted to just ask the team whether AI made them faster, hold that thought. Perception is the shakiest instrument in this business. In early 2025, the research group METR ran a careful trial with experienced developers on their own code, and the people doing the work felt about 20 percent faster, while the measured time actually ran longer in that snapshot.
Source: METR measurement update, February 2026 – https://metr.org/blog/2026-02-24-uplift-update/
Read that as a caution about measurement, not necessarily a verdict on the tools; that figure was one setting at one moment, and METR later had to redesign the experiment because adoption broke their control group. The durable lesson is simple and a little humbling. Smart people cannot accurately gauge their own productivity, so if your instrument is a vibe, your number is fiction.
Four ways to instrument AI-assisted work honestly
You do not need a lab. You need a defensible comparison and a couple of signals that tell the truth. Here is my practical version.
1. Sample and tag, then actually look
Aggregates hide everything interesting. Instead, tag a sample of recent work as AI-generated, AI-assisted, or human, then have an experienced engineer read ten of them and sort the review comments into real defects versus style nits. A small, honest sample beats a big, confident average because it tells you what kind of work your AI is actually producing.
2. Build a comparison you can defend
A clean before-and-after is hard to achieve once everyone adopts at once, which is exactly the trap that caused METR's control group to collapse. Where you can, create a comparison on purpose: a small holdout, or a staggered rollout where one team or one service starts a few weeks later. That gap gives you a rough control group without stopping anyone from working.
Where a holdout is not realistic, do the minimum viable version: freeze a baseline of your health metrics before a team ramps up AI, so later you are comparing against something real instead of against your memory of how things used to feel.
3. Track rework and review load, not acceptance
Acceptance rate measures adoption; rework and review load measure cost. Watch how much recently merged code gets rewritten within about two weeks, and watch comments per pull request and review wait time. GitClear's analysis of 211 million changed lines found churn: code reworked shortly after merge, roughly doubling since AI tools went mainstream. This is a method-specific signal from one dataset rather than a universal law, but it points exactly where you should be looking.
Source: GitClear AI Copilot Code Quality research, 2025 – https://www.gitclear.com/ai_assistant_code_quality_2025_research
This is the same pattern behind why delivery metrics start lying to you: output rises while the rework queue quietly grows, and the ratio of new work to rework is the honest number underneath the speedup.
4. Put it on a cadence, not a one-time audit
A single measurement is theater; a habit is instrumentation. Give it a short recurring slot: a flow review where you look at throughput alongside stability and rework, so a surprising number becomes a question you investigate rather than a verdict you react to. When a metric jumps, treat it the way we describe in reading an AI surprise as a signal about your system, then run the cheapest check that explains it.
For the quantitative anchor, DORA's 2025 report, drawing on nearly 5,000 professionals, is the safe ground: AI adoption now aligns with higher throughput and, at the same time, greater delivery instability. Speed and fragility rising together are precisely why you watch a pair of signals, never one alone.
Common traps that quietly ruin your read
The first trap is switching everyone on at once, which erases the baseline you would have needed to prove anything. If you can, stagger the rollout; if you cannot, at least record your health metrics before the ramp so you have something to compare against.
The second trap is trusting a self-report. "The team feels faster" is the single least reliable input you have, so pair any perception with one measured signal before you put it in a deck. The third trap is optimizing acceptance rate, because the moment you target it, people learn to accept more, and you have taught the org to game a number that never meant much. Aim your targets at outcomes and health, and let activity metrics stay diagnostic.
Try this next week
Add one label to your workflow. For the next two sprints, tag every pull request as AI-generated, AI-assisted, or human, and change nothing else about how you work. You are just making the invisible visible.
At the end, compare one thing across the three groups: how much of each got reworked within two weeks. That single comparison will tell you more about your real AI productivity than any acceptance-rate chart, and it costs you a dropdown, not a budget. The next step is to bring that comparison to a flow review and commit to one change, not seven.
If you want a structured way to build this measurement habit into a product role, our AI for Product Owners course works this exact muscle, less about the model and more about the product decisions that make AI gains real.