Two weeks after the team switched on the AI coding assistant, the review comments per pull request roughly doubled, and the first reaction in the room was to blame the tool.
That instinct is understandable, and it is also the wrong read. That spike was not noise. It was the most honest feedback the team’s system had produced in a year.
Most leaders treat a surprising AI result as a verdict on the tool. A good surprise means the tool is great; a bad one means the tool is broken. There is a more useful way to read it.
The comfortable lie about a surprising AI result
The comfortable lie is that the surprise is about the AI. The uncomfortable reality is that the model is roughly the same for everyone using it, so the surprise is local to you. That makes it a reading on your system, not a grade on the vendor.
This is the amplifier idea turned into a daily habit. DORA's 2025 research found that AI tends to increase both throughput and instability, and that the results stem mostly from the system around the tool. So when AI hands you something you did not expect, the interesting question is what it just revealed about your system.
A worked example: reading the code-review spike
Back to that doubling in review comments. The spike by itself does not tell you what is wrong; it just tells you something changed. Your job is to turn one surprising number into a short list of system explanations, then run the cheapest check that separates them.
Here are the four explanations I would put on the table, and the cheap check for each:
The AI output quality is low for your stack. Check: pull ten AI-assisted pull requests and sort the comments into real defects versus style nits. If most are real defects, the output is the issue. If most are nits, look elsewhere.
Review has just become your constraint. Check: Did the review wait time grow as the number of comments grew? If pull requests now sit waiting while everything upstream has gotten faster, AI did not speed up delivery; it moved the bottleneck onto your reviewers. That pattern is the quiet center of why your delivery metrics can lie to you.
Loose standards just became visible. Check: Were these comment types around before, or only rarer? AI may simply be producing more of what your code always tolerated, and the volume finally made it impossible to ignore. Many of those comments map to known patterns, which we cataloged in the 10 AI code anti-patterns.
Volume simply overwhelmed the reviewers. Check: count pull requests per reviewer per week, before and after. If each reviewer is now carrying twice the load at the same quality bar, the comments are a capacity signal, not a quality one.
Notice what happened. One surprising number became four testable explanations, each with a different fix. The spike did not tell you what to do; the checks did. That is the whole move, and it works on any AI result that makes you raise an eyebrow.
Good surprises are signals too
This is not only about bad news. Say a batch of AI-assisted code sailed through review with almost no comments. The easy story is that the tool is excellent, and maybe it is.
It could also mean the review got rubber-stamped because the code looked clean and polished, which is exactly when reviewers stop reading closely. Run the same discipline: sample a few of those clean pull requests and have a senior engineer actually dig in. A good surprise you have never encountered is just a risk you have not yet met.
A copy-ready prompt to read your own surprise
Here is a prompt I use to turn a surprising result into candidate explanations and cheap checks. Paste it into your assistant, fill in the four lines at the top, and keep the constraints; they are what stop it from confidently blaming the tool.
You are a delivery coach. Help me read a surprising AI-assisted result as a signal about my team's system, not as a grade on the tool. What happened: - The surprise: [the spike, drop, or anomaly in plain words] - When it started: [date or sprint; "not sure" is allowed] - What else changed around then: [new tool, policy, deadline, people out, reorg; put "nothing I know of" if truly nothing] - What I can measure: [paste any numbers: review comments per PR, change-failure rate, rework %, lead time. If none, write "no metrics yet"] Do this: 1. List the 3 to 5 most likely SYSTEM explanations, ranked by likelihood given what I told you. For each, say whether it is a strength being amplified or a weakness being amplified. 2. For each, give ONE cheap check that would confirm or kill it: a query to run, a question to ask the team, or a number to pull. 3. Name the single most important question to put in front of my team this week. 4. If I gave you too little, say which ONE data point would most change your answer. Do not guess past the evidence. Constraints: - Do not assume the tool is the cause or the cure. - If a "good" surprise could hide a risk (for example, reviews getting rubber-stamped), flag it. - Keep the whole response under 250 words.
Filled in with a messy, real input, it looks like this:
- The surprise: review comments per PR roughly doubled - When it started: not sure, maybe when we turned on Copilot - What else changed around then: two reviewers were on PTO - What I can measure: no metrics yet, just a gut sense Likely output: 1. Reviewer capacity dropped (weakness amplified). Two reviewers out means the rest carry more, so comments per PR can rise from load, not quality. Check: PRs per active reviewer, this month vs last. 2. AI output quality is low for your stack (weakness amplified). Check: tag 10 recent AI PRs, sort comments into real defects vs style nits. 3. Standards were always loose (weakness made visible). Check: did these comment types exist before, just rarer? Most important question this week: "Are the new comments real defects, or the same nits we always let slide?" One data point that would change this most: comments per PR for AI vs non-AI PRs over the last 30 days.
One refinement note from using this in the wild. If you leave the "what else changed" line blank, the model will reliably anchor on the AI tool as the cause and miss the reorg or the deadline crunch that actually drove the spike. Fill that line even with a guess, because the context is what keeps the read honest.
Common traps when reading the signal
The first trap is acting on the surprise before you check it. A spike feels like a verdict, so teams rip out the tool or double down on it without running a single check. Slow down for one query; it is cheaper than a wrong decision.
The second trap is to read the signal once and move on. These readings are most useful as a habit, not an incident response, which is why a standing cadence beats a one-time scramble; that is the idea behind the flow review in a practical learning operating model. Build the read into a rhythm, and the surprises stop ambushing you.
Try this next week
Pick one AI result from the last month that surprised you, good or bad. The velocity jump, the review spike, the suspiciously clean pull request, whatever made you look twice. Just one.
Run it through the prompt above, or do it by hand: write three or four system explanations and one cheap check for each. Then bring that short list to your next working session and ask the team which explanation fits. You are converting a reaction into evidence, and that is the difference between managing AI and being surprised by it.
The next step is to run one of those checks before the meeting so you walk in with a number rather than a hunch. If you want a structured way to build this reading habit for product roles, our AI for Product Owners course works on this skill, using real artifacts instead of toy examples.