Somewhere in your organization, a team added a batch of new dashboards this quarter. I'd bet on it. Delivery health, a few flow metrics for the coaches, an AI-generated rollup, maybe a quality scorecard for the support team. My favorite is always the executive summary: the dashboard that rolls up the other four dashboards. At this point even the dashboards have a dashboard.
Now walk down the hall. There's usually a mounted monitor in a corner of IT ops that's been showing a session-timeout screen since March, and nobody has noticed. It is quietly the most honest metric in the building. Every one of the dashboards I just described got approved. Every one refreshes on a schedule. And when the thing that actually mattered went off the rails last month, all five of them watched it happen and none of them said a word.
I don't think anyone was ignoring the data. I assume good intentions. The data just had nowhere to go. That gap, between everything you can see and everything you actually decide, is what this is about. I call it the visibility paradox.
The paradox shows up in the numbers
Let me use real research here, the way I usually do. Last fall Digital.ai released its 18th annual State of Agile Report. One thing up front: Digital.ai sells enterprise delivery tooling, so this is a vendor survey of roughly 350 self-selected practitioners, mostly coaches and consultants inside very large enterprises. I'm going to treat it and grade it that way. But it still tells a story worth sitting with.
The infrastructure side looks strong. In that survey, 55% report complete visibility into what's being developed and delivered across the software lifecycle, and 64% say their Agile teams have visibility into the DevOps pipeline. For those people the tooling is in place, the pipelines are lighting up, and the charts exist. The data is flowing.
Then the same block of questions reports that 63% of organizations struggle to deliver reliable, high-quality software. The report calls that a 12-point jump from the prior edition. And here's the part I respect: the same survey also found a healthy share of respondents saying most of their applications ship on time with high quality. Digital.ai states plainly that both of those things cannot be fully true. Read that twice. When a measurement can point in two directions at once, it can't change a decision.
One scope check before anyone overreacts. This is self-reported, self-selected, and vendor-published, with a sample around 350. A 12-point swing in a sample that size deserves interrogation before it deserves a reaction. Hold that thought, because I want to come back to it.
This is a definition problem, not a data problem
None of this is the data's fault. I'm not going to claim dashboards cause bad outcomes, and nobody has proven they do. Dashboards are just the vehicle. The problem is that most of us never defined what a given metric is for. It showed up on a checklist, someone built it, and it stayed. We're all so busy keeping up that we never stop to ask the question.
I watched this play out last quarter with a leadership group. Two dashboards, both green, describing the same system that everyone in the room agreed was on fire. So I stopped and asked: it's green, but it's on fire, and has anyone paused the meeting to reconcile that? Nobody had, because reconciling it is work, and the agenda has to keep moving. Maybe that's the tell. Maybe we should stop calling these things meetings and start treating them as working sessions, where reconciling a contradiction is the point rather than an interruption.
Transparency, inspection, adaptation, in that order
I spend a lot of time in the Scrum Guide, which builds its whole model on empirical process control: transparency, inspection, and adaptation. The Guide is blunt about the order. Transparency enables inspection. Inspection without transparency is, in its words, misleading and wasteful. And inspection that never leads to adaptation is pointless.
If you run Scrum, you already run this chain in miniature every sprint. The review makes the increment transparent so people can inspect it, and the inspection only mattered if the backlog or the increment changed because of what everyone saw. The chain has to end in a change, or it was theater. That idea doesn't belong only to Scrum, by the way. Every agile process demands the same empirical loop. Scrum is just more adamant about it than most.
Now let me push one step past the Guide, and I'll own this part as mine. A metric that changes no decision is overhead. And overhead that looks like insight is worse than overhead that looks like overhead, because nobody ever schedules its removal. It just persists.
We audit spend. We audit headcount. We audit software licenses nobody uses. The dashboard wall gets a pass, because it feels like the opposite of waste. It looks like diligence. It photographs well in the ops review. It reassures executives that there's rigor in the building. And it can sit there for years without anyone asking whether a single decision ever changed because of it. I'm guilty of this too. This isn't condemnation. It's a call to step back and breathe.
Most of what a metric does is noise
There's an older idea that helps here, from Mark Graban's book Measures of Success. Graban's argument is that most movement in a metric is noise, not signal. React to the noise and you burn your improvement capacity chasing ghosts. He built that method for metrics tracked over time, not for comparing two surveys, so I won't stretch it too far. But the discipline underneath it travels: interrogate a move before you react to it.
Remember that 12-point swing I asked you to hold? That's exactly the kind of move to interrogate rather than react to. And the same rule applies to every chart on your wall. Graban did a lot of this work in hospitals, where the data is dense and lives are at stake. When a number moves in a hospital, people act. Most of us don't have that sensitivity about delivery data. So before you ask what a number did this week, ask a harder question: has this metric ever changed what you do?
Why this is coming to a head now
This isn't new, but AI is making it worse, because AI changed the economics of producing visibility. A dashboard used to cost something. Somebody had to build it, wire up the data warehouse, model the star schemas, and keep it alive. I've been that somebody. That cost was a filter. Weak metrics died of neglect because nobody wanted to spend the capacity to build them.
Now a rollup is one prompt away. The marginal cost of another chart is close to zero, and the filter is gone. So the overhead grows without limit. Ask anyone who has watched an AI rollup of a rollup land in their inbox, unrequested, condensing reports they already weren't reading.
Delivery research has something to say about the tools underneath this. DORA analyzed its 2024 survey data and estimated that a 25% increase in AI adoption was associated with roughly 1.5% lower delivery throughput and about 7% lower delivery stability. Those are cross-sectional estimates with wide uncertainty, not proof of cause. The picture shifted in 2025, and that shift matters: the throughput association flipped from negative to positive, while the instability association held. So that's two consecutive years in which AI adoption travels with lower delivery stability, even as everyone reports feeling more productive.
DORA's 2026 ROI report frames early adoption as a J-curve, a temporary dip it calls the tuition cost of transformation, and it tells leaders to budget for what it names the instability tax. Some of the time AI saves gets spent handling the instability that comes with it. Put the two halves together. AI adoption travels with the very instability your reports are supposed to catch, and that same technology is now producing the reports. More charts, produced faster, about a system that is harder to keep stable. That's the visibility paradox. It isn't a claim that AI causes failure. It's a claim that the cost structure changed and our metric habits didn't.
The dashboard-to-decision audit
Here's the tool. One record per active metric, five lines each. Our Wednesday guide has the full downloadable version, but the five lines are the whole idea.
1. The metric. Name it. One metric or dashboard per record, no bundling.
2. Who reads it. Not who receives it. Who actually looks at it. Those are two different lists, and the gap between them is usually the whole story. Ask the named readers directly and this list gets short fast.
3. Last decision it changed, and when. Not informed. Not supported. Changed. If the metric had said something different, would you have done something different? If you can't name a date, that's your answer.
4. If it disappeared tonight, what breaks? Be specific. "People would miss it" is not a difference. A decision made later, a risk caught slower, a conversation that stops happening: those are differences. Think of a smoke detector. Nobody acts on a smoke detector they've never heard, but if it were gone, the one thing built to warn you of a fire is gone, and you find out too late.
5. The verdict. Keep it, cut it, or merge it into something that earns its place. One rule to keep everyone honest: a metric that hasn't changed a decision in 30 days is a candidate for removal. Not automatic removal, a candidate with a case to answer. Only your organization can answer it, but push hard.
Some metrics are like rent. Compliance reporting, board rollups, anything a contract requires. Put those on their own list so they stop polluting the audit, and move on.
Run it on your executive summary
Go back to the dashboard I opened with, the executive summary that rolls up the other four. Run the five lines. Who reads it? It gets presented monthly, and presented is not the same as read. Last decision it changed? Can anyone in the room name one this quarter? If it disappeared, the four underlying dashboards still exist, so nothing is lost except the assembly work. Then you cut it.
Notice what that buys you. Somebody stops assembling a report nobody acts on. That's real waste removed. Then the four remaining dashboards each face the same five questions, one at a time. The team relearns something that's rare in corporate life: a metric on the wall is a claim that it changes decisions, and claims get tested.
This is leadership work, not team housekeeping
In that same Digital.ai survey, only 15% of respondents said business and executive leaders are actively involved in sustaining and shaping agile practices. That was a single-select question answered by practitioners rating their own executives, so read it as a floor on involvement, not proof that leaders don't care. I think you do care. The problem is that people don't believe you do.
That's exactly why this audit is yours to run. When a leader cuts a metric in public and explains the reasoning, the whole organization learns what visibility is for, and that teaches more than any tooling rollout you'll fund this year. It also gives every team permission to stop feeding the charts they privately knew were dead. If a dead metric can survive at the executive level, imagine how many are surviving below it.
What visibility is actually for
Transparency, inspection, adaptation. The Scrum Guide put them in that order because the chain is supposed to end in a change. A dashboard that feeds no inspection is decoration. An inspection that feeds no adaptation is expensive decoration, because attention is the one budget line you cannot buy more of, and it drains fast. Visibility is not insight, and insight is not a decision.
So here's my challenge for the week. Run one metric through the five lines and name the dashboard you keep but never actually act on. That's your first audit record, and acknowledging the problem is where every real fix starts. If you want the full downloadable audit and want to keep working through problems like this with us, take a look at the classes and workshops we run at Big Agile. It's the kind of thing we work on together.