Activity Is Not ROI: Why Lines of Code Can't Prove Your AI Investment Paid Off

I'd like you to picture a board meeting. It's the end of the quarter. Sounds familiar.

The VP of engineering is up, and she's glowing. The company rolled out AI coding tools this year (you've heard that one before), and this is the slide that's supposed to show they paid off. One big green number sits right in the middle: lines of code shipped, up 30%.

Then the CFO, who has said almost nothing all meeting, leans forward with his glasses down and asks one question.

"What would that number have looked like if the tools had actually made things worse?"

Everybody looks around. Nobody answers. Awkward silence. Somebody finally says, "Why don't we take that offline? We'll get back to you on that."

Some of you have sat through that version of the board meeting. So let's go back and answer the CFO, who I like to call the chief ROI officer. If the tools had made things worse, what happens to lines of code shipped?

I think it still goes up. It might even go up faster, because fixing defects is code too. It's a recursive problem. A number built to go up has no way to show you a loss. And if it can't show a loss, what is it telling the board about the return on that investment?

The Question Finance Is Really Asking

That board meeting happens to a lot of us. The VP's teams probably are producing more code. I won't dispute that; I see it happening all the time. She's also under enormous pressure to show something for that spend, and I don't believe that pressure lives only in her head.

When finance asks about ROI, they're really asking one question: what did we get back for what we spent? It's a great question, and it's the one they center on for the whole business.

DORA's ROI of AI-assisted Software Development report says board members and finance executives want clear justification for this spending. They've been burned on technology too many times. If we owned the business, we'd want the same thing. The report also says leaders have to translate technical metrics into financial outcomes.

That's exactly the job our VP was trying to do. I just think she reached for the wrong number, like many of us do. I've been there too.

Speed Is Easy to Report. Return Is Hard to Prove.

The numbers I've been reading this year look a lot like that board meeting. Plenty of great-looking numbers that give an illusion.

Atlassian, the maker of Jira, published its State of Teams 2026 research, which surveyed 173 Fortune 1000 executives in January and February of this year. 89% of those executives say AI increases speed. Only 6% are sure they have clear examples of organization-wide AI ROI.

That second number has a high bar built into it. It's about being sure, which is different from whether there was any return at all. Still, it's a striking gap.

A second number points the same way. According to Gartner, only 35% of software engineering leaders report significant ROI from AI in the software development lifecycle. Gartner doesn't publish a sample, date, or method alongside that figure, so read it as directional. They consult with clients, so it's probably data they have to be careful with. I still find it interesting.

To me, both of these describe the meeting I just made up. Speed is easy to report. Return is the question hardly anyone in the room can answer.

So why couldn't our VP answer it? I don't think she's bad at her job. She's doing the best she can. She brought a measure of activity to a question about return. Those two things are radically different, and we mix them up all the time.

The Team Dashboard and the ROI Slide Answer Different Questions

A quick word about last week's episode, because I don't want these two to sound like they disagree. Last week I said to put pull requests waiting on a first review right beside pull requests opened. I still stand by that pair. That's your team's operating view. It tells the team where work is piling up, day to day, in their own system.

This week's claim is about something else: what goes to the board, the executives, or the CFO as the return on the tooling. That's a different audience asking a different question.

The team wants to know where work gets stuck. We should keep asking that. The board wants to know whether the investment paid off, and they should be asking that all the time. A volume number can live on the team's dashboard just fine, as one signal among several. The trouble starts when it becomes the ROI claim.

What a 2:30 a.m. Heat Map Taught Me About Activity Metrics

So what's actually wrong with lines of code? As one signal on a team dashboard, not much. It depends on how you use it. Let me tell you a story.

Back in 2008, I was a Scrum Master for the very first time, on the very first team I had ever helped. The CEO directed me to make sure every team started its Daily Scrum at 9 a.m., because he wanted every developer in the office and ready to work like the rest of the organization.

I didn't know much back then, but that felt unsettling based on the training I'd had. So I reached out to my mentors. They pointed me right back to the Scrum Guide and the purpose of the Daily Scrum. The time and place should be consistent, and the team sets them. If they want to hold it in the afternoon, that's up to the team.

Then I thought harder. What did he really know about developer productivity? Was 9 to 5 actually the productive window for every engineer?

"What were you doing at 2:30 this morning?"

I'm a data guy by trade. We had a fancy new tool back in '08 that could show check-ins, merges, and deploys as a heat map, so I started pulling the data with SQL queries. At the time, check-ins and merges were a pretty good indicator that developers were finishing work to ship as a usable increment.

I scheduled a meeting with the CEO early one morning and opened with one question: "What were you doing at 2:30 this morning?" I probably had a smirk on my face. He laughed and said, "Sleeping."

Then I pulled out my report. Most of the work on his product was happening between about 11:30 p.m. and 3:30 a.m. Many of our developers seemed to do their best thinking at night, with few distractions, on their hardest problems. They still needed to come together and collaborate. So they might arrive at 10, leave at 4 or 5, and put in a ton of hours while everybody else was asleep.

Force them into eight hours between 9 and 5, and I believe you lose all that discretionary, self-managed time the team was spending building a great product.

I knew the least I've ever known, and it was one of my best performances. When you work with leaders, be brief, be bright, and be gone. The message stuck, and we never had to apply that rule.

I see this in organizations all the time. Leaders focus on activity because that's how business school trained them. Leaders are trained to fill vacuums. That's part of why they're good at their jobs. In agile, our job is to help the team solve its problems before a vacuum forms and a manager steps in to fill it.

Activity Metrics Need Counterweights

I wish I'd had this research back in '08. In 2021, Nicole Forsgren and colleagues published The SPACE of Developer Productivity. SPACE stands for satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow.

It warns that activity counts like pull requests and commits should never be used on their own to judge productivity, and it calls out lines of code as a problematic metric. Same mistake as assuming everyone in the office 9 to 5 creates efficiency.

Notice what SPACE does with activity, though. It doesn't throw it out. It adds counterweights. Here are your activity metrics; now put metrics in tension with them. That's what I love about it.

That work is foundational. It's from 2021, before most of today's AI coding tools, and it's about productivity rather than ROI. I think the logic carries straight over, though, and it gives us a way to talk to the people asking for ROI.

The counterweights are what make a number like lines of code safe to look at. Take them away, put the number alone on the ROI slide, and something changes.

When a Measure Becomes a Target

Once lines of code is the claim, it's also the target. There's an old idea for what happens next: Goodhart's Law. The version most people quote comes from anthropologist Marilyn Strathern, writing in 1997: "When a measure becomes a target, it ceases to be a good measure."

My version is shorter. A number that can be gamed will be gamed the week it goes on a slide.

With lines of code, nobody even has to cheat. We all know how to create more lines of code. A number that only goes up, and looks good no matter what's happening underneath it, isn't a good number to put in front of a board.

Agile goes further and tells us not to write code we don't need. Remember principle number 10 from the Agile Manifesto? "Simplicity, the art of maximizing the amount of work not done, is essential." Lines of code was never the goal. The goal is to deliver high value early and often with the least cost, effort, and friction.

Every Metric Can Be Gamed. Pair Them So the Gaming Shows.

This made me stop and look hard at the metrics I'm about to recommend. Can these be gamed too? They can. That's the problem.

DORA's software delivery metrics are a great source of information, and they come with their own warning. DORA's metrics guide says that ignoring Goodhart's Law and turning these metrics into blanket targets makes it more likely teams will try to game them. So why do we keep doing it?

I'm not going to tell you there's a metric out there that can't be gamed. I don't know of one. If you find one, I'd like to hear about it. What you can do is make gaming show up. You pair numbers that pull against one another: counterweights.

Fixing the Slide: Lead Time and Change Failure Rate

Let's go back to that board meeting and fix the VP's slide. Take out lines of code shipped (or whatever volume number you're using) and put in two numbers instead: change lead time and change failure rate.

Change lead time is how long a change takes from the moment it's committed to version control to the moment it's deployed to production. We've talked about it in a lot of other episodes.

Change failure rate is the share of your deployments that need someone to step in right away. Think rollbacks and hotfixes. You probably don't have those. I have them all the time.

Both come from DORA's software delivery metrics, which DORA sorts into two families: throughput and instability. Lead time is a throughput metric. Change failure rate is an instability metric.

One more instability metric worth knowing: rework rate

Try saying "rework rate" five times. Deployment rework rate is the share of deployments that weren't planned and happened because of a problem in production. It's about unplanned deployments to fix something that broke, which I'd call escaped defects. Normal refactoring doesn't count.

That's why I tell leaders to track change failure rate and rework rate together, so faster delivery isn't paid for in the currency of stability. In agile, stability is exactly what we want.

Ask the CFO's question of the new slide

What would these look like if the tools had made things worse? If the damage is in quality, change failure rate climbs, and you'd see it on the slide. That's the loss the old number couldn't show.

What gaming the pair looks like

Now think about what happens if somebody tries to game the pair.

Push changes out faster than anybody can check them, and lead time looks awesome. Change failure rate is where I'd look for the damage. Sometimes we get lucky. Most of the time we don't.

Play it safe instead. Ship less, go carefully, and change failure rate looks awesome. Lead time is where I'd look for the problem.

Neither number is safe from gaming by itself. Together, gaming one leaves residue on the other, and you can see it.

Two Cautions Before This Goes to the Board

Scope: one service at a time

These metrics work best applied to one application or service at a time, and DORA says the same. Many services go into building a product. Blending them across teams or a whole organization can be problematic, like a lot of metrics.

Show them per service, as a trend over time. If your board insists on one company-wide number, say out loud that rolling these up tends to hide what each service is doing. Those services operate independently to make a collective product, and a lot of people don't understand that.

Money: don't claim the revenue

I wouldn't stand up and say faster lead time made the company money. Tying more deployments to revenue lift is speculative, and a CFO will likely challenge it in a way you can't defend. I love that the DORA ROI report names the exact person from our opening: the CFO.

The report points to engineering efficiency and infrastructure savings as the safer claim. It also doesn't want the value story limited to hard cost savings.

Here's how I'd lay out the chain. Better delivery metrics show up first as productivity, developer experience, and user experience. Those are leading indicators. Revenue and cost savings show up later, as lagging indicators. That's a model, and every link has to work for the whole chain to hold.

So my advice is don't try to measure every step from delivery metrics to money. Make your slide's claim in the early part of that chain: delivery got faster, steadier, or both, service by service, and here's the trend.

The cost side is something a CFO already knows how to count. Every unplanned deployment to fix production is work the team paid for twice. DORA's ROI report makes the same point: some of the biggest returns come from cutting unnecessary rework and reclaiming that capacity. Show that number. I guarantee the CFO can calculate what that rework cost from the team's size and salaries. If they can't, we can help them.

Try This Next Week

Here's a simple experiment. Pull up the last ROI slide your team sent upstairs. Find every number on it that typically only goes up, and ask the CFO's question of each one: if the tools had made things worse, would this number show it?

If the answer is no, don't delete it yet. Move it to a team dashboard, where it sits as one signal among others. Nothing wrong with that, as long as people know how to use it.

Then put change failure rate and lead time on the slide for one service, as a trend. One service to start.

To help, we're publishing a free guide here on the blog this Wednesday. It takes each output number a team reports as return and pairs it with a flow or stability metric that can carry the ROI claim up to the boardroom. It also flags which of your numbers are easiest to game. It won't tell you to stop tracking anything. It moves numbers off the ROI slide you send upstairs, while they stay on your dashboard as signals.

If It Can't Show a Loss, It Can't Show a Return

One more trip back to that board meeting. The CFO asked what lines of code shipped would have looked like if the AI coding tools had made things worse. The answer was about the same, maybe higher.

That's why the room went quiet. Nobody was hiding anything. The number just couldn't say. And once that happens, people start getting suspicious of every metric on the slide.

So remember this about your metrics: if a number can't show you a loss, it can't show you a return. Bring the boardroom a pair that can show both.

There are no real experts in AI yet. The more we collaborate and share, the better we'll all get. If you want help building metrics your leadership can trust, or you want your coaches and Scrum Masters ready to have this conversation with executives, take a look at our upcoming classes. I'd love to help.