Introduction: The Problem with Traditional Agile Metrics
If you're an agile coach, what metrics are you capturing? Let me guess. Is it velocity? Very common metric. We'll find if your velocity is soaring, however, and your customers are still grumpy, are we really measuring progress or just a whole lot of motion?
Hi, my name is Lance Dacy, also known as Big Agile. My goal is to help people who find themselves in a position of trying to coach organizations on effective change and it's hard work, so let's do it together.
And I got passionate today thinking about how many agile coaches are trying to track metrics valiantly. They have all these lists of metrics they're trying to meet with leadership, but they just aren't landing the results or worse yet, they're getting relegated to a team level position. In fact, metrics can actually be a scary word for all of us involved. And I typically like to say I'm not afraid of metrics.
I love metrics, I love data. What I am afraid of is how they're being used. So what I'd like to do today is kind of reset the dashboard. I'm not saying your metrics are bad, but let's introduce maybe something you haven't been aware of or known about or known how to use them in your day-to-day process.
Introducing Process Behavior Charts: Separating Signal from Noise
What I'd like to talk about today are process behavior charts. What that's going to allow us to do is try to separate the signal of how our teams are performing against all of the noise that we can get gravitated down into when we're presenting metrics. So I'll show you kind of the three basic metrics that I'm showing on my blogs right now are cycle time, lead time, and defects. Those are kind of like the top three metrics that I like to use, and to me, they tell the real story of how we're progressing and flowing work through the system.
So what I'd like to do today is introduce process behavior charts, XMR charts, and then give you maybe a 15 minute agenda that you could use to run with your leaders and managers. My hope is that you stop reacting to noise and steer the signal using these process behavior charts. And so that's what I want to talk about today.
The Velocity Chart Problem
Now let's talk about the problem of a velocity chart. So I've got just a standard velocity chart here. It's pretty boring. Let's say it's story points over time. And basically you can have a velocity chart, a burn down chart, ritual counts, whatever it is that you're showing just as kind of boring. So we see here how much the team actually committed to delivering versus what they actually delivered. And people see this and they're like, okay, well what's that really telling us? I see an upward trend and then I see a downward trend and it's really hard to deduce any kind of signal from this.
Understanding XMR Charts and Natural Process Limits
So what I'd like to introduce is what's known as an XMR chart. They're called process behavior charts. And they actually originated from statistical processes back in the 1920s, I'd say, sure, back in the 1920s was a statistician. And later what happened is Demming and Wheeler kind of took these to the next level in business practices in the 1940s and really started to elaborate on the idea of a behavior chart. And they're often referred to as an XMR chart.
And so what happened is we're trying to filter out the noisy signals or the noisy patterns from actual signals. And what we do is we end up establishing what's known as natural process limits. So we have an upper natural process limit and a lower natural process limit. And those are actually calculated empirically from the data of your velocity or your cycle time or whatever metric that you're trying to grab.
So signals become a lot more visible when that data point is breaching the upper or lower natural process limits. And so what we want to see here as related to the, let's say this is a velocity chart as you have the observed data and you see the dotted red lines on the upper and lower natural process limits. And what we're going to talk about are rules that help us guard against just overreacting to data points on a graph.
And you can see plainly there in the middle that we also have the average, which is important, but averages are usually a biased estimator and can be highly influenced by outliers, which could be good, could be bad depending on what you're using these charts for.
The Three Essential Rules for XMR Charts
What I typically like to do is follow a few rules when we're talking about XMR charts. This one's a little bit better indicator of where those upper and natural or upper and lower natural limits fall and they've been shaded.
Anything in the red is digressing down. And then you've got the green as our natural process limits. And so this chart has been shaded to help us kind of see those limits a little bit better, but you don't have to start with that. What we can do to get started is basically try to filter out all of the noisy data points and really teach leaders how to react to the signals, not the noise in the data.
So we're going to follow three rules when we do this.
Rule Number One: Points Outside Limits
And the first rule is any point that is going to be outside of our limits. So here's rule number one that we're going to look at in our charts we have our regular noisy data, but you see two of these points that are outside of our upper natural processing limit is kind of breaching rule number one, which is something that we would want to try to look at.
Now they're just two single points here, but they have occurred pretty close back to back together to one another. So it's probably something we wouldn't look at. But anyway, that is rule number one is we're going to target any point that is outside of our upper and lower natural processing limit.
Rule Number Two: Run of Eight
Rule number two is what we call a run of eight. And what we're looking at here is if we see any of our data points that are following a natural signal there of eight or more successive values on the same side of the average line. So they really haven't violated our upper and lower control limits yet, but we do see eight points really close together consecutively on the same side of our average line.
That's going to be something that we probably want to look at with our leaders and we might actually say we're establishing a new baseline based on that. So whatever has happened in the system, that might be our new baseline, which we would need to go update our upper and lower process signals for that as well.
Rule Number Three: Runs Near the Limits
The last one that we would look at here would be rule number three and if we have any runs that are near the limits. So if you have some runs that are going together, maybe four runs or so that are near those upper or lower natural processing limits, those would be something that we would probably want to look at and investigate a little bit further. And it's not that you have to have the answer to all of these either. This is just creating some rules around how to display these metrics.
The Three Key Metrics to Track
And so I like to use cycle time, lead time defects, we call 'em escape defects, the ones that we didn't catch. If we catch defects, let's say inside of a sprint and we fix 'em, well that's a good thing, but what I'm really trying to track, well it's not a good thing, but it's better than if we don't find defects and our customers find them.
So you got to be careful with what you're tracking with defects. But I like to use those three metrics as a start for our XMR limits and following those three rules and start teaching our leaders how to do that.
The 15-Minute Weekly Routine: A Practical Implementation Guide
So what I'm going to challenge you to do is starting tomorrow, maybe figure out how you can capture that data or if you already have it, start putting some meetings on the agenda with your leaders, managers, maybe even some representative of the teams, and talk about how we could do a, let's say a 15 minute weekly routine where we take one action only where there is a signal and we start getting people desensitized to the noise.
Part 1: Prep the Room (0-6 minutes)
And so what I like to do is the first part of the agenda is prep the room. So we're going to open our three charts that we've decided on or whatever you come up with, which is cycle time, lead time, and I like to do escape defects.
And you're going to make sure that those charts show the average. So going back to our first graph here, make sure that they are showing the average and a center line, I guess is what we would say. And then the limits. And we have spreadsheets that we can help you use to figure those things out.
But for each one of those signals, when you see those on the chart, you're going to ask, are there any points outside the limits? Remember that's rule number one. Are there eight or more points on either side of those average signals? And that could indicate a shift in the baseline or the other rule, six or more consecutive increases or decreases, we call that trend or drift. And are there moving ranges that are spiking that could be rising volatility, likely instability in the system. And if you don't see any signals at all, you say no signal here, let's move on.
So you start getting people used to seeing the up and down variance of whatever metric you're tracking, but we're trying to put a container around it with the noise.
Part 2: Address Special Causes (6-10 minutes)
Now the next part of the event is maybe six to 10 minutes into the session we're going to talk about the special causes. So we're going to name the event, contain it, and add a prevention check. And you'll make notes on these in whatever tool that you decide to track these metrics. And if you see a shift, you're going to confirm the change, set a new baseline, capture what changed, and go alter the data.
If you see a trend that you don't like, pick an experiment to stop it. We're going to time box it, put a small experiment on it and write an action about it. So per metric at most, don't be trying to do a lot of shifts inside of the metrics, but per one metric at most, try to account for owners and due dates of the experimentation and the time box and then spend a little bit of time towards the end of the meeting, 10 to 12 minutes in and discuss whether the change we tried last time, change the chart in any way that makes sense.
If not, retire it and try a smaller cleaner experiment. My guidance to use do not stack multiple actions because you might be causing issues that are unrelated to one other even though they may look like it. So just like we teach a team in a retrospective, don't try to change everything all at one time.
Part 3: Summarize and Close (12-15 minutes)
And then the last 12 to 15 minutes, we want to summarize and close. And so we're going to recap in one minute, what were the signals we have seen, what are the actions, A, B, C, whatever you came up with. And then set the next check-in date, which should just probably be a weekly recurring meeting. And then we want to log a short note in our event section so that in the future, any new people or key players that were absent, they can see what was discussed and what was changed.
Defining Roles and Responsibilities
So the last thing I want to talk about with that 15 minute agenda are the roles. So we have our leaders here, they're the ones that are going to keep time, make sure that the meeting happens on time and ask for signals or noise. We're going to train them to start asking for those things. And then what were some of the actions that we took and debrief on some of that.
As a scrum master or an agile coach, you're going to be the one that gathers the data, opens the charts, explain signals in plain language to everyone, and record action and events almost like a secretary of the event.
And then we have a team representative or maybe even the product owner would be there, is going to supply context and confirm the feasibility of the action that the leaders want to see addressed. So again, we're not pushing leaders out of our self-managing team, we're putting boundaries on the team, letting 'em inside the team decide how best to manage their work.
But the leaders have some control of being able to see the signals and ask the team to address those to mitigate risk.
Ground Rules for Success
Now the ground rules act on signals. Learn from the noise. If there's no signal, don't really change anything unless we're trying to make experiments in the team to get better. But the leaders don't change anything, just leave everything as it is and we want one action per metric maximum. Or you'll not know what caused the change.
And so last thing I would say is never use our process behavior charts to police people. These are for learning, they're for system decisions. They're not for holding people and individuals accountable to these metrics. And that's one of the biggest problems we see with leaders and managers.
Understanding What Signals Really Mean
So what signals really mean are special cause. Anything that is beyond our upper and lower natural processing limit, that's usually an unusual event. Let's investigate the event, not the people.
A shift in eight plus points on one side of the average. That is the system has changed. Set a new best baseline and update the next data set for expectations.
Any trend lines you see six plus six minus up or down, whatever it is that is a slow drift. Inspect what we have in the way of upstream policies, handoffs, capacity, depending on the metric that you're evaluating.
And then if you see a high moving range, that's a lot of volatility, look for unstable inputs, batching decision latency within the process flows.
Conclusion: Moving from Noise to Signal
So my goal here is to just give you some different ways to think about helping leaders have visibility into the system in which they are largely responsible for as well, and getting them desensitized to looking at the volatility of the noise of the data and start reacting more to the signals.
So if you would like to learn more about this, sign up for our Big Agile newsletter. We talked about this specific topic and it's only available to our newsletter subscribers through our email. So be sure you do that.
And give us a like or subscribe on this video. We're really trying to find another medium to bring you more next gen level agility concepts so that it can help you shape or reshape how your team is performing.
Ready to Transform Your Agile Coaching Practice?
If you're ready to move beyond traditional metrics and start implementing process behavior charts in your organization, Big Agile offers comprehensive classes designed to help agile coaches like you master these advanced techniques. Our expert-led sessions will give you the practical tools and frameworks you need to separate signal from noise and drive real organizational change. Don't let your coaching efforts get lost in the data – explore our classes today and start making the impact you know you're capable of.