Recently I got a request from my leadership:  “We’re working hard to improve and change the organization, what metrics are out there to show us how we’re doing?”

This isn’t the first time I’ve heard this question, and I’m sure it won’t be the last.  And let’s be clear what leadership is usually talking about here – quantitative measurement, or measurement against so-called “hard data”.  I have mixed feelings about metrics and if you follow any of the Scrum or Agile Communities online you’ll know this is a hot topic.

On the one hand I feel it is important to measure and show progress in hard numbers.  On the other hand, experience has taught me that qualitative tools are at least as important, if not more important, than ‘hard data’. I’m talking about measurements like Impediment Boards, “Spotify-Style Health Checks“, Team Dysfunction Surveys, Maturity Models, Happiness Surveys, etc.  My belief:  Your people already know your biggest problems, so using them as a “barometer” for your organization is just as good as any “hard data” (AKA – Trust what your people are telling you).

But here’s the dilemma:  If leadership is asking you about metrics, they are going to try and measure quantitatively regardless of what you do.  Depending on their experience with Agile, Measuring, etc, this could be ok.  But if you choose not to participate, you are giving up your voice in the process.

So, I decided to engage by framing the problem in this way: “If we are going to measure, how do we create metrics that satisfy leaders and assists our Agile transformation in a positive way?”.

Risks with Metrics

One of my favorite summaries on the downsides of metrics is this Industrial Logic post, and I encourage you to read that entry for more info.  But let me try to “summarize their summary”:

  • Measuring can be problematic because of the potential for gaming, triggering unconscious behavioral changes, and misuse by misguided leaders setting “targets” people should try to reach.
  • You can become obsessed with measuring (and measuring too much), thinking that measuring tells you all you need to know, and end up missing other important things (human factors, other critical measures, etc).

There is a situation I’ve encountered which I think summarizes all these concerns:  I started working with a set of teams that had been doing “Scrum” for many years, but kept telling me they were tired of it and hated it.  The mere mention of the words “sprint” or “velocity” and the team would rebel.  As I earned the team’s trust they shared one of those dreaded “Top-Down (Bad) Agile” stories.

Previous managers felt velocity was the “ultimate measurement” and used it to set targets which were unrealistic and rarely met.  An example of those targets: “you’re not moving fast enough, I expect you to double your velocity by next Sprint or else…”.  This experience taught those teams that this “Agile” thing was some horrible management tool, and they vowed to avoid anything associated with it in the future.  Moving these teams past this mindset took over a year of intense coaching work.

So clearly metrics (especially ones that try to measure “productivity” or “throughput”) can be misused and we need to be very conscious of these risks when measuring.

So, What Should We Measure?

I decided the key was to find metrics that encourage the behaviors we would like to see out of our teams and organizations.  Even if metrics like this are gamed they are still encouraging “better” behaviors.

Behaviors like:

  • Building an organization around happy people (Happier people are more motivated and innovative)
  • Working in Smaller Increments (so we can get feedback faster and take advantage of the 80/20 rule)
  • Ensuring we are not sacrificing quality or craftsmanship for speed (or at least that we’re aware that we’re doing it)
  • Making sure we can deliver value quickly and respond to change

One final request from leadership was to have some Metrics that have “Industry Standards”.  They didn’t all need to have this, but at least some of them.  The risk here is these could be used as those “targets” we mentioned above, but the idea was to at least understand how we were doing compared to the rest of the industry.

Putting it all together, the final “Acceptance Criteria” for this Metrics request:

  • Mixes Qualitative and Quantitative Data
  • Inspires/Encourages “Agile Behaviors” (See Above)
  • Some Metrics have Industry Standards
  • Have a Minimal Impact on the Teams to Measure

Without further ado, here are the metrics I suggested.

Net Promoter Score (NPS) – This can capture a sense of the “Happiness” of our teams and individuals which we can track over time.

There is no perfect way to measure happiness.  I’ve experimented with a 1-6 happiness scale; expanding that 1-6 happiness scale to measure multiple things (Product, Personal, Organization, etc); Measuring using an AMP model (Autonomy, Mastery, and Purpose), Asking a simple yes/no “are you happy?”.  Overall, I’ve found NPS to be the simplest, most accurate, and most revealing measurement.

Bonus: The 2nd part of this question – “What would improve your score?” – is probably the most important part.  It gives you valuable feedback on why your people scored you the way they did, and gives you a place to start fixing the problems in your team/org.

(Average) Cycle Time of Issues – This can tell us how long it takes product development teams to finish a piece of work from the time they pick it up, to the time they mark it done.

It’s far from a perfect measurement, but what it can provide us a sense of the average size of the increments of work the teams are doing while also providing us part of the picture of how long it takes us to get things from ideation to in front of the customer.

Bonus: It’s also useful to measure your cycle time’s standard deviation so you start to get a sense of your system’s variance.  The more variance you see here, the less accurate your forecasts/estimates will be.

# of Issues Closed – This is a raw measurement of the number of issues (bugs, stories, etc) each of the teams and the org closed in a given period.  This gives us a very rough sense of “throughput”.

Used in conjunction with Cycle Time, we can use this data to start to forecast how much work we can get through our organizations, and how long it will take us to do it.  Particularly useful when forecasting/”ballparking” new larger pieces of work, such as Epics/Missions.

Lead Time to Change – This measures how long it takes us to get stuff to production (aka in front of the customer) after our product development teams mark it “done”.

With this data we can start to paint a picture around our release processes.  It also encourages us to move towards Continuous Integration/Delivery/Deployment, and gives us another piece of critical information in our product delivery stream.

Bonus: This combined with cycle time can tell you on average how long it takes you to get something from the time your teams pick it up to the time it is in front of the customer.

Note: This is one of the “Industry Standard” metrics from the “State of DevOps” paper published by Puppet, and I was alerted to this paper by Jez Humble.

Change Risk Anti-Patterns, or C.R.A.P. Originally derived from a couple engineers at Google, this metric tries to measure how “risky” it is to change a piece of code.

Monitoring this metric over time can tell you if the work you are doing is creating additional technical debt.

Code Coverage % This tells us what percentage of our code is covered by automated tests.

Monitoring this can encourage our teams to use more automated test coverage.  Additionally, when used in conjunction with the C.R.A.P. Metric, we can start to answer the question “do we have appropriate automated coverage” of our code base?

There are some downsides to this, and you may have heard them before:  “This doesn’t tell you anything about the quality of the tests”, “I can fake this metric easily”, etc.  I know all of that.  But we must start somewhere, and we want to encourage automated testing as we move away from manual.

# of Deployments – This is a raw measurement of how many times we deploy to production…aka how often we are getting new stuff in front of our customers.

We’re trying to move our organization toward smaller increments and quick delivery to our customers to reduce the length of the feedback loop of our product delivery value stream.  Measuring this tells us where we are in comparison with the rest of the industry, and encourages our teams and organizations to deploy more frequently.

Note:  This is another one of the “Industry Standard” metrics from the “State of DevOps” paper published by Puppet

Mean Time to Recover – This measures how long it takes to get the system back up if there is a problem.

This metric follows the line of thinking from Mark Zuckerberg of “move fast and break stuff…with the best infrastructure”.  To win in today’s market, we must keep moving faster and experimenting as quickly as possible.  The reality of that environment things will break in the process.  We have to ensure we’re able to recover quickly when we do break stuff.  This measures how fast on average we can do that.

Note: This is another one of the “Industry Standard” metrics from the “State of DevOps” paper published by Puppet

Change Failure Rate – This metric measures how often we break something when we deploy to production.  This gives us a sense of the reliability/health of our release process.

Note: This is the last of the “Industry Standard” metrics from the “State of DevOps” paper published by Puppet

What You Don’t See and Why

There is one giant omission here:  “Business Value” metrics.  The metrics I shared here are potentially applicable to multiple teams, products, and organizations and we weren’t sure how to measure “Business Value Delivered” in a standard way.  The argument I’ve heard (and I have trouble countering) is that each product has different metrics that show if they are delivering business value.  This is why we left them out and instead focused on happiness, software delivery rate, and craftsmanship.

What do you think of these metrics?  What do you measure?  Do you have thoughts on “standard business metrics”?  Share your thoughts in the comments section below.

“Agile” Metrics

Also published on Medium.

One thought on ““Agile” Metrics

  • November 21, 2017 at 3:10 pm
    Permalink

    An important area of focus, that many times gets overlooked, is supportability. An interesting metric to track are the number of enhancements requested by support teams that are implemented by development. Support teams are the closest to your customers & most often the most knowledgable, yet their voice is seldom heard by product teams. An increase in this metric can have a direct correlation on both customer satisfaction, support & development team happiness.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *