Metrics in Agile: How to Effectively Measure Your Transformation Journey

Image of a person at a desk, pouring over a stack of papers, a laptop, and an ipad filled with charts and data. Metrics in Agile can sometimes feel overwhelming. Learn how to measure the right things so you can focus on what matters.Velocity. It is one of the most commonly used, abused, and misused metrics in Agile software development. Teams, their managers, and even their stakeholders often focus on “improving velocity” without considering the entire value delivery system. Then they are shocked when they don’t get what they really want, for example, predictability or speed.

In this article, we explore healthy ways for your organization to use metrics to gain meaningful insights into the outcomes of your experiments as you and your teams pursue the craft of creative knowledge work during an Agile transformation.

Measure for a Purpose

As Simon Sinek famously says: “It all starts with why.” You need to understand what you think you are going to measure and why you want to measure it. When it comes to metrics in Agile, the data itself is not the goal — instead, it’s a means of tracking your journey, testing hypotheses, and providing feedback as you head towards your next goal. 

That goal, the Big Why, should be focused on outcomes, not outputs. Here are nine business outcomes (that come from Agile Velocity’s Path to Agility® framework) to think about:

  1. Employee Engagement: Employees are more satisfied in their work, willing to go the extra mile, passionate about the purpose of their jobs, and committed to the organization.
  2. Continuous Improvement: The ability of the organization to relentlessly pursue optimizations in all aspects of business functions.
  3. Innovation: New ideas, creative thoughts, or novel imaginations provide better solutions to meet new requirements, unarticulated needs, or known market needs.
  4. Customer Satisfaction: Customers are satisfied with the experience, benefits, and outcomes when using your product or service.
  5. Market Responsiveness: The ability of the organization to pivot quickly to respond to ever-changing market demands.
  6. Productivity: Teams maintain a predictable cadence of delivery enabling the business to make informed business decisions.
  7. Speed: The time it takes to deliver an idea into the market.
  8. Quality: The product or service meets the expectations of the market for usability, reliability.
  9. Predictability: Increase the business value realized while maintaining or reducing costs.

We recommend that your organization focus on a select few outcomes. (Sorry, you can’t have them all at once!) Limiting your organizational WIP helps create a clear sense of urgency. 

Reasons To Measure

There are many reasons to measure as you track progress towards organizational outcomes. These include:

  • To know where to invest your money – and to do so based on more than just gut instinct
  • To know if you are building the right things for our market
  • To measure your performance and alignment — where your inventory (the software that the development teams create) that addresses features, defects, risks, and debt is not easily visible
  • To know if your customers and employees are delighted (or not)

Why Don’t More Organizations Measure More Things?

In our practice, we routinely discover that many organizations don’t have much in place in terms of metrics. Why not? Some reason include:

  • People are afraid of weaponized metrics – used not for purposes of continuous improvement, but instead for comparison and punishment.
  • They were collecting vanity metrics, ones that didn’t offer any predictive power and eventually considered measurement programs “useless.”
  • Folks believed the “right things” were just too expensive to measure, and they failed to poke around to find existing data that would suffice.
  • They mistakenly thought the “right things” were immeasurable — and didn’t consider finding proxies.

Successful Programs for Metrics in Agile

Successful organizational metrics programs focus on global outcomes and meaningful feedback loops. This is true at any point along the transformation journey. 

Here are a few idea starters for each of the business outcomes listed above:

Business Outcome Example Measures
Employee Engagement
  • eNPS
  • Bad turnover
  • Employee referrals
  • Offer acceptance 
  • Feedback surveys
  • Team learning logs
Continuous Improvement
  • Value stream flow efficiency
  • Reduction in recurring impediments
  • Cumulative flow
Innovation
  • Market Share
  • Number of validated business-level hypotheses developed and tested
  • Slack time
Customer Satisfaction
  • Number of times a week team members talk to an actual customer
  • NPS 
  • Retention (DAU/MAU)
  • Referrals
Market Responsiveness
  • AARRR (Acquisition, Activation, Retention, Referral, Revenue)
  • HEART, (Happiness, engagement, adoption, retention, task success)
Predictability
  • % complete of Sprint plan
  • Velocity variability
  • Say-Do ratio 
  • Unplanned work items 
Speed
  • Cycle time 
  • Lead time
  • Deployment frequency
  • Meantime to restore (MTTR)
Quality
  • Defects in production
  • Change fail percentage
  • Automated test coverage
  • Ratio of fixing work vs feature work
Productivity
  • Value delivered
  • ROI
  • KPIs
  • Thieves of time

Common Anti-patterns for Metrics in Agile

There are many anti-patterns to be mindful of when implementing metrics in Agile settings:

  • Hawthorn Effect: Individuals modify their behavior in response to their awareness of being observed. (“Oh, watching the burndowns are we… well, here’s one set for you. The real set is on the back of the rolling whiteboard.)
  • Goodhart’s Law: All metrics of evaluation are bound to be abused. It’s just human nature. Individuals try to anticipate the effect of a policy and then take actions that alter its outcome. (Things like Class of Service, or rules like FIFO can be gamed. “That’s an expedite ticket, and I’m judged on time to first response? I’ll send it right back to ya with a question. Bingo, I hit my SLA!”)
  • Friedman’s Thermostat: Correlation does not mean causation, but it sure can be a clue.
  • Comparing metrics across teams: Teams all have different contexts. (Velocity, sorry to mention it again, is relative and team dependent. Same with throughput – “comparison” leads to over the wall development. Same with operational stability – “comparison” can lead to painful change management processes.) Cross team comparisons remove safety and learning. As Deming reminds us: ”Whenever there is fear, you will get wrong figures.”
  • Focusing on lagging over leading Indicators: Taking a macro-economics view for example: the level of unemployment is a lagging indicator. By the time you see an upward trend, it’s a bit late to stop it. A good leading indicator for this domain would be Architectural Design Firms submitting bids — a sign of pending economic growth, with job creation as part of it. Leading indicators in software: automated test coverage, code complexity, and team health.
  • Choosing vanity metrics: Good for feeling awesome, but bad for taking action. Funny things happen when we rely on vanity metrics. When they go up, all of us will be the first to take credit. When they go down… well… it wasn’t me!
  • Ignoring companion measures: Keep in mind there’s no free lunch. Measure holistically, so you see the tradeoffs. If you’re working on improving cycle time or throughput, you better also keep an eye on quality as well as team morale.

Metrics: Values, Intentions, and Purpose

When you:

  • Are transparent about what and why you measure what you do…
  • Think about your team members and customers as humans (not resources or wallets)…
  • Support your teams by thinking long term about values, intentions, and purpose…

…Then your organization will be able to handle the adversity that comes with today’s VUCA world. You’ll build and maintain resilience as you change the status quo, all the while developing a culture rich with collaboration and innovation.

Good things will happen — if you just stop focusing on velocity.

Want to Learn More?

If you need help implementing good processes/practices around metrics including gathering baseline data, we offer a service that does just that!  Our Agile Assessments help organizations develop a baseline for their metrics and understand key next steps in their Agile journey, all while maintaining a focus on what’s important: business outcomes.

Learn more here. 

Leave a Reply