Blog

Business Outcome-Based Metrics: How to Effectively Measure Your Agile Transformation Journey

By: Andy Cleff | Feb 11, 2020 |  Agile Transformation,  Article

Outputs or Outcomes?

Many organizations are stuck measuring and making decisions based on outputs – like velocity. In fact, team velocity is one of the most commonly used, abused, and misused metrics in Agile software development as well as during digital transformations. Teams, their managers, and even their stakeholders often focus on “improving velocity” without considering the entire value delivery system. Then they are shocked when they don’t get the business outcomes they really want, for example predictability or speed.

In this article, we explore healthy ways for your organization to use metrics to gain meaningful insights into the results of your experiments in the course of your Agile transformation.

Measure for a Purpose

As Simon Sinek famously says: “It all starts with why.” You need to understand what you think you are going to measure and why you want to measure it. When it comes to metrics in Agile, the data itself is not the goal — instead, it’s a means of tracking your journey, testing hypotheses, and providing feedback as you head towards your next goal. 

That goal, the Big Why, should be focused on business outcomes, not outputs. Here are nine business outcomes (that come from Agile Velocity’s Path to Agility® framework) to think about:

  1. Employee Engagement: Employees are more satisfied in their work, willing to go the extra mile, passionate about the purpose of their jobs, and committed to the organization.
  2. Continuous Improvement: The ability of the organization to relentlessly pursue optimizations in all aspects of business functions.
  3. Innovation: New ideas, creative thoughts, or novel imaginations provide better solutions to meet new requirements, unarticulated needs, or known market needs.
  4. Customer Satisfaction: Customers are satisfied with the experience, benefits, and outcomes when using your product or service.
  5. Market Responsiveness: The ability of the organization to pivot quickly to respond to ever-changing market demands.
  6. Predictability: Teams maintain a predictable cadence of delivery enabling the business to make informed business decisions.
  7. Speed: The time it takes to deliver an idea to the market.
  8. Quality: The product or service meets the expectations of the market for usability, and reliability.
  9. Productivity: Increase the business value realized while maintaining or reducing costs.

We recommend that your organization pick one. Sorry, you can’t have them all at once! Limiting your organizational WIP (Work in Progress) helps create a clear sense of urgency.

Reasons To Measure

There are many reasons to measure as you track progress towards organizational business outcomes during your transformation. These include:

  • Knowing where to invest your money – and to do so based on more than just gut instinct
  • Knowing if you are building the right things for our market
  • Measuring your performance and alignment — where your inventory (the software that the development teams create) that addresses features, defects, risks, and debt is not easily visible
  • Knowing if your customers and employees are delighted (or not)

Why Don’t More Organizations Measure More Things?

In our practice, we routinely discover that many organizations don’t have much in place in terms of metrics. Why not? Some reasons include:

  • People are afraid of weaponized metrics – ones used not for purposes of continuous improvement, but instead for comparison and punishment.
  • They were collecting vanity metrics, ones that didn’t offer any predictive power, and eventually considered measurement programs “useless.”
  • Metrics are actionable, however, they are presented in a way that people don’t find useful (for example endless tables of numbers instead of clear graphic representations).
  • Folks believed the “right things” were just too expensive to measure, and they failed to poke around to find existing data that would suffice.
  • They mistakenly thought the “right things” were immeasurable — and didn’t consider finding proxies.

Meaningful Agile Metrics for Digital Transformations

Successful organizational transformations put in place metrics programs that focus on global outcomes and meaningful feedback loops. This is true at any point along the transformation journey.

Here are a few idea starters for each of the Path To Agility business outcomes listed above:

Business Outcome Example Measures
Employee Engagement
  • eNPS
  • Bad turnover
  • Employee referrals
  • Offer acceptance 
  • Feedback surveys
  • Team learning logs
Continuous Improvement
  • Value stream flow efficiency
  • Reduction in recurring impediments
  • Cumulative flow
Innovation
  • Market Share
  • Number of validated business-level hypotheses developed and tested
  • Number of failed experiments
  • Slack time
Customer Satisfaction
  • Number of times a week team members talk to an actual customer
  • NPS 
  • Retention (DAU/MAU)
  • Referrals
Market Responsiveness
  • AARRR (Acquisition, Activation, Retention, Referral, Revenue)
  • HEART, (Happiness, engagement, adoption, retention, task success)
Predictability
  • % complete of Sprint plan
  • Velocity variability
  • Say-Do ratio 
  • Unplanned work items 
Speed
  • Cycle time 
  • Lead time
  • Deployment frequency
  • Meantime to restore (MTTR)
Quality
  • Defects in production
  • Change fail percentage
  • Automated test coverage
  • Ratio of fixing work vs feature work
Productivity
  • Value delivered
  • ROI
  • KPIs
  • Thieves of time

Common Anti-patterns for Metrics During Agile Transformations

There are many anti-patterns to be mindful of when implementing metrics in Agile settings:

  • Hawthorn Effect: Individuals modify their behavior in response to their awareness of being observed. (“Oh, watching the burndowns are we… well, here’s one set for you. The real set is on the back of the rolling whiteboard.)
  • Goodhart’s Law: All metrics of evaluation are bound to be abused. It’s just human nature. Individuals try to anticipate the effect of a policy and then take actions that alter its outcome. (Things like Class of Service, or rules like FIFO can be gamed. “That’s an expedite ticket, and I’m judged on time to first response? I’ll send it right back to ya with a question. Bingo, I hit my SLA!”)
  • Friedman’s Thermostat: Correlation does not mean causation, but it sure can be a clue.
  • Comparing metrics across teams: Teams all have different contexts. (Velocity, sorry to mention it again, is relative and team dependent. Same with throughput – “comparison” leads to over the wall development. Same with operational stability – “comparison” can lead to painful change management processes.) Cross team comparisons remove safety and learning. As Deming reminds us: ”Whenever there is fear, you will get wrong figures.”
  • Focusing on lagging over leading Indicators: Taking a macro-economics view for example: the level of unemployment is a lagging indicator. By the time you see an upward trend, it’s a bit late to stop it. A good leading indicator for this domain would be Architectural Design Firms submitting bids — a sign of pending economic growth, with job creation as part of it. Leading indicators in software: automated test coverage, code complexity, and team health.
  • Choosing vanity metrics: Good for feeling awesome, but bad for taking action. Funny things happen when we rely on vanity metrics. When they go up, all of us will be the first to take credit. When they go down… well… it wasn’t me!
  • Ignoring companion measures: Keep in mind there’s no free lunch. Measure holistically, so you see the tradeoffs. If you’re working on improving cycle time or throughput, you better also keep an eye on quality as well as team morale.

Measuring Your Agile Transformation Journey: Values, Intentions, and Purpose

When you:

  • Are transparent about what and why you measure what you do…
  • Think about your team members and customers as humans (not resources or wallets)…
  • Support your teams by thinking long term about values, intentions, and purpose…

…Then your organization will be able to handle the adversity that comes with today’s VUCA world. You’ll build and maintain resilience as you change the status quo, all the while developing a culture rich with collaboration and innovation.

Good things will happen — if you just stop focusing on velocity.

 

What’s Your Organizational Agility Score?

In less than an hour, you’ll get valuable insights into your organization so you can improve team performance and achieve your business goals faster.

Learn more

Leave a Reply

Your email address will not be published. Required fields are marked *

< Back