Agile Metrics - Keeping meaningful metrics... meaningful
Updated: Nov 24, 2019
Companies around the world are literally sculpting the future through new product and service offerings that make this period of time in human history truly exciting to be a part of. Most of us take these historic technological advancements for granted. Some of us actually get to experience ‘how it’s made’. Fewer actually focus their careers on defining engineering processes that will maximize end user value and keep organizations competitive. I am one of those lucky few that get to work with world class talent to build out tailored processes that fit the need of the organization I support. I have found a passion in process improvement, the drive to continuously improve, and an affection for meaningful metrics that allow leaders to make data driven decisions and to help teams succeed while building this amazing technology. In an agile world, there are a lot of metrics that your teams produce which can tell you a lot about your teams performance and where opportunities exist to get better. But before you run off and start building agile dashboards keep this in mind… Agile metrics are only as meaningful as the culture that supports them.
With great data… comes great responsibility! Or something like that. What I am trying to say is that the culture of your organization will determine whether or not the agile metrics your teams are generating will be meaningful. From my experience, most organizations fall into two cultural categories. The first is a trust based culture where employees are empowered to fail fast, learn, and are trusted to be productive. The second is a fear based culture which has a big brother feel to it and leverages your employees fear of failure as the primary driver to encourage productivity.
In a trust based culture of continuous improvement, your teams will achieve high performance by celebrating both good and bad metrics, leveraging them to have conversations early and often, and sharing these experiences with your team and stakeholders. Most companies do a good job celebrating success, but I would make the argument that the celebration of failure is just as important if not more important. To paint the picture a little more clearly… How do we learn that the proverbial stove is hot? Well we either touch it ourselves or someone that has already burnt themselves tells us what happens if you touch it. If that failure wasn't shared, everyone would likely burn themselves on the hot stove as part of 'growing up'. What's my point? Well, making resources comfortable identifying and sharing their failures can save your company a lot of time and money in the long run. A trust based cultural environment drives improved moral, continuous learning opportunities, and allows teams to keep metrics open, honest, and meaningful so that teams can improve.
In a fear based culture, your leaders will intentionally or unintentionally use fear to drive an expected outcome and the metrics generated are used against the team when they are negative. Naturally, your resources are going to spend more time highlighting their successes over their failures. In some cases your resources are going to proactively hide failure metrics and indicators. In the most extreme cases, your resources are going to influence inputs that drive metrics so that they are reporting a false reality that makes their team look good. All three of these situations are bad but they happen all the time. If you are a leader in an agile organization, it is critical that your resources trust your reasons for tracking these metrics. If they don’t they will find ways to make it appear that they are a high performing team even when they are not.
So, why do we track agile metrics?
There are many reasons why we track metrics but I will keep it to my top 3:
Continuous improvement - It’s not about where your teams start, but how they improve over time. How do we know our teams are improving if we do not have metrics, a starting point, and trends over time? In my opinion, this should be the main driver for tracking agile metrics and management should cut the relationship between personal performance and the agile metrics that matter (An entirely different topic for another day). If teams are able to build continuous improvement into their every day lives, they will succeed.
Can’t manage what’s not being measured - Agile metrics don’t always tell the whole story but they are a great starting point that can be used to drive the right conversations and changes across the team. How can you manage something if you don’t know where you are to begin with? Gut feel only goes so far and can get you into a lot of trouble if you rely on it too often.
Data driven decisions - When you have legitimate metrics it makes the decision making process a whole lot easier and much faster. Not only does it eliminate risk associated with your decision, it also makes it a lot easier to drive organizational change and generate buy-in as your company matures.
A few metrics and how a poor culture can ruin them:
1) Velocity, the wrong way - Management only focuses on team velocity as a productivity metric. They expect that your velocity should always go up over time because to them, this means that your team is more productive. Your personal goals are tied to this velocity metric and it’s not trending in the right direction.
Outcome - Unfortunately, when a metric is used as stated above you are likely living in a fear based culture. Number one, the metric is misunderstood and is being used to drive the wrong behavior. Number two, it’s super easy to game. In order to get better performance grades the team starts to artificially increase the story points they assign to each story and boom, they are productive again… NOT.
How you should use velocity - Velocity is NOT a productivity metric and it’s NOT a predictability metric. Velocity is a consistency metric. The purpose is to measure how consistent a team is in delivering work each sprint. A highly volatile velocity will produce a lot of coaching opportunities that can be used to continually improve your teams to become more consistent. Highly volatile velocities can indicate scope change mid-sprint, unplanned work, poor requirements authoring, requirements that are too big, or a number of other things that the team can address which will result in a consistent velocity.
2) Work in progress (WIP), the wrong way - Your PO or manager constantly makes everything the highest priority and consistently makes promises to stakeholders without consulting the team. Your Scrum Master has WIP limit goals set but management continues to ignore them.
Outcome - WIP metrics no longer matter. The team will work on items in the backlog without pulling them into the proper workflow. This will show that they are meeting their WIP limits although in reality they are working on requirements that are in the backlog or not listed at all in order to satisfy their manager. This renders WIP limits meaningless and makes teams much less efficient due to constant context switching.
How you should use WIP metrics - Your team and stakeholders should agree to WIP limits that make your team most efficient. These should be honored and communicated to set the right expectations. There is no magic time and your resources most likely aren’t sand bagging so stick to the script and prioritize your backlog effectively.
3) Cycle time, the wrong way - Your manager finally stumbled across the important cycle time metric. They noticed that cycle times are high which results in risky sprint delivery every two weeks. Without identifying reasons for the high cycle time they decide to lower the teams cycle time by tying it to their performance goals.
Outcome - In response, the team starts to focus less on quality so that they can complete the work a little quicker which results in a lower cycle time and meets their performance goal objectives.
How you should use cycle time - Understanding the root cause of high cycle time will help you to coach the team and make them much more efficient. This measurement can help to identify bottlenecks in the current workflow that prevent the teams from completing work on time. A lot of times a team is missing a role or a skillset that makes it much harder to complete work. As their manager, you could make changes that would set the team up for success such as shifting/adding a resource, providing learning opportunities, or cross-training individuals that want to learn.
I absolutely love generating and tracking metrics across the teams that I lead because it makes it much easier to find coaching opportunities. The key is to use your metrics as a vehicle for continuous improvement and learning opportunities and not to point fingers, blame others, or tie them to bonus criteria. Metrics generate the right conversations with the right people when used properly and are extremely valuable within the right corporate culture. If you find that you are in a fear based culture see what you can do to change it or find ways to make sure your resources know that you intend to use the metrics the right way. Regardless, spend some time and put yourself in your resources shoes. Do they feel they are in a trust based or fear based culture?
Thanks for the read!
Author – ‘Process Pat’ McClain
Experience – I am a process coach that has been hired to lead large scale global process change initiatives that drive competitive advantages in different areas such as increased predictability, improved productivity, cost reduction, and increased efficiency across Product Development and other related organizations. These results are achieved through efforts I have led related to agile maturity, capex reporting process changes, and toolset analysis/consolidation efforts that align with the people and processes of the organization I am working with.
Disclaimer: This article is not affiliated with current employer and is based off prior professional experience over the last decade.