Agile Metrics – Predictability
Updated: Nov 24, 2019
By: ‘Process Pat’ McClain
Predictability… everyone wants it, few achieve it. It’s all about setting and delivering the right expectations with your internal and external stakeholders so that you are able to develop trust, create confidence, and deliver value to your end users frequently and on time. Every company is going to have its own unique factors that play into a team’s ability to achieve a high level of predictability but there are consistent measures that every company should adopt, but likely aren’t. Like most things, if you do not execute at the fundamental level how in the world are you going to execute on the big picture that you have already communicated to stakeholders? My techniques start by mastering and building on the basic scrum fundamentals while using specific metrics that will help your teams continuously improve and become much more predictable. A high degree of predictability is one of the most difficult feats to achieve when it comes to software development and I believe that’s because most companies are doing it all wrong.
Velocity has nothing to do with predictability:
This may be controversial, but I believe that a team’s velocity has very little to do with predictability. It’s a useful metric and it’s a necessary input along with a team’s capacity to understand how much work should be committed to in a given sprint, but it has little to do with the team’s ability to deliver their sprint commitments. To prove this, take this example which I find to happen often among scrum teams:
On the surface, Team XYZ seems to be highly predictable. They consistently deliver between 20 and 22 story points every sprint with a velocity of 21.25 (Average story points delivered over the last 4 sprints). It is very difficult for teams to achieve a stabilized velocity (Mastering the basic scrum fundamentals) so they definitely deserve the kudos everyone should be giving them.
But… among further investigation, we found that while Team XYZ can predict how many story points they can deliver in a sprint, they rarely deliver what they planned for in sprint planning. As you can see below in sprint 4, Team XYZ committed to 22 points, had 12 story points that came in mid-sprint and they ended up completing 22 story points. The sprint was planned based off priority order, right? What is the point in planning if we aren’t executing that plan? Well, as does often happen, there was scope creep after the sprint started. Regardless of whether or not that scope creep was legitimate, it is impacting the plan and our team’s ability to be predictable.
So… the question remains. How much of the 22 completed story points were part of the original commitment? Did Team XYZ deliver 10 of the original 22? 15 of the original 22? 22 of the original 22? There is no way of telling how predictable the team was if you are just looking at velocity. It is very possible that Team XYZ delivered 10 of the 22 originally planned story points. In this case, your velocity tells you they are predictable, but I am telling you that you are not.
In order to be predictable, a team should be able to plan work in a sprint and then deliver to an agreed upon Planned Complete Percentage. What is Planned Completed Percentage (PCP)? PCP is the percentage of the original sprint commitment (In story points) that was completed. Some organizations may say 90% PCP is desired while others may shoot for 60% as their goal. Tracking and understanding your scrum team and organizational PCP will set you up for success when it comes to longer term planning initiatives. If my scrum teams cannot execute their PCP sprint goals how can I expect them execute on client commitments, release plans, quarterly plans, or any other longer-term planning cadence your team follows?
If a team is unable to deliver what they planned but they achieved 100% of their velocity… are they really predictable? Not in my opinion.
Here are some of my thoughts on velocity, adjusted velocity, and planned complete percentage metrics that should be used in combination to drive better predictability across your organization:
Measurement: Average number of story points delivered by a scrum team per sprint – sum of story points / number of sprints (Typically average the last 3-6 sprints but try not to count outliers in your calculations such as sprints that overlap with heavy holiday weeks)
Purpose: Understand how much ‘work’ we can count on a team to deliver per sprint given the team dynamic stays consistent
This is the most misunderstood metric of all time when it comes to agile in my opinion. Most companies track it, fewer are able to stabilize it, and even fewer understand that the goal is to be predictable and stable, NOT to continuously increase their team’s velocity. If a team’s velocity drops or goes up, your Scrum Master should be able to point to a change that resulted in that drop or gain.
Measurement: Velocity adjusted to meet your teams sprint capacity – i.e if your teams’ velocity is 100 story points and your team is available for 80% of the sprint due to holiday schedules, only commit to 80% of your 100 story points for this sprint which would be 80 story points.
Purpose: Sprint planning represents a team commitment to get that work done. When resources are not available for part of the sprint, we cannot expect the team to deliver the same amount of work as if they were at full capacity. We plan so that we are predictable and so that we can set expectations with internal and external stakeholders. If we do not adjust our velocity to the real world, we will consistently miss commitments which will drive mistrust and burn out your teams.
Velocity + capacity planning = adjusted velocity which sets your teams up for success
Planned Complete Percentage:
Measurement: Capture work planned when you start your sprint and calculate the percentage of that work that was completed when the sprint ends.
Purpose: This is the true measure of a scrum teams or organizations predictability. ‘Of what we planned, this is what we completed’
A lot of times legitimate unplanned work will be introduced mid-sprint. It’s important to shield your teams from the work so that the right decision makers can vet it against your current priorities before pulling it into your current sprint and interrupting your planned work. This will happen but it should be infrequent. Make sure your teams are paying attention to that unplanned work by measuring the PCP each sprint and having a conversation on how to become more predictable over time.
Author – ‘Process Pat’ McClain
Experience – I am a process coach that has been hired to lead large scale global process change initiatives that drive competitive advantages in different areas such as increased predictability, improved productivity, cost reduction, and increased efficiency across Product Development and other related organizations. These results are achieved through efforts I have led related to agile maturity, capex reporting process changes, and toolset analysis/consolidation efforts that align with the people and processes of the organization I am working with.
Disclaimer: This article is not affiliated with current employer and is based off prior professional experience over the last decade.