Managers encounter continued pressure to deliver more software in less time and they tend to introduce many different KPIs to measure success. But why do they introduce KPIs in the first place? Which are the good KPIs? Which ones are not useful? And which ones are the harmful ones? This white paper presents some of the most common KPIs and the expected outcomes that you could find using them. Agile Vs. Waterfall

1. Agile KPIs vs. Traditional KPIs – A mind shift
1. Executive summary
Managers encounter continued pressure to deliver more software in less time and they tend to
introduce many different KPIs to measure success. But why do they introduce KPIs in the first
place? Which are the good KPIs? Which ones are not useful? And which ones are the harmful
ones? This white paper presents some of the most common KPIs and the expected outcomes
that you could find using them.
2. Historically why did we measure success?
Success is difficult to measure because it has many different variables and each stakeholder will
have a different view of the definition of success: the business owner usually considers success
a short “time to market” and within budget, while a technical architect might consider stability
or scalability as the success criteria.
Historically KPI measure has been used for the following reasons:
1. Analyse the current process to identify and understand gaps;
2. Promote behaviours that will lead to better results;
3. Understand how successful we have been and produce a lessons learnt report;
4. Punish or reward teams, individuals or departments
The way the KPIs will be used in the teams can lead to great success or tremendous failure. A
KPI is like any other tool that you could find in a toolbox. A hammer can be use to hang a
painting or it can be used to break a window. It is up to the relevant stakeholders to understand
how should they apply the information provided by the KPI.
The strength of Agile is to fail fast and cheaply, learning from the mistakes occurred. Whilst
setting up the Agile environment, one should always ask the following two questions:
A) Is the KPI allowing me to improve the process?
B) Is the process going against the Agile principles?
Using the punishment approach in an Agile environment will discourage the people to be free
to speak up their mind. That can lead to the situation when the people would hide their
mistakes and therefore no useful and relevant lessons learnt report can be applied. Ultimately,
the teams will increase the technical and non-technical debt. Eventually, there will be a price to
pay for that debt and the longer is unidentified and consequently fixed, the more exponentially
costly it will be to repair the damages produced.
Furthermore, one of the strengths of Agile comes from an environment where the team
members can feel free to search for the best value for the product owners and for the final users.
Set up KPIs that will not affect the stability, the processes or the team.
The positive reward approach is not necessarily a good approach either. For instance, do not
believe that working hard for many hours is a good KPI. The University of Nebraska proved that
working extra 10 hours a week can reduce the overall productivity by up to 40% (Raynar 1997).
Also, it is important to note that positive reinforcement for one team member can be seen as a
punishment for another one. As a consequence, avoiding punishment or reward is the better
solution.

2. 2.1. Traditional vs. Agile KPIs
The KPI to understand success is an inheritance of the Waterfall mentality as historically the
organisations did not have any earlier warning tools. Managers tend to use this mechanism
because the “lessons learnt” stage is at the end of the project.
However, the KPIs to improve the process are the foundation of Agile:
Tips:
1. Setting up the right KPIs that will improve the processes and will lead to more
quality software in less time.
2. Setting up managerial KPIs to analyse the quantity or quality of your software will
harm your Agile processes.
3. Work smart, not hard.
2.2. The right KPIs in Agile
Agile is a methodology that promotes change and adaptability. Managers have to understand
that the KPIs should reflect the reality, should be careful not to restrict freedom, and should
avoid rigidity and static behaviours. The classic example of this is to standardise a value per
point in terms of time or cost. The ultimate outcome of such action is that the teams won’t have
a sense of achievement and success during the sprints.
Finding the best KPIs for your environment requires a high level of understanding of the
situation, empathy and patience. The golden rule should be: find your teams’ weakness and
setup KPIs that will promote Agile principles. Also analyse:
a) Does this KPI promote individuals and interactions or processes and tools?
b) Does this KPI promote more working software or more comprehensive documentation?
c) Does this KPI promote customer collaboration or more contract negotiation?
d) Does this KPI promote flexibility to change or rigidity to follow a plan?
the So that so we We want to
do software
We make more
software in the
next sprint
Processes
Setup a KPI to
improve
so we so that We want to
do software
How much
software we will
have developed
We can
understand
project success
Setup a KPI to
understand

3. Examples of KPI (let’s play a bit of Game Theory):
• “We are going to compare team’s efficiency to promote the next managers”. If we apply
some Game Theory principles to this KPI we will come to the following conclusion: team
members are competing among themselves for the best results. What initially seems
like a good idea, in reality, it contradicts principle “A” and, for the long term, the output
of this is KPI will be that team members will no longer collaborate. Thus, the general
combined efficiency of the teams will be reduced.
• “We are going to measure the velocity of the teams”. Velocity is a great internal tool that
each team uses to understand their strengths and weaknesses, and bring better
estimates in future, therefore reducing the risk. If a manager tries to compare velocities,
then the logical reaction is that there will be an inflation of the point value. That will
dilute the information and the estimations will become unstable and increase the risk.
This happened because it was against the principle “A”.
• “We are going to measure client satisfaction”. Satisfaction of the clients can only be
achieved by increasing customer collaboration (principle “C”). That KPI will positively
influence your teams to improve your process.
• “We are going to measure the number of Agile best practices used during the sprints”.
This will promote almost all principles, improving processes that will thus lead to more
working software.
To note that there are a few KPIs that can be used in any environment:
• Continuous Improvement Index; create a culture of improving and of sharing
ideas;
• Agile best practices;
• Engagement levels with the users;
• Teams velocity.
3. Agile Maturity Assessment
Many people have theoretical knowledge of Agile but not everyone is aware of the cultural and
organizational challenges that the transformation towards Agile requires. Generally, creating
the right environments requires a dual approach:
a) Top-down approach: the organizational needs to understand how mature they are
and its next steps. This will require buy in and support from senior stakeholders;
b) Bottom-up approach: how are the technical processes, architecture and tools
running on the day to day.
Once the assessment has been completed, the right KPIs can be suggested to promote Agile
behaviours in your environment.
Important tip: Do not try to setup KPIs before conducting an Agile Maturity Assessment by an
Agile Coach with extensive practical experience. He should understand deeply the strengths
and weakness of the individuals, the teams, the organization, the dynamics of the business
processes and the appetite for change.

4. 4. KPI analysis
The following table shows some of the possible KPIs and what do they promote. It is important
to note that some KPIs are useful to improve the processes for the Agile teams (i.e., chickens)
but some other stakeholders (i.e., pigs) could misuse them.
Managerial KPI
Agile KPI
Punish or reward
Understand success
Prom
ote behaviours
Understand gaps
KPI
The KPI promotes
Customer satisfaction X X X
Do not use any KPI as punishment or reward
X X Client oriented focus
Best Agile practices used eg. XP,
Lean, SAFE, Kanban Scrum.
X X X
Increase productivity
Velocity (points per sprint) X X X Increase efficiency
Team stability X X Will generate better teamwork
Attrition X X Team collaboration
Time to market X X X X X Flexibility of the teams to adapt
Agile adoption X X X Promotes best practices
Business involvement
X X X X
Close the gap between the IT and the
business
Working software
X
It can lead to an endless loop
understanding productivity (quantity vs.
quality)
Number of releases
X X X
It can promote releasing more or less than
what is actually needed
On time delivery
X X X
This could lead to under-commitment from
the teams
Value delivered
X X
Good KPI but problematic as sometimes
value is out of the control of the teams
Compare teams velocity X Inflation of points
Individual velocity
Poor KPI as it will lower morale and
increase micromanagement
Number of code lines Poor KPI as developers write inefficiently
Percentange of sprint
commitment
X X X
Attention required as this KPI can reduce
quantity committed and reduce moral
Defects density
X X X
Good KPI but attention not to lower
amount of software needed
Value of points Poor KPI - Inflate points
Responsiveness to change X X X X Improves collaboration
Time spent in improvements X X X X Improves processes
Unfinished stories X X X X X Understand waste and value
Use and Agile Maturity
assessment (Fuller 2014)
X X X X
Understand Agile journey and create the
right KPI
Convert or measure points into
days
X X
Poor KPI: motionless mentality. The team
does not feel that they are succeeding
Compare teams/ people X Terrible KPI: no collaboration
People performances
X
Terrible KPI: Burnout effect/ low moral.
Low collaboration
These KPIs can be used in any environment as they promote good Agile practices.

5. Some other KPIs that you could consider could be:
While choosing your KPI you can also take a look to some of the following metrics: lead time,
defect count (at various phases; i.e., what’s a bug?), work in progress, code coverage,
responsiveness to change, team stability, velocity (story points or story count per sprint), return
on investment, innovations per sprint, continuous improvement time, failure load (fire-fighting
time), iteration burn-down, unfinished stories, innovation index, customer satisfaction, un-
deployed stories, number of blocks, budget/schedule compliance flow efficiency (lead
time/touch time), release burn-up, projected capacity of next sprint (in person days), projected
capability (in story points range) of next sprint, planned capability of the sprint (in story
points), accepted velocity - story points done and accepted by product owner in the sprint,
discovered work added to the backlog (in story points and stories), scope increase/decrease
during the sprint/release, technical debt incurred this sprint (in story points), cost of sprint,
sprint predictability, actual stories completed vs. committed stories, retrospective process
improvement, communication, understanding of sprint scope and goal, user stories delivered
versus user stories accepted, defect-removal efficiency (DRE), value delivered, on time delivery
(Leigh 2015).
5. Summary and take away tips
a) Setting up KPIs that will promote the improvement of the processes and will lead your
teams to more quality software in less time.
b) Setting up managerial KPIs to analyse the quality and quantity of the software will harm
your processes.
c) “Increments of improvements” is the best KPI to make more software with better
quality. Do not use KPIs as means to punish or reward your people in Agile.
d) Every few sprints conduct an Agile Maturity Assessment to understand your progress
and the next steps.
About the Author
Javier Espinosa de los Monteros Foret is an independent Agile Coach that after
obtaining two first class degrees, Management and IT, worked for more than 10
years in international projects across Europe leading Agile and Waterfall
software projects. He obtained his MBA from IE Business School and he is also a
Scrum Master Certified, Change Manager certified APMG, Project Manager
(Prince2 Practitioner) and Programme Manager (MSP Practitioner) certified.
yo@javierespinosa.com
https://uk.linkedin.com/in/javierespinosadelosm
Acknowledgements
Special thanks to Simson Leigh, my good friend and excellent Agile coach, for his constant advice
and support.

6. Bibliography
Fuller, Dan. Measuring Agile Adoption Maturity. 21 May 2014.
http://www.slideshare.net/cognizant/shuhari-measuring-agile-adoption-
maturity (accessed Septembre 10, 2014).
Raynar, Randolph Thomas and Karl A. “SCHEDULED OVERTIME AND
LABOR PRODUCTIVITY: QUANTITATIVE ANALYSIS.”Journal of Construction
Engineering and Management 123 (1997): 2.