2. Metrics -> Improvements
Use metrics to
• Improve quality
• Shorten product cycle time
• Decrease costs
• Decrease risks
• Increase predictability and reliability
3. Control Limits and Variances
We can have limits for
• WIP
• Defects
• Team size
• Velocity
• Time-boxes
• Others?
We can analyze variances to understand
the special causes of variation
4. Focus on Trends
• Trends matter more than values
• Trends and spikes are only the symptoms
• Trends are leading indicators
Images from MountainGoat Software
5. Improve quality
◦ Quality is measured by passed tests and
customer acceptance
◦ Quality is defined by Definition of Done
and Acceptance Criteria
◦ Defect trends to use in Agile projects
Number of open defects
Escaped defects
Defect Cycle Time
Defect spill-over
◦ Test Automation is critical!
6. Defect Injection and
Resolution
What is the trend of total Active Defects?
Image from MountainGoat Software
8. Escaped Defects
• Used to measure the quality of delivered
code
Calculation:
9
◦ For all releases, find 8
7
all defects related to
6
the release found
5
after the release date
4
3
◦ Add up all the defects 2
1
◦ Can be captured per
0
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
day / week / month /
… or per sprint / Escaped Defects over Time
release
9. Defect Cycle Time
• Rapid resolution of defects is important
• Average bug-fix time can also be tracked for
different priority bugs
Defect Cycle Time
20
18
16
Desired Threshold
14
12
Hours
10
8
6
4
2
0
1 2 3 4 5 6 7 8 9 10
Sprint No.
10. Defect Spill-over
• Resolution of defects for a sprint story within the sprint is
important
• Definition of Done can include a criteria like “No P1 defects”
• We can also tracked whether spilled-over defects are being
closed in the next sprint
Defect Spill Over
9
8
7
No of defects
6
5
4
3
2
1
0
1 2 3 4 5 6 7 8 9 10
Sprint No
14. What does this CFD say?
Not started
Started
Completed
Too much Work-In-Progress!
Image from MountainGoat Software
15. Identifying bottlenecks with a
CFD
1 2 3 4 5 6 7 8 8 10 11 12 13
Day
Where is the bottleneck?
In DB Procs – After the widening Analysis phase
Image from http://www.soliantconsulting.com
17. Earned Value Analysis
PV – Planned Value
EV - Earned Value
AC - Actual cost
BAC - Budget @
Completion
EAC - Estimate @
Completion
ETC - Estimate to
CompletionCost Variance (CV) = EV – AC
VAC- Variance @ Completion (SV)
Schedule Variance = EV – PV
Cost Performance Index (CPI) = EV / AC
Schedule Performance Index (SPI) = EV / PV
Estimate at Completion (EAC) = BAC / CPI or
= AC + BAC – EV
Estimate To Complete (ETC) = EAC – AC
Variance at Completion (VAC) = BAC – EAC
18. EVM - Interpretation
PV EV AC Schedule Cost
$1 $1 $1 On schedule On cost
$2 $2 $1 On schedule Under cost
$1 $1 $2 On schedule Over cost
$1 $2 $2 Ahead of schedule On cost
$1 $2 $3 Ahead of schedule Over cost
$1 $2 $1 Ahead of schedule Under cost
$3 $2 $1 Behind schedule Under cost
$2 $1 $3 Behind schedule Over cost
$2 $1 $1 Behind schedule On cost
19. EVM - Progress
EV PV AC
CV
SV
Ref: Earned Value and Agile Reporting
Anthony Cabri, Mike Griffiths
20. Wastage
Any time not dedicated to working on the
highest priority story / task on the sprint
backlog
Should be tracked over time and by cause
Wastage (in hrs) Wastage (in hrs)
60 160
140
50 120
100
40 80
60
30 40
20
20 0
10
0
1 2 3 4 5 6 7 8 9 10 11 12
23. Metrics to Avoid
• Velocity comparisons
• Productivity e.g. Story Points per day
• Effort Estimation accuracy
• Number of stories completed
• LOC/Function Points
• Code quality