1. Enterprise Metrics on Agile Sagi Smolarski Director, Software Development Sagi.Smolarski@itg.com [CC] Daniel Goodwin via Wikimedia Commons

2. Disclaimers The information contained in this presentation is provided for informational purposes only. All trademarks, service marks, and trade names not owned by ITG are owned by their respective owners. …in other words, the data presented has been modified and does not represent real data for ITG teams.

3. Agenda Why do we need metrics for agile? How do we generate those metrics? Which metrics do we look at? Pros and cons of looking at those metrics.

4. Investment Technology Group NYSE: ITG www.itg.com Leading provider of trading solutions Trading Front-Ends Trading Algorithms Research (fundamental analytic, pre/post-trade) Trading desks Electronic trading connectivity Liquidity pool Development operation: 300 Developers, 100 QA Analysts, ~60 teams

5. Iteration Metrics Quality Metrics Process Baseline ITG’s Agile Transition Timeline ITG Triton XP Enterprise rollout plan Rally +Scrum, +Lean 90% of teams have been trained and are using Rally Best Execution Management System XP Pilot 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Massive transition Informal, spotty, inconsistent adoption

6. Our Process Baseline – How We Expect Teams to WorkExcerpt

7. The journey to Agile is a long, winding road… Are we moving forward? [CC] – Sten via Wikimedia Commons

8. Why Measure? “If you cannot measure it you cannot improve it.” Lord Kelvin

10. How can we improve?

12. Which teams need our help?Executives (Govern & Steer): What are we getting out of the agile transition? Are teams sticking to our process baseline and to enterprise initiatives? Is productivity/quality improving?

13. Team A Teams Process HealthData is readily available in Rally Team B Velocity Velocity vs. Burndown/Story Acceptance Burndown/Story Acceptance vs. Isn’t this enough?

14. Why Metrics? - Dumbing down … This is too complex for team members to interpret and monitor… Are the charts okay?

15. Why Metrics – Scaling up How do we watch 80 teams? Lines of products? The whole enterprise? ?

16. Types of Metrics Qualitative Satisfaction of stakeholders Product Management, Client Services, Product Support (deployment, service), Delivery team Teams adherence to practices Agility questionnaire Quantitative Quality metrics Process health metrics Time-to-Market We’ll be focusing on these

17. What Would We Want to Measure? Productivity Quality ROI Satisfaction Some of these are not easy to measure so we have to find proxies

18. What We Are Actually Measuring Satisfaction Quality Process As a partial proxy for productivity Stakeholders Surveys Production defects Defects Debt Work-in -Progress Velocity Stability Full Completion or work Gradual acceptance of work Churn Small Stories Practices Radar map

19. How We Generate Metrics How do we obtain data?Rally’s Web Services API Rest and SOAP Access to virtually all data in the tool Users, Releases, Iterations, Stories, Tasks, Defects, Test cases… Ruby toolkit… or any WS library We use mostly C# Automated generation, monthly How do we make sense out of mounds of data?

20. The Metrics

21. Quality Metrics

22. Quality MetricsDefects Found in Production Metric #1 Production Defects Goal Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb 2010 2011 Team Level & Enterprise Level

23. Metric #2 Quality MetricsFix-Bugs-First If you are fixing defects first As part of existing development, within the iteration Before new development for regression defects Then… The number of defects in your backlog should be small The age of open defects should be low Together… Defects Debt should be low Defects Debt = [# open defects] x [avg age of open defects]

24. Quality MetricsProject-Level Defects Debt Metric #2

25. Quality MetricsEnterprise Defects Debt Metric #2 Defects Debt

26. Process Health Metrics

27. Team Process HealthAgility Questionnaire

28. Scope CreepChurn Metric #3 Iteration Cumulative Flow No Scope Creep Plan Estimates Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Churn (Scope variation) = 0

29. Work in Progress & Scope Creep Metric #3 Iteration Cumulative Flow Plan Estimates Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Churn = 30%

30. Work in Progress & Scope Creep Metric #3 Iteration Cumulative Flow Plan Estimates Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Scope Creep: Team Cannot Commit Disruptive (task switching?) Less efficient Day 12 Day 13 Day 14 Churn = 30%

31. Work in ProgressWIP Average Metric #4 Iteration Cumulative Flow No Scope Creep Work In Progress Plan Estimates Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Planned WIP average = 10% In Progress Accepted

32. Work in Progress & Scope Creep Metric #4 Iteration Cumulative Flow Plan Estimates Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Planned WIP average = 70% In Progress Accepted

33. Work in Progress & Scope Creep Metric #4 Iteration Cumulative Flow Large WIP = Long Cycle Time Risk of not completing work within iteration Plan Estimates Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Planned WIP average = 70% In Progress Accepted

34. Full Acceptance – Final Acceptance % Metric #5 Iteration Burn-up 100% Accepted Plan Estimates (Points) 50% Acceptance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Day 12 Day 13 Day 14 Accepted Acceptance rate on the last day = 100%

35. Full Acceptance Metric #5 Iteration Burn-up 100% Accepted Plan Estimates (Points) 50% Acceptance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Day 13 Day 14 Day 12 Day 13 Day 14 Accepted Acceptance rate on the last day = 55%

36. Full Acceptance Metric #5 Iteration Burn-up 100% Accepted Plan Estimates (Points) 50% Acceptance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Day 13 Day 14 Partial Completion Hard to plan Greater cycle time Inefficient Day 12 Day 13 Day 14 Accepted Acceptance rate on the last day = 55%

37. Gradual Acceptance – 50% day Metric #6 Iteration Burn-up 100% Accepted Plan Estimates (Points) 50% Acceptance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 >50% points accepted Day 12 Day 13 Day 14 Accepted Day of 50% Acceptance = 8days/14days = 57%

38. Gradual AcceptanceFull Acceptance Metric #6 Iteration Burn-up 100% Accepted Plan Estimates (Points) 50% Acceptance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Day 12 Day 13 Day 14 Accepted Day of 50% Acceptance = 14/14 = 100%

39. Gradual Acceptance Metric #6 Iteration Burn-up 100% Accepted Plan Estimates (Points) 50% Acceptance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11 Day 12 Day 13 Day 14 Day 12 Day 13 Day 14 Accepted Late Acceptance High risk of not completing work Late feedback on work Day of 50% Acceptance = 14/14 = 100%

40. Velocity StabilityVelocity Variation Metric #7 Velocity 60 50 40 30 20 10 Iteration 23 Iteration 24 Iteration 25 Iteration 26 Iteration 27 Iteration 28 Iteration 29 Iteration 30 Iteration 31 Iteration 32 Day 12 Day 13 Day 14 Accepted within iteration Velocity is fairly stable Accepted after iteration Variation = 7% Never accepted

41. Velocity StabilityVelocity Variation Metric #7 Velocity 60 50 40 30 20 10 Iteration 23 Iteration 24 Iteration 25 Iteration 26 Iteration 27 Iteration 28 Iteration 29 Iteration 30 Iteration 31 Iteration 32 Day 12 Day 13 Day 14 Accepted within iteration Velocity is unstable Accepted after iteration Variation = 87% Never accepted

42. Velocity Stability Metric #7 Velocity 60 50 40 30 20 10 Iteration 23 Iteration 24 Iteration 25 Iteration 26 Iteration 27 Iteration 28 Iteration 29 Iteration 30 Iteration 31 Iteration 32 Day 12 Day 13 Day 14 Unstable Velocity Low predictability Hard to plan Inefficient? Accepted within iteration Velocity is unstable Accepted after iteration Variation = 87% Never accepted

43. Defects Debt Velocity Stability Churn % Completion WIP, Day of 50% Acceptance, Story Sizing Our Process Baseline… and metrics we can collect

44. Team DashboardAll together now… Day 12 Day 13 Day 14 Day 12 Day 13 Day 14 Day 12 Day 13 Day 14 Day 12 Day 13 Day 14 Day 12 Day 13 Day 14 Day 12 Day 13 Day 14

45. Teams Process HealthEnterprise Process Health Map (Probably) Healthy Process Line of Business #1 Line of Business #2 Line of Business #3 Line of Business #4 Line of Business #5 ProjY Proj k Proj X Proj B Proj N Projא Project A Prohj Z Proj E Projב Proj 1 Proj C Proj W Proj ג Proj 2 Proj ד Proj 3 Projה Proj D Proj 4 Proj 5 Proj M Proj L Projו (Possibly) Unhealthy Process

46. Problems with Metrics Metric-Driven Dysfunctions Teams misreporting defects Not tracking unpredictable work in Rally We can limit these by not using these metrics to reward, reprimand False positives, False negatives Some metrics are sensitive to how Rally is being use These metrics don’t cover some important aspects of teams’ process Metrics should be treated as an indication that further examination is needed Hey, I just figured out how we can double our quarterly sales. From now on, each quarter will last six months.

47. Looking at Next… Small Stories Time to Market Trends By Simon A. Eugster (EigenesWerk) [GFDL (www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

48. Q&A Q&A How are you measuring yourselves?