Our first journey in the engineering delivery metrics World

1. Exploring
Team Metrics
(a first naive tentative)
Shark Dev Team

2. Without data,
you're just another person
with an opinion
William Edwards Deming

3. Agenda
● What are engineering
delivery metrics?
● Lead and cycle time
● About responsiveness and
performance
● Calculate WIP limit

4. What are engineering
delivery metrics?

5. Start doing
Refinement

6. Start doing
Refinement

7. Start doing
Refinement

9. Delivered to the customer
Delivered to the customerTech Team started working on it
Touch point between Product and Tech Team

10. What is Lead Time? “Lead time” is the period
between a new task’s
appearance in your workflow and
its final departure from the
system.

11. In our case, we want to
investigate on the time between
two Jira workflow transitions:
from “doing” status to “trying it in
prod” status.
This measure is called “cycle
time”.

12. What is Cycle
Time? Cycle time begins at the moment
when the new arrival enters “in
progress” stage and somebody is
actually working on it.

13. Lead Time vs Cycle Time

14. How to export it
from Jira?
● Jira plugins

15. ● Jira plugins
We aren’t able to install plugins
at the moment.
How to export it
from Jira?

16. ● Spider
How to export it
from Jira?

17. Spider? We did it

18. ● Spider
We can call Jira Rest API using
OAut 2.0 user token.
How to export it
from Jira?

19. Jira Rest API data exporter

20. ● Only issues done and
resolved
● Exclude Wallman issues
● From 1st June 2020 to 15th
November 2020
● Start date is
min(transitions(“doing”))
● End date is
min(transitions(“trying it in
prod”))
Export criteria

21. ETL + analysis with Jupyter Notebook

22. ETL
Extract, transform
and load
● Normalize data
● Merge Jira export with
transitions data
● Calculate “Cycle time”
● Focus on on data from 1st
June 2020 to 15th November
2020
● Remove outliers and wrong
data
● Anonymize data

23. Task Size Distribution

24. Task Size Distribution
June - July 2020 June - November 2020

25. Task Size Distribution
June - July 2020 June - November 2020

26. Task Size Distribution
June - July 2020 June - November 2020

27. Task Size Distribution
June - July 2020 June - November 2020

28. Task Size Distribution
June - July 2020 June - November 2020

29. Track Responsiveness (Cycle Time)

30. This is our Cycle Time in days

31. This is our Cycle Time in days

32. This is our Cycle Time in days
June - July 2020June - November 2020

33. This is our Cycle Time in days
June - July 2020June - November 2020

34. Cycle Time in days per Story Points

35. Cycle Time in days per Story Points

36. Cycle Time in days

37. Cycle Time in days
Outlier?

38. Cycle Time per Assignee and Story Points

39. Cross Tabulation: Story Points/Assignee (count tasks)

40. Cross Tabulation: Story Points/Assignee (count tasks)

41. Cross Tabulation: Story Points/Assignee (avg cycle time)

42. Cross Tabulation: Story Points/Assignee (avg cycle time)

43. Cross Tabulation: Story Points/Assignee (avg cycle time)

44. Track Performance (Throughput)

45. Task Started and Completed per Week

46. Task Started and Completed per Week

47. Task Started and Completed per Week
Throughput increase

48. Task Started and Completed per Week
Smaller tasks? Throughput increase

49. Task Completed per Week

50. Throughput

51. Throughput
avg

52. Recap
● Cycle time is about
responsiveness
● Throughput is about
performance
But...

53. When a measure becomes a target,
it ceases to be a good measure.
Goodhart's law

54. In other words, when we set one specific goal,
people will tend to optimize for that objective
regardless of the consequences.
Goodhart's law unintended consequences

55. Little’s Law

56. Team

57. TeamArrivals

58. TeamArrivals
Tasks in queue
Tasks in service

59. TeamArrivals Departures
Tasks in queue
Tasks in service

60. We can consider the Team like a
stable queue system (steady
state condition).Team

61. We can consider the Team like a
stable queue system (steady
state conditions).Team

62. L = 𝝺W
Little’s Law

63. L = 𝝺W
Little’s Law
L := average number of tasks in the queue system

64. L = 𝝺W
Little’s Law
L := average number of tasks in the queue system
𝝺 := average service time in the queue system for a task

65. L = 𝝺W
Little’s Law
L := average number of tasks in the queue system
𝝺 := average service time in the queue system for a task
W := average number of tasks arriving per unit time

66. The long-term average number L of customers in a
stationary system is equal to the long-term average
effective arrival rate λ multiplied by the average time W
that a customer spends in the system.
The relationship is “not influenced by the arrival process
distribution, the service distribution, the service order,
or practically anything else”.

67. L = 𝝺W
Little’s Law in Kanban World
L := WIP
𝝺 := Throughput
W := Cycle Time

68. In Kanban terms, Little’s Law is
expressed a little differently, but
the idea is the same:
WIP = Throughput * Cycle Time
If we imagine the Kanban board
as the store, WIP is equivalent to
the number of customers inside
at any time, throughput is the
rate of customers passing
through the store and cycle time
measures how long each one
spends inside the system.
Little’s Law

69. This means that if two of the
three values are known, the third
value can be calculated without
knowing anything else about the
tasks, team or project:
● WIP = Throughput * Cycle Time
● Cycle Time = WIP/Throughput
● Throughput = WIP/Cycle Time
Little’s Law

70. Little’s Law for Shark Dev Team

71. Let’s assume that:
Avg. WIP = Avg. Throughput * Avg.
Cycle Time
● Avg. Throughput (weeks) =
5.14
● Avg. Cycle Time (days) = 7.20
● Avg. Cycle Time (weeks) =
7.20/7 = 1.03
So that:
Avg. WIP = 5.14 * 1.03 = 5.29
Little’s Law

72. The power of Little’s Law to
Kanban teams is not its ability to
predict WIP, Thoughput or Cycle
Time.
The true power lies in its ability
to influence team behavior with
its underlying assumptions.
In other words, if you want to:
● increase Throughput then
limit WIP;
● speed up the process, i.e.
reduce Cycle Time, then
once again limit the WIP.
The power of
Little’s Law

73. About task estimation

74. It always takes longer than you expect,
even when you take into account
Hofstadter's Law
Douglas Hofstadter

76. Don’t estimate.
Measure and forecast!

77. Don’t estimate.
Measure and forecast!
Only with discipline and data
we can forecast!

78. Stay hungry,
stay data-driven!

79. Proposal for next steps
[1] Start using new Jira Workflow
[2] Improve Team metrics using Jira plugins
[3] Introduce new Team metrics (performance, quality, etc.)
[4] Define Team metrics goals
...
[..] Introduce team metrics on Product team side?

80. References
[1] Agile Reloaded, Ferdinando Santacroce, 2020.
[2] Throughput and Cycle Time [2020-09-27].
[3] Measure Your Lead Time And Cycle Time [2020-09-27].
[4] Understanding Agile Team Metrics: Measure Many Things [2020-09-27].
[5] Unintended Consequences and Goodhart's Law | by Will Koehrsen [2020-09-27].
[6] Stable Systems: Little's Law and Kanban [2020-09-27].
[7] Little’s Law – the basis of Lean and Kanban [2020-09-27].
[8] Chapter 5 Little's Law [2020-09-27].
[9] Accelerate. Nicole Forsgren, Jez Humble, Gene Kim.
[10] Estimates. Vasco Duarte.

81. Meeting OKR
Objective: Share our experiments about delivery metrics
as measured by
Key Result 1: Finish meeting on time
Key Result 2: Answer to all questions
Key Result 3: Discuss next steps