Practical agile analytics: Measure predictability and quantify risk with cycle time

SagePath Technologies, LLC Copyright 2017
Practical Agile Analytics:
MEASURE PREDICTABILITY AND QUANTIFY RISK
WITH CYCLE TIME
1

One of the promises of Agile is that SW development
teams, and teams of teams, will be more predictable.
Ok, but...
 Why does predictability matter?
 How do I quantify it?
SagePath Technologies, LLC Copyright 2017 2
What’s in these slides

 These slides focus on using user story cycle time data to measure the
predictability of Agile software development teams.
 The same data that gives us our predictability measure is then used
to quantify the likelihood of user story completion.
 Once we know the likelihood of user story completion, we can put a
number on the risk of missing software delivery commitments.
What’s in these slides

• Predictability in software development
• Why predictability matters
• Uncertainty and Agile team metrics
• Cycle time
• Cycle time data analysis
• In a nutshell: A few essentials to remember
• About the data visuals
Outline

Predictability and Software Development

When we say someone is predictable typically we mean:
 We have a pretty good idea of what we are going to get and when.
 We expect to see similar outcomes in the future.
Predictable people make promises they can keep:
 They do what they said they were going to do.
 They complete things when they said they would.
6
What do we mean by Predictability?

Predictable Agile teams:
 Consistently deliver on sprint commitments
 Typically deliver code that is working, tested, and maintainable
 As a team of teams, consistently meet program objectives
 There is high confidence this behavior will continue
Characteristics of Predictable Agile Teams

Why Does Predictability Matter?

 Are more teams needed? If so, how big should the teams be?
 Is there a risk of missing delivery commitments?
 Should outside vendors be considered?
 What delivery timeframe should be communicated to customers so they can
effectively plan out their testing, maintenance, and upgrades?
 What training is needed and when should that be scheduled?
Predictability and Software Deliveries
Knowing how much software is likely to be delivered in a
given timeframe helps answer key planning questions such as:

Predictability and Resource Allotment
The ability to
 Accurately estimate delivery dates
 Properly resource projects
 And with confidence, make commitments to customers
Directly impacts funding and investment decisions

Predictability is a key element to achieving a successful
scaled Agile process.
• Effective and efficient collaboration between teams requires trust that the
other teams will complete their work when they said they would.
• Large complex SW projects and systems often have multiple, far-reaching
dependencies.
• Predictability is fundamental to effective dependency management.
Predictability and Scaling Agile

Poor predictability of software delivery teams leads to:
 Missed delivery dates and missed revenue targets
 Low confidence in delivery commitments and loss of customer trust
 Erosion of trust between development teams and senior management
 Increased schedule pressure and a corresponding drop in quality
 Schedule delays and the associated financial consequences
 Re-planning and priority shifts that disrupt development and exacerbate
project risk
Poor Predictability is Insidious

Uncertainty and Agile Team Metrics

Software development is knowledge-based work that by its
nature always contains some level of uncertainty.
A few examples of uncertainties that can arise in a software project:
• Gaps in development team capabilities and skillsets
• The challenge of estimating work effort when innovation is required to get the job done –
How do you schedule a breakthrough?
• Ambiguity in requirements and unclear technical paths
• Test planning and execution – How much testing is enough?
Uncertainty Makes it Harder to be Predictable

 Typical Agile team metrics include things such as story points, velocity,
capacity, and task hours.
 These metrics can help teams understand how well they are doing with
respect to their past performance.
 At the program level these metrics can be difficult to interpret.
 It is often unclear how these metrics can be used to assess value delivery
and program level productivity.
15
Agile Team Metrics

One way to address this is to use a metric that
reflects actual work delivered at a macroscopic level.
Such a metric is Cycle Time.
16
Agile Team Metrics
Additional challenges with Agile team metrics can include:
- Unstable team velocities
- Inconsistent tracking of actual task hours
- Committing to user stories that are too big
- Story point estimation not being normalized across all teams

Cycle Time

What is Cycle Time?
Cycle Time:
The amount of elapsed time it takes for a given work
activity to be completed.

 It’s easy to measure
 It’s easy to understand
 It measures something real and important
 It’s a difficult metric to “game”
Cycle Time as a Metric

Cycle Times in an Agile Process
There are a number of different cycle time measures that can be
applied to an Agile/scrum process. A few examples include:
• Backlog-to-Ready
• Ready-to-In Process
• Backlog-to-Deployed
In these slides our interest is in the predictability of SW teams so we
will look at the amount of time spent on story implementation.
This is referred to as the time In-Process.

In subsequent slides the term cycle time refers to:
The number of days between when a team pulls a user story into a sprint to begin
work on it and when the story is accepted by the Product Owner.
In the parlance of work flow:
This is the elapsed time between the date a user story goes into the In-Process state
and the date it goes into the Accepted state.
In-Process Cycle Time

 It’s a direct measure of the time it takes to get something done.
 It doesn’t require detailed information from individual
developers such as actual task hours.
 It mitigates the variabilities of unstable velocities and non-
normalized story points.
 It applies to both scrum and Kanban teams.
Why Use the In-Process Cycle Time?

Cycle Time Data Analysis

• The data used for analysis is from an Agile team of teams.
• This team of teams uses an electronic tool to track workflow. Cycle time data is
easily extracted from this tool using an API.
• The cycle time data is for user stories started and completed in 2015 and user
stories started and completed in 2016.
• This team of teams made two significant process changes in January of 2016:
 They moved from 3 week sprints to 2 week sprints
 They adopted a scaled Agile process
About the Data

Some data scrubbing is often required before proceeding with
analysis. A few of the reasons for this include:
• Missing data points
• Data that needs reformatting. E.g. string data that should be numeric.
• Data that is suspect and may be in error
• Inconsistencies in the processes for gathering and input of some data points
The following user story categories were removed from analysis:
• User stories where the in-process date and the accepted date are the same.
• System test user stories.
• Stories flagged as obsolete.
Data Scrubbing

When it comes to data analysis, one of the first things a
person should do is plot the data and look at it.
 The idea is to get a general sense of what we are dealing with and see if there
are any obvious trends or unusual features.
 One good tool for doing this is a scatter plot.
 Sometimes it is also useful to include a corresponding histogram to help gauge
the density of the data.
Looking at the Data

About the Chart:
• The chart on the left is a scatter plot of user stories
started and accepted in 2015.
• Each circle corresponds to a specific user story.
• The x-axis is the date a user story was accepted.
• The y-axis is user story cycle time in days.
• To the right of the scatter plot is a cycle time histogram
of the 2015 user stories.
What does this scatter plot tell us?
• The first thing to note is how spread out the data is in
along the y-axis (i.e. cycle times are all over the place).
• Since most of the teams are using scrum, the expectation
is that story acceptance would be clumped near or
below the sprint duration of 21 days.
• The histogram confirms what we are seeing. A significant
number of stories took 20 or more days to complete.
User Story Scatter Plot

About the Chart:
• The chart on the left is a scatter plot of user stories
started and accepted in 2016.
• Each circle corresponds to a specific user story.
• The x-axis is the date a user story was accepted.
• The y-axis is user story cycle time in days.
• To the right of the scatter plot is a cycle time histogram
of the 2016 user stories.
What does this scatter plot tell us?
• Contrary to the 2015 scatter plot, in this scatter plot the
user stories are more closely grouped along the lower
part of the chart.
• The histogram shows a strong peak between 10 and 15
days. This correlates well with the change to 2 week
sprints in 2016.
User Story Scatter Plot

About the Chart:
• In this chart the 2015 and 2016 scatter plots have been overlaid.
This plot is referred to as a “splatter plot†”
• The diameter of the marker circle for each user story is scaled
according to the size of the story it represents.
What does this plot tell us?
• The first thing to note is that the maximum story size is much
greater in the 2015 data than the 2016 data.
• The larger user stories have cycle times significantly longer than
one or two sprints.
• Somewhat unexpected: there are a number of user stories that are
small in size, but also have cycle times significantly longer than one
or two sprints.
Spatter Plot
†The earliest reference to the term “splatter plot” that I am aware of is by Dennis
Sweitzer in his presentation When Projects go Splat: Introducing Splatter Plots.

Quantifying Predictability
Predictability is the likelihood that any particular user story will
be completed within a given time frame such as one sprint.
If we can measure this likelihood, we can quantify predictability.
This likelihood can be determined from the cycle time data by
calculating the cumulative percentage of story completion over
time.

Measuring Predictability
About the Chart:
• This chart shows the cumulative percentage of 2015 stories that
were accepted over time.
• Note that the x-axis is in units of sprints rather than days.
• In the ideal, all stories that were committed to in a given sprint are
completed within that sprint.
• At the 1 sprint line, the larger the percentage of completed stories
the closer the teams are to the ideal.
Of Note:
• Only about 43% of the user stories were completed within 1 sprint.
• After 2 sprints the completion percentage is at 74%.
— This means that over one quarter of the user stories took longer than 2
sprints to complete
• Based on this data, the likelihood that any particular story would be
completed within one sprint is well short of 50-50.
2015

Measuring Predictability
About the Chart:
• This chart overlays the cumulative percentage of accepted user
stories from 2015 and 2016.
A look at the numbers:
• In 2016 the acceptance percentage within 1 sprint is 73%. This is a
significant improvement from the 43% seen in 2015.
• After 2 sprints the 2016 completion percentage is above 90%.
• Between 2015 and 2016 the likelihood that a user story will be
completed within 1 sprint increased from 43% to 73%.
• In 2016 user stories were 67% more likely to be completed within
1 sprint than in 2015. This means SW delivery risk decreased in
2016.
• Between 2015 and 2016 the aggregate team of teams
predictability increased by a factor of about 1.7.
Improvement in
Predictability
2016
2015

In a Nutshell: A few essentials to remember

Predictability can be Measured and Quantified
By tracking cycle time we can quantify predictability.
In other words:
We can put a number on the likelihood that user story
commitments and program objectives will be met.

Quantifying Software Delivery Risk
Timely completion of user stories is a highly desirable objective.
Therefore, we can define SW delivery risk as:
Risk = 1 – (likelihood of on-time completion)
Once we quantify the likelihood of user story completion, we
can also quantify the corresponding project and program risk.

 When you want to help teams understand their own work trends,
focus on team metrics such as velocity.
 For forecasting elapsed time of software deliverables focus on work
delivered at a macroscopic level.
 Forecast the likelihood of timely Agile team deliveries by looking at in-
process cycle time.
 Cycle time allows us to quantify software project delivery risk.
Key Takeaways

About the Data Visuals

The Data Plots:
• The data was analyzed using Python 3.6 and pandas from the Python Data Analysis Library .
• The data was plotted using the Python Matplotlib plotting library.
Background Images:
All background images are licensed under the Creative Commons Zero (CC0) license from the following sources:
unsplash.com : slides 17, 18, 21, 22, 26, 30, 34-37
pixabay.com : slides 1, 3, 5, 8, 13, 33
pexel.com : slides 23, 24
Data Visuals

Practical agile analytics: Measure predictability and quantify risk with cycle time

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Practical agile analytics: Measure predictability and quantify risk with cycle time

Similar a Practical agile analytics: Measure predictability and quantify risk with cycle time (20)

Último

Último (20)

Practical agile analytics: Measure predictability and quantify risk with cycle time