The document discusses different types of metrics that can be used to measure success, including activity-based, output-based, and outcome-based metrics. It also provides examples of metrics that could be used to measure different phases of a project such as planning, development, design, testing, and maintenance. Finally, it discusses challenges with measuring success, including differences between plans and actual outcomes.

1. Metrics that matter

2. Waterfall
Agile
Mobius
Ac#vity
based
tracking
Output
based
tracking
Outcome
based
tracking
What do we measure?
Planning
Development
Design
Testing
Maintain
Launch

5. OPTION A
Improve network
Increase # checkout conversions
Baseline:15% Target:30%
Current:18%
5
Op#on
Value
Risk
Cost
Impact
Op#on
A
?
M
M
?
Op#on
B
?
H
H
?
Op#on
C
?
L
L
?
People start the buying process
but don’t complete it
HYPOTHESIS
Improving the
page load speed
will help
conversions
Increase checkout
conversions Improve page
load speed
DATA
85% of users
abandon
checkout
How to create outcomes case study
OPTION B
Build a local
datacenter
DATA
86% of users
abandon
checkout when
page loads times
= > 4 seconds
CREATE OPTIONS
Generate ideas on how to
fix the page load speed
INSIGHT
The further users
are from the
Datacenter, the
slower the page
loads OPTION C
Compress page
elements
IMPLEMENT
Image compression
EXPERIMENT
Test one server with
0.002% split test
RESEARCH
Run network
diagnostics with a
trace route
NEXT IMPROVEMENTS
What else can we do to
improve the conversion
rate?
IMPLEMENT:
Add instrumentation
and exit survey to
pages
OPTION A
Fix the network
OPTION
Find out why
people are
leaving the
service

6. Tom Gilb’s ‘Ilities’
( Reliability
( Availability ------ ( Maintainability
( ( Integrity ------------ ( Threat
( ( Security
(
(
( ( Connectability
( ( Flexibility------------ ( Tailorability ---( Extendibility
Quality ------------ ( Adaptability ----- ( ( Interchangeability
( (
( ( Upgradeability ----- ( Installability
( ( Portability
( ( Improvability
(
( (
(
Entry level experience
( ( Training attribute
( Usability --------- ( Handling ability
( Likeability
( Demonstratability
( Financial saving
Resource saving ---( Time saving
( Effort saving
( Equipment saving
( Throughput
Workload capacity ( Response time
( Storage capacity
Quality outcomes

7. How contracts measure success
PLAN
OUTPUT
ACTIVITIES
OVERARCHING
GOAL
CONTRACT
CONTRACT
Measurement
of
success

8. Why there is friction
ACTUAL
OUTCOMES
ACTIONS
PLANS
OVERARCHING
GOAL
KNOWLEDGE
GAP
The
difference
between
what
we
would
like
to
know
and
what
we
actually
know
EFFECTS
GAP
The
difference
between
what
we
expect
our
ac#ons
to
achieve
and
what
they
actually
achieve
ALIGNMENT
GAP
The
difference
between
what
we
want
people
to
do
and
what
they
actually
do
Source:
‘The
Art
of
Ac#on’
by
Stephen
Bungay

9. Medium
to
large
projects
experience
25-­‐35%
requirements
change.
Capers
Jones
on
So,ware
Development
Our
customers
say
that
if
it
weren’t
for
the
contract
making
it
so
expensive
and
#me-­‐consuming,
they
would
have
had
more
changes,
and
could
have
built
beSer
products.
Locking in scope, locks out opportunities

10. FRAMEWORK
AGREEMENT
Warran#es
IP
Constraints
($,
Time,
Regulatory)
Statement
Of
Target
Outcome
(SOTO)
$
The contracts have to be Agile

11. Snowden’s
Cynefin
model
Cause
and
effect
rela#ons
repeatable,
perceivable
and
predictable
Cause
and
effect
are
only
coherent
in
retrospect
and
do
not
repeat
Cause
and
effect
separated
over
#me
and
space
CHAOS
SIMPLE
(Known)
COMPLEX
(Unknowable)
COMPLICATED
(Knowable)
No
cause
and
effect
rela#onships
perceivable

12. Benefits of Agile

13. Questions?

14. Nearly 50 countries use Systematic’s solutions
Systematic’s offices
Manage
complex
projects
to
succes
using
CMMI,
Lean
and
Scrum
hSp://www.systema#c.com
Public Sector Intelligence &
National Security
Healthcare Defence

15. Standard Project Status Report
Managing normal projects is a prerequisite to manage large projects
CMMI and Lean provides a solid foundation for both
Status
Analysis
of
measures
Risk
top
3
Ac#ons
Risk
Measure
12
std
KPI’s

16. Example: Flow of story implementation and fix-time after failed build
Productivity is optimized:
• Flow of work - maximized
• Fix-time of failed builds - minimal
• Sprint test and release - minimal
à Variance reduced
Measures
inspired
from
Lean
Fix time for failed builds
Flow of story implementation

18. Questions?

19. You get what you measure
“Perhaps what you measure is what you get.
More likely, what you measure is all you get.
What you don’t (or can’t) measure is lost.”
H Thomas Johnson (2006)

20. Friction
“Fric7on
manifests
itself
when
human
beings
with
independent
wills
try
to
achieve
a
collec7ve
purpose
in
a
fast-­‐changing,
complex
environment
where
the
future
is
fundamentally
unpredictable.”
‘The
Art
of
Ac#on’
by
Stephen
Bungay,
2011
• Fric#on:
Ø An
accumula#on
of
innumerable
peSy
circumstances
e.g.
uncertain#es,
errors,
accidents,
technical
difficul#es,
the
unforeseen
Ø …
which
impact
on
decisions,
morale
and
ac#ons
Ø …
thereby
preven#ng
you
from
achieving
your
goal

21. Causes of friction
Human
finitude
• Services
are
performed
by
people:
Ø People
are
imperfect
Ø People
are
intrinsically
unpredictable
Ø People
have
different
interpreta#ons
• The
environment
is
complex
and
chao#c:
Ø The
environment
is
non-­‐linear
and
fundamentally
unpredictable
Ø External
factors
contribute
towards
a
change
in
the
environment
Ø Even
if
the
environment
is
stable,
some
of
the
effects
of
our
ac#ons
will
be
unintended.

22. Pay with a one
step pay
button
Step
Search for
book
Step
Decrease time to pay
TO
Decrease time to find
a book
TO
Find one book and get
it shipped
Goal
Setup one step pay
option
Goal
Call and pay
Step
Enter payment
details
Step
Save payment
details
Step
Our
team
tries
to
get
feedback
and
hit
the
outcomes
with
the
least
amount
of
effort.
Goal
mapping
helps
us
set
a
goal
and
find
the
simplest
path
to
reach
it.
22

23. Outcomes Op#ons
In
Progress Done
23
Decrease
page
load
#me
Setup local
datacenter
Option
Cache page
elements
Option
Decrease
#me
to
find
products
Redesign
search results
Option
Clickstream
analysis
Research
Datacenter
split
test
Australia
Research
Test
caching
one
page
Research
Split
test
redesign
Research
Trace
route
analysis
Research
Teams
track
outcomes
and
op#ons
visually
during
delivery.

24. How
much
progress
did
we
make?
Decrease
#me
to
find
products
Baseline:
7
minutes
Target:
<
3
minutes
Current:
5
minutes
Decrease
page
load
#me
Baseline:
12
seconds
Target:
<
4
seconds
Current
#me:
9
seconds
We
saved
2
minutes
We
reduced
page
load
speed
by
3
seconds
24

25. We
are
constantly
adap#ng
our
model
based
on
feedback.
When
we
see
the
value
diminish,
we
can
choose
other
op#ons
or
outcomes
to
explore
25
x
Point
of
diminishing
returns
Time
Outcomes
(value)

26. 26
Value
Type
Value
Driver
Scale
-­‐
What
to
Measure
Scale
Defini#ons
Meter
-­‐
How
to
Measure
Performance
Reliability
Mean
#me
between
failure
over
#me
period
Time
Period:
current
month,
trailing
12
months
Monitoring
system
Performance
Responsiveness
Elapsed
#me
between
user
reques#ng
and
receiving
informa#on
for
percen#le
of
transac#ons
Time:
seconds
;
Percen#le:
90th
;
Transac#ons:
<Define
specific
transac#ons>
Performance
tes#ng
system
Profitability
Increase
Revenue
Total
revenue
over
#me
period
Time
Period:
current
month,
trailing
12
months
Financial
system
Delivery
Improve
Customer
Sa#sfac#on
Average
scores
pertaining
to
survey
ques#ons
for
current
customers…
Survey
Ques#ons:
<define
specific
ques#ons>
or
Net
Promoter
score
Survey
system
Quality
Reduce
Incident
Rate
Rate
of
new
incidents
over
#me
period
Time
Period:
current
month,
trailing
12
months
Report
from
issue
tracking
system
Example

27. A comparison of the activities
PRODUCT
BACKLOG
Waterfall
Project
Agile
Project
REQUIREMENTS
Discrete
modules
of
somware
Integra#on
late
in
the
process
Each
sprint/itera#on
builds
on
the
earlier
sprints/itera#ons

28. Outcomes
are
the
desired
results
that
deliver
value.
E.g.
Increase
checkout
conversion
rates
by
19%
What do you want to improve?
©
Gabrielle
Benefield
28

29. Create
target
outcomes
Set
a
baseline
e.g.
Currently
only
11%
of
people
complete
checkout
Set
a
target
e.g.
30%
of
people
complete
checkout
Baseline:
11%
Target:
30%
Give
it
a
name
in
verb
+
noun
phrase
form
Increase
checkout
conversion
rates
29

30. Set
target
outcomes
for
the
‘ili#es’
30
Scalability
Must scale to 3 million
users.
Availability
System must have a
minimum of 99.9
uptime
Usability
Must be easy to use
and fast to learn.
Interoperability
Must integrate with all
customer systems.

31. 31
Pay with a one
step pay button
Step
Cache page
elements
Step
Search for
book
Step
Decrease time to
pay
TO
Decrease time to
load pages
TO
Decrease time to
find a book
TO
Find one book and
get it shipped
Goal
Setup one step pay
option
Goal
Call and pay
Step
Read details
Step
Create
a
Goal
map
Start
with
the
Goal
and
work
back
to
find
the
least
amount
of
steps
needed
to
reach
the
goal
Enter payment
details
Step
Save payment
details
Step
Decrease the page
load times
Goal
Setup 1 server
and split test
Step

32. Which
op#ons
should
we
do
first?
We
can
guess,
or
we
can
test
with
an
experiment
?
32
Decrease
page
load
#me
Setup local
datacenter
Option
Test caching elements
on the payment page
Experiment
Cache page
elements
Option
Setup one server test
with 1% of traffic
Experiment

33. Create
experiments
?
33
Take
a
target
outcome
Design experiments to
test ideas or de-risk
your work
Experiment

34. Outcomes Options In Progress Done
Visually
track
outcomes
and
op#ons
during
delivery
©
The007model.com
34
Decrease
page
load
#me
Setup local
datacenter
Option
Cache page
elements
Option
Decrease
#me
to
find
products
Redesign
search results
Option
Clickstream
analysis
Research
Datacenter
split
test
Australia
Research
Test
caching
one
page
Research
Split
test
redesign
Research
Trace
route
analysis
Research

35. How
much
progress
did
we
make?
Decrease
#me
to
find
products
Baseline:
7
minutes
Target:
<
3
minutes
Current:
5
minutes
Reduce
customer
complaints
Baseline:
250
calls
per
day
Target:
<
150
calls
per
day
Current:
190
calls
per
day
We
saved
2
minutes
We
reduced
calls
by
60
per
day
©
The007model.com
35
Redesign
checkout process
Option

36. Snowden’s
cynefin
model
Cause
and
effect
rela#ons
repeatable,
perceivable
and
predictable
Cause
and
effect
are
only
coherent
in
retrospect
and
do
not
repeat
Cause
and
effect
separated
over
#me
and
space
CHAOS
SIMPLE
(Known)
COMPLEX
(Unknowable)
COMPLICATED
(Knowable)
No
cause
and
effect
rela#onships
perceivable

37. Outcomes
are
the
desired
results
that
deliver
value.
E.g.
Increase
checkout
conversion
rates
by
19%
What do you want to improve?
©
Gabrielle
Benefield
37

38. We
are
constantly
adap#ng
our
model
based
on
feedback.
When
we
see
the
value
diminish,
we
can
choose
other
op#ons
or
outcomes
to
explore
38
x
Point
of
diminishing
returns
Time
Outcomes
(value)

39. READY and competence match
When work allocated to sprint is READY, flow and stability is achieved
0,00
20,00
40,00
60,00
80,00
100,00
120,00
140,00
160,00
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
Actual effort
Ready NOT Ready
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
80,00%
90,00%
100,00%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
Flow
Ready NOT
Ready
Flow Actual effort spent
Projects measures how ”READY” a sprint plan is
Measures of key competencies
needed
in
project
Meeting READY criteria creates stability

40. Cost of
change
Time
Low
High
Plan
Develop
Design
Test
Launch
The
point
where
we
have
the
least
informa#on,
the
contract
is
signed
locking
in
that
ignorance
for
the
whole
project.
When
the
customer
sees
what
has
been
built,
when
tes#ng
begins.
When
the
cost
of
change
and
fixing
defects
is
highest.