"What?", "So what?", "NOW WHAT?" How to influence people and accomplish change

@LMaccherone @TheAgileCraft
What?
So what?
NOW WHAT?
Using metrics to
influence developers,
executives, and
stakeholders
Topic presented by:
Larry Maccherone
@LMaccherone

What?
So what?
NOW WHAT?

What?

So what?

What?
Visualization is like photography.
Impact is a function of focus,
illumination, and perspective.
Credit: Edward Tufte

NOW WHAT?

Don’t
Launch!

Prevent your own
disastrous decisions

Larry Maccherone
@LMaccherone

Cognitive bias
works against good decisions

@LMaccherone @TheAgileCraft @LMaccherone @TheAgileCraft
We don't see things
the way they are.
We see things
the way we are.
~The Talmud

Next slide is a movie
click to play

Denying the Evidence

Truths about cognitive bias
1. Very few people are immune to it.
2. We all think that we are part of that
small group.
3. You can be trained to get much, much better.
Douglass Hubbard – How to Measure Anything
4. We do a first-fit pattern match. Not a best-fit pattern
match. And we only use about 5% of the information
to do the matching.
5. We evolved to be this way (survival trait).

An example of overcoming
cognitive bias

Trained/Calibrated
Untrained/Uncalibrated
Statistical Error
“Ideal” Confidence
30%
40%
50%
60%
70%
80%
90%
100%
50% 60% 80% 90% 100%
25
75 71 65 58
21
17
68 152
65
45
21
70%
Assessed Chance Of Being Correct
PercentCorrect
99 # of Responses
We are overconfident when assessing our own uncertainty
But, training can “calibrate” people so that of all the times they
say they are X% confident, they will be right X% of the time
Copyright HDR 2007
dwhubbard@hubbardresearch.com

Equivalent Bet calibration
What year did Newton published the Universal Laws of
Gravitation?
Pick year range that you are 90% certain it would fall within.
Win $1,000:
1. It is within your range; or
2. You spin this wheel and it lands green
Adjust your range until 1 and 2 seem equal.
Even pretending to bet money works.
90%
10%

Agile Teams, Programs, and
Portfolios
benefit from similar
calibration exercises

How to avoid cognitive bias in decision making
 Don't focus on consensus.
Ritual dissent is a much more successful approach.
 “But that doesn’t explain _______”.
An FBI agent knew that some folks were being trained to fly
but not take off and land.
 Assign someone the role of devil’s advocate.
Israel’s 10th man.
In other words… Really consider the other ALTERNATIVES

Criteria for
great visualization
Credits:
Edward Tufte (mostly)
Stephen Few
Gestalt School of Psychology

1. Answers the question, "Compared with what?”
(So what?)
 Trends
 Benchmarks

2. Shows causality, or is at least informed by it
The primary chart used by the
NASA scientists showed O-ring
failure indicators by launch date.
Tufte's alternative shows the same
data by the critical factor,
temperature.
The fateful shuttle launch occurred
at 31 degree. Tufte's visualization
makes it obvious that there is great
risk for any launch at temperatures
below 66 degrees.

3. Tells a story with whatever it takes
 Still
 Moving
 Numbers
 Graphics
And …
 Maybe some fun

4. Is credible
 Calculations explained
 Sources
 Assumptions
 Who (name drop?)
 Drill-down
 How?
 Etc.

5. Has business value
(or value in it’s social context)
The ODIM framework
O U T C O M E
D E C I S I O N
I N S I G H T
M E A S U R E
THINK
EFFECT
like Vic Basili’s
Goal-Question-Metric (GQM)
but without
ISO/IEC 15939 baggage

6. Shows differences easily (1)
aka:
Save the “pie” for dessert Credit:
• Stephen Few (Perceptual Edge)
• http://www.perceptualedge.com/ar
ticles/08-21-07.pdf

6. Shows differences easily (2)
Can you compare the market share from one year to the next?
Quickly: Which two companies are growing share the fastest?
One pie chart is bad. Multiple pie charts are worse!!!

6. Shows comparisons easily (3)
How about now?
Can you compare the
market share from one
year to the next?

7. Allows you to
see the forest
AND
the trees

8. Informs along multiple dimensions (1)

8. Informs along multiple dimensions (2)

9. Leaves in the
numbers
where
possible

10. Leaves out glitter
Examples of how NOT to do it.

Top 10 criteria for great visualization
1. Answers the question,
"Compared with what?”
(SO What?)
2. Shows causality, or is at least
informed by it.
(NOW WHAT?)
3. Tells a story with whatever it
takes.
4. Is credible.
5. Has business value or impact in
its social context.
6. Shows
differences
easily.
7. Allows you to see the forest
AND the trees.
8. Informs along multiple
dimensions.
9. Leaves in the numbers where
possible.
10. Leaves out glitter.
Credits:
• Edward Tufte
• Stephen Few
• Gestalt
(School of Psychology)
11. Uses good visual grammar

11. Good visual grammar (1)
When to use a straight line series?
1. Cumulative data
(later data points include
items from earlier data points)
example: Burn or Scope
series
2. Benchmark for column series
3. Connected by “ordinal inertia”
(ordinal usually = temporal)
However, spline is usually
what you want in this
instance.

When to use a dashed line?
1. “Ideal” or reference
2. Forecast
3. Regression or fit
line

When to use a column (vertical bar)
series?
1. Counts/sums independent of
neighbors
example: Throughput or
velocity chart

When to use a bar (horizontal)
series?
(Rarely)
1. The x-axis variable is causal?
2. In a table
3. Stylistic reasons
• Length of labels
• Two-variable display

When to use stacked area?
1. Multiple cumulative series
(the ordinal sum means
something)
example: a cumulative
flow diagram (CFD)

When to use a single area series or
multiple (non-cumulative) area
series?
NEVER?

When to use a single spline series?
1. Filling in for missing granularity
2. Rolling average

When to use a single scatter series?
1. Two variables that are
imperfectly
correlated

Lying with statistics

What’s wrong with this?
84 86 88 90 92 94
Vendor Y
LeanKit
Atlassian/Jira/Greenhopper
VersionOne

Other ways to lie with statistics
 Sampling bias
 Self selection bias
 Leading question bias
 Social desirability bias
 Median vs Mean
 The big “zoom-in”
All together now…
 Correlation does not necessarily mean causation

Every decision is a
forecast!

You are forecasting that
your choice will have better
outcomes than the other
alternatives

Quality of decision depends upon:
1. alternatives considered, and
2. models used to forecast the
outcome of those alternatives.

Group decisions

1. Different Models
2. Different Values
3. Different Risk Tolerance
Why do people disagree?
favor different
alternatives
Fear-based decision
making

Models and Values
 Models calculate probability in terms of proxy variables
 Values translate those probabilities into money
 Different models example:
 Joe forecasts that alternative A will make the most money
 Sally forecasts that alternative B will make the most money
 Different values example:
 Betty favors the alternative with higher quality
 George favors the alternative that will get to market faster

An
agile delivery date forecast
example

Monte Carlo Forecasting
What it looks like
Live demo: http://lumenize.com (use Chrome)

Seek to
change the nature of
the conversation

Argument is about who is right.
Decision making is about what is right.

1. Different Models
2. Different Values
3. Different Risk Tolerance
Why do people favor
different alternatives?
✔

Example model and “values” system
Team performance feedback
 Duplicate great
performance
 Bend the curve early on
declining performance
 Decide what process
improvements to
implement next
Dimensions of performance
1. Productivity
2. Predictability
3. Time-to-market
4. Responsiveness
5. Defect-free-ness
6. Customer satisfaction
7. Employee engagement
8. Build-the-right-thing
9. Code quality

Economic value weighting of team performance
 Proxy variables translated to something unit-less
 Weight (adding to 100%) applied to chosen dimensions
 Examples:
 Medical device manufacturer values QUALITY
 60% for Quality
 10% each for Productivity, Predictability, Time-to-market, and
Responsiveness
 Mobile game team values TIME-TO-MARKET
 40% for Time-to-market
 20% each for Productivity and Responsiveness
 10% each Quality and Predictability

Avoid milk toast values

Influencing with data

The rider and the elephant
 Direct the rider
 Motivate the
elephant
 Shape the path
Jonathan Haidt
The Happiness Hypothesis
(also mentioned in Switch)

… but for those brave enough to journey
into the dangerous world of
agile measurement
there are great riches to be had.
The trick is to slay or avoid the dragons.

The Dragons of Agile Measurement
If you do metrics wrong, you will harm your agile transformation
1. Dragon: Measurement as a lever
Slayer: Measurement as feedback
2. Dragon: Unbalanced metrics
Slayer: 1 each for Do it
fast/right/on-time, and Keep doing it
3. Dragon: Metrics can replace
thinking
Slayer: Metrics compliment
thinking
4. Dragon: Expensive metrics
Slayer: 1st work with the data you
are already passively gathering
5. Dragon: Using a convenient
metric
Slayer: Outcomes 
Decisions  Insight  Metric
(ODIM)
6. Dragon: Bad analysis
Slayer: Simple stats and
simulation
7. Dragon: Single outcome
forecasts
Slayer: Forecasts w/
probability

Now what?
Enter feedback in Sched App
Come to the AgileCraft booth:
• Questions answered
• Demo of how AgileCraft gets you
from “What?” to “So what?” and
then “NOW WHAT?”

When you come to a
fork in the road…
take it!
~Yogi Berra
What?
the metrics and analysis
So what?
how it compares/trends
what it means
NOW WHAT?
every decision is a forecast

"What?", "So what?", "NOW WHAT?" How to influence people and accomplish change

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (11)

Similar a "What?", "So what?", "NOW WHAT?" How to influence people and accomplish change

Similar a "What?", "So what?", "NOW WHAT?" How to influence people and accomplish change (20)

Último

Último (20)

"What?", "So what?", "NOW WHAT?" How to influence people and accomplish change

Notas del editor