Risk Mitigation Trees - Review test handovers with stakeholders (2004)

Risk Mitigation Trees:
Review test handovers with stakeholders

12th European Conference on Software Testing, Analysis & Review
29 November - 03 December 2004 – Köln, Germany
Management Track session T6

Neil Thompson
Thompson information Systems Consulting Limited
www.TiSCL.com

© Thompson
information
Systems
Consulting Limited
1 www.qualtechconferences.com

1. Introduction
• Primary objective of presentation – to share two simple
techniques (used successfully on recent projects):
– Testing Review Boards (TRBs) and
– Risk Mitigation Trees (RMTs)
• Used for making collaborative decisions at development-
test handovers, then at go-live
• Go well together, though can be used separately
• TRBs are meetings of testing’s stakeholders at handovers
• RMTs are a diagramming technique to split residual risk
• There is also a secondary objective of this presentation…
© Thompson
information
Systems
Consulting Limited

Introduction (cont’d)
• Secondary objective is to generalise TRBs & RMTs into a
broad framework of decision-making:
– so we can better understand the nature of the decisions
– so we can improve the process
– in future we may be able to benefit from better reliability theories
and more user-friendly statistics;
– there are relevant considerations in decision theory & game theory
– process overall may be integrated by cybernetics & systems
science (self-organising systems)
• Most of the esoteric stuff is in the accompanying paper
• This presentation is based on my experiences in large
projects & programmes, but principles should be valid in
other handover contexts, eg software product development
© Thompson
information
Systems
Consulting Limited

Agenda
• 2. Traditional methods of controlling handovers
• 3. Judging the best date for handover
• 4. Co-operation: Testing Review Boards
• 5. Pragmatism: Risk Mitigation Trees
• 6. Decision theory
• 7. Game theory
• 8. Systems theory and cybernetics
• 9. Conclusions and way forward

© Thompson
information
Systems
Consulting Limited

2. Traditional methods of
controlling handovers
• “Over the wall”:
– bugs may be passed over undetected, amplifying problems later
– even if the software works well, could be insufficient collaboration
– not only waterfall, could also affect iterative methods (but not truly
Agile methods, which emphasise interpersonal communications)
• Handover certificates:
– can languish in in-trays; very often signed with caveats
• Correspondence of exit & entry criteria:
– only the simplest waterfall method has no overlap
– entry includes own responsibilities in addition to donors
– with overlap, entry criteria less demanding than exit

© Thompson
information
Systems
Consulting Limited

3. Judging the best date
for handover
• Risk-benefit balance:
– “Good Enough Quality” is popular principle…
– but different stakeholders in testing are likely to have different
views of the benefits, and especially the risks
– We don’t (yet?) have an objective view of the risks, in terms of the
likely bug rate after go-live, despite the long history of…
• Reliability theories:
– dominated the early literature on testing, but have faded
– mathematics drown realism
– Bayesian techniques seem most promising, but
– whole subject still has “holy grail” status for practical use

© Thompson
information
Systems
Consulting Limited

Judging the best date for
handover (cont’d)
# bugs
• Metrics – use quantitative for a while:
– S-curves have empirical evidence and
quasi-theoretical basis Cumulative bugs
– can be applied to both progress
through tests and bug-fixing…
– but subject to caveats: uniform test Resolved
strategy, “tyranny of numbers” Awaiting
– approach target as a “glide path”… fix Deferred

# Closed
tests
date
Target tests run

Fail
Actual tests run Pass
date
© Thompson
information
Systems
Consulting Limited

Judging the best date for
handover (cont’d)
• Metrics:
– that “glide path” takes a long time to reach target, and never totally
smooth (also, target is short of perfection to begin with)
– to time the landing (handover), what will be important will be the specific
nature of the shortfalls, and their impacts
– so are qualitative metrics better than quantitative?
– I say use both: coarse-tune with quantitative, then fine-tune with
qualitative
• So, we need mechanisms to interpret metrics and agree
what to do:
– (co-operation) Testing Review Boards to review progress and agree
confidence
– (pragmatism) Risk Mitigation Trees to analyse residual risk and agree
viable compromises and mitigations

© Thompson
information
Systems
Consulting Limited

4. Co-operation: Testing
Review Boards 7
Live

• Advantages of TRBs over sign-off forms: OAT
5 6
– forum for discussing & resolving differences
of view among stakeholders UAT
– force a decision on given date 3 4

– robust shared decisions System Testing
2

Integration Testing
1

Development & Unit Testing
TRB attenders TRB # 1 2 3 4 5 6 7
BUSINESS SPONSOR ------- ------- ------- ------- ------- attend attend
IT OPERATIONS MANAGER ------- ------- ------- ------- attend attend attend
PROJECT MANAGER attend attend chair attend chair attend chair
TESTING / QUALITY MANAGER chair chair facilitate chair facilitate chair facilitate
BUSINESS ARCHITECTS ------- attend attend attend ------- attend attend
TECHNICAL ARCHITECTS attend attend ------- attend attend ------- attend
DEVELOPMENT MANAGER attend attend ------- attend ------- ------- attend
© Thompson
information
Systems
Consulting Limited

Testing Review Boards
(cont’d)
• Imagine the variation in caveats from these different
viewpoints!
“quality” (or
• Some may not sign low risk)
in time, or
at all. TESTERS
BUSINESS
ARCHITECT

IT OPERATIONS scope (of
TEST
MANAGER
MANAGER
USER testing done)
BUSINESS
SPONSOR
IT PROJECT DEVELOPMENT
low DIRECTOR MANAGER MANAGER speed of testing,
cost timeliness of
implementation
© Thompson
information
Systems
Consulting Limited

(cont’d)
• Not anarchy: it should be pre-agreed who is the prime decision-maker
at each meeting: the “casting vote”
• That should be the recipient (varies between different TRBs)
• So incorporates traditional acceptance hierarchy… (examples)
5 7
agrees to receive accepts
IT OPERATIONS MANAGER BUSINESS SPONSOR

recommends recommends
PROJECT MANAGER PROJECT MANAGER

give information & opinions give information & opinions
TECHNICAL OPERATORS TEST ETC. BUSINESS USERS TEST ETC.
ARCHITECTS MANAGER ARCHITECTS MANAGER
entry to
go-live
Operational (IT) Acceptance Testing
© Thompson
information
Systems
Consulting Limited

(cont’d)
• Some obstacles & pitfalls: • Ways to cope:
– too many meetings already! – are any existing meetings less
– may be multiple sites, perhaps important?
multiple companies – audio- & video-conferencing
– difficult to get aligned diary – electronic scheduling
slots – deputisation allowed
– input information volatile, – clerical & tool assistance
right up to the meeting – take simultaneous snapshots,
– timing of meeting is finely- even if analysis lags
balanced; late disappointments – go-ahead with extra mitigation
threaten postponement is better than postponement
• I’ve seen TRBs work. But may become so popular that
managers request also mini-TRBs, and even micro-TRBs!
© Thompson
information
Systems
Consulting Limited

5. Pragmatism: Risk
Mitigation Trees
etc (fix & retest)

lo-pri significant

passed failed unresolved bugs
not hi-pri hi-pri

run not run

coverage any coverage
OK now shortfall

Tests Bugs specific
planned “zero-bug nirvana” problems
test coverage
planned coverage bug acceptance criteria

handover target handover target

date
date
© Thompson
information
Systems
Consulting Limited

Risk Mitigation Trees
(cont’d)
specific confidence in benefits in decisions
impacts mitigations fix coming having that function,
problems in time even if faulty
needed

business limit delay
value threat exposure exposure
customers internal users Go /
internal costs manual fix on no go
transactions table entries workarounds fail
frequency
numbers affected of use potential
mitigations
© Thompson
information
Systems
Consulting Limited

Risk Mitigation Trees
(cont’d)
• Advantages of RMTs:
– they can make a previously daunting and amorphous set of test shortfalls, and
bugs, seem manageable
– not only seem manageable, but are manageable (“divide & conquer”)
– the specific impacts of shortfalls, and ways to mitigate them, can be (perhaps
surprisingly) easier to make decisions about than arbitrary numbers such as
“<3 highs, <20 medium bugs tolerable”
– the numbers of un-run tests and un-fixed bugs are viewed in the light
of high (usually) numbers of successful tests and fixed bugs
• Risk & impact mitigation:
– options include pre-empt/react, avoid/mitigate, transfer
– examples:
• some functions need not be used for several months
• some failures would be in such small numbers and so easily seen that fix-on-fail is
less risky than a software upgrade
© Thompson
information
Systems
Consulting Limited

6. Decision theory
• That’s the end of the straightforward practical advice: now
for some interesting background (ideas for future thought
and process improvement)
• Decision theory: “a body of knowledge and related
analytical techniques of different degrees of formality
designed to help a decision-maker choose among a set of
alternatives in light of their possible consequences”
• Decisions typically made at TRBs: do we go live, or delay
by a week, or delay by a month, or call in the lawyers?
• So can decision theory help us here?

© Thompson
information
Systems
Consulting Limited

Risk or uncertainty?
Different conditions under which decisions are made…

Certainty Risk Uncertainty

Alternatives A B C A B C A B C

Consequences a b c a1 b1 c1 a1 b1 c1

a2 b2 c2 a2 b2 c2

b3 c3 b3 c3

Probability b4 b4

of each p(a1), p(a2), p(b1), p(b2), p(b3), p(b4), ?, ?, ?, ?, ?, ?,
consequence p(c1), p(c2), p(c3) ?, ?, ?

© Thompson
known unknown
information
Systems
Consulting Limited

Uncertainty-Based
Testing!
• So all this we’ve been saying Simplifying
about Risk-Based Testing: it
TRB framework…
should properly be called decision
Uncertainty-Based Testing Alternatives:
A - go live now
• We do not know the probability B - delay 1 week
of each possible consequence A B C C - delay 1 month
• We don’t even have a fixed set
Consequences (additive):
of alternatives: could be various a1 b1 c1 1 - cost of future live bugs
2 - benefit of going live
combinations of delay, overtime, a2 b2 c2 3 - cost of longer project
4 - penalty payment costs
descoping etc b3 c3

• But suppose we could simplify c4

(example as earlier)… ?, ?, ?, ?, ?, ?, p(c4) 100%,
?, ?, ? p(b3) & p(c3) 100%,
p(a2), p(b2) & p(c2) uncertain,
p(a1), p(b1) & p(c1) unknown
© Thompson
information
Systems
Consulting Limited

Uncertainty-Based
Testing (cont’d)
• “Simplifying” framework has left two core uncertainties:
– what will be the benefits of going live at a particular date?
– what will be the costs of the bugs which will affect live operations?
• Decision theory offers two main approaches to uncertainty:
– reduce the uncertainty to mere risk by deeming the unknown
probabilities “known” by using subjective estimates from experts
and/or previous experience
– look at the choice criteria using Game Theory (about which, more
later)
• Types of uncertainty:
– philosophical analysis (see paper)
– related to testing…

© Thompson
information
Systems
Consulting Limited

Types of uncertainty in
testing
• “Knowledge incompleteness due to inherent deficiencies
with acquired knowledge”:
– we know what we’ve tested, but:
• right things? Any low-payback waste?
– we know bugs found, but:
• how many faults remain? Which will trigger failures? Impact?
– we know bugs fixed, but:
• how good was our regression testing? Impact of knock-ons?
• “Ambiguity, approximations, randomness & sampling”:
– what is signal, what is noise; what trends are statistically
significant, eg if high-impact bugs out of testing over last 3 weeks
numbered 1, 0 then 2, what does that tell us?

© Thompson
information
Systems
Consulting Limited

Types of uncertainty in
testing (cont’d)
Ideally we’d understand defect-fault-failure chains for all
of our existing failures, and prevent/mitigate future mistakes…

 Mistake:
UNCERT’Y a human action that (false alarm):
produces an incorrect
or Change Request,
result (eg in spec-
writing, program-
coding)
 Anomaly:
or testware mistake

an unexpected result
RISK OF MIS-
 this fits its usage
Note:
during testing
INTERPRETING
RISK RISK OF MISSING
in inspections

Defect:  Fault:  Failure:
incorrect results RISK an incorrect step, RISK an incorrect result
in specifications process or data
definition in a
computer program  Error:
amount by which
RISK (ie executable result is incorrect
software)
Direct programming mistake
© Thompson
information
Systems
Consulting Limited

Can more trees help?
• Event trees & fault trees can be used to investigate defect-
fault-failure relationships, and in theory the risk of future
live bugs is the sum of a set of micro-risks (but this kind of
analysis is easier after the event than before!)
• Attempts have been made to use decision trees (not the
same as RMTs) to assess risk (ref. Shari Lawrence Pfleeger). But:
– risks have a probability distribution, not a single definite probability
– quantitative risk assessment can have misleading “precision” and
can greatly differ from people’s perceptions
– is lo-probab hi-impact really the same as hi-probab lo-impact?
– if not, how to tweak the factors?

© Thompson
information
Systems
Consulting Limited

What about
mathematics?
• Decision theory advises us to use utility functions
• Utility may be defined as “the real or fancied ability of a
good or service to satisfy a human want”
• This sounds like acceptance testing: working backwards,
each handover decision is part of a sequence of decisions
• Each decision affects next: eg handover too soon from
development, & resultant disruption may exceed time saved
• Simple decision theory inadequate (deterministic, Markov)
• Bayesian networks look promising, but are difficult for non-
mathematicians and need much computing power

© Thompson
information
Systems
Consulting Limited

7. Game Theory
• If we can’t (yet?) use reliability theory / utility functions /
Bayesian networks to assess risk-benefit balance at
handovers, let’s fall back on the decision theory alternative…
• Game theory:
– big in social sciences, and well-established in business strategy
– has already been invoked for software development as a whole: a
resource-limited, co-operative, finite, goal-seeking game of invention
& communication Alistair Cockburn
– is a research effort to use in automatic test case generation Microsoft
• The relevance to Testing Review Boards is that most or all
of the participants are playing a game called Career: the
project, and the testing phases, are just subgames
© Thompson
information
Systems
Consulting Limited

Snakes & Ladders?

PROJECT MANAGER: BUSINESS SPONSOR:
↑ Will I get a reputation as a ↑ Will this project enhance
skilled risk-balancer? my career? Will I be seen
↓ Will I get a reputation as a as a brave decision-taker?
deadline-misser / budget- ↓ Will I lose respect? Will this
buster? project damage the bottom
line?

DEVELOPER: USER:
↑ Will this project help me get ↑ Will system make our work
a promotion / interesting new more fulfilling & value-
project? adding?
↓ Will someone find faults in ↓ Will live failures give us
my modules after go-live? extra frustrating work?
Will I get recalled?

© Thompson
information
Systems
Consulting Limited

n-person Game Theory
• Game Theory: a maths-based
framework for competing (i) moves:
α β
individuals or groups to each extensive αA
βA 10, 4
try to maximise their utility form βB 1, 5
βA
through a series of moves COMPETITOR ALPHA αB
βB
9, 9
0, 3
(sequential or simultaneous) αA 10, 4
COMPETITOR BETA βA
• Competitors may or may not αB 9, 9
αA 1, 5
know the moves and utilities of βB αB 0, 3
other competitors
(ii) strategies: COMPETITOR BETA
• In a TRB, there are more than normal units of utility
two “competitors” who may be form to each Alternative Alternative
competitor βA βB
arguing for different outcomes
and have an imperfect view of Alternative 4 5
COMPETITOR αA 10 1
utilities
ALPHA Alternative 9 3
• Suggests we need the most αB 9 0
complex and abstract part of
game theory… (iii) payoffs: >2 COMPETITORS
characteristic v(x)=…
Material combined from Game theory – a critical introduction,
Hargreaves Heap & Varoufakis (Routledge 1995); and N-person function multi-dimensional space,
game theory, Rapoport (Dover 1970-2001)
form may involve coalitions
© Thompson
information
Systems
Consulting Limited

But for now:
a game of three halves
In effect, can consider TRB as a two-competitor game having
also a fore-game and an after-game…
“quality” (or low
risk)
Mitigating actions Delay how long?
TESTERS
BUSINESS (as from RMT) Add any scope?
ARCHITECT
scope (of
IT
TEST USER testing done)
OPERATIONS
MANAGER
MANAGER
BUSINESS Go No-go
IT PROJECT DEVELOPMENT
SPONSOR COMPETITOR
low cost speed of testing,
MANAGER MANAGER COALITION
DIRECTOR
timeliness of
implementation BETA A B
(“doves”)
Utility
fore-game U = u(a1)+u(a2) or u(b1)+(b2), Alternative Alternative
ALPHA view or BETA view A: go B: no-go
• consider preferences Alternative
COMPETITOR hi-U ok-U
(A, B etc) based on A: go after-game
COALITION hi-U hi-U
views
ALPHA Alternative illogical & ok-U
• form hawk / dove
(“hawks”) B: no-go impossible ok-U
coalitions (real or “usual case” but can’t
virtual) rubber-stamp shared-fear continue to differ,
© Thompson case case need to negotiate
information
Systems
Consulting Limited

8. Systems theory &
cybernetics
Philosophy
“things” “thoughts”
Metaphysics Logic Epistemology

Natural sciences Systems sciences Social sciences
General
Mathematics systems Languages
thinking
Systems theory Cybernetics
(structure) (function)

control communication
TRBs &
RMTs Decision Complexity Theory
Theory

value / utility / game Decision-maker
self-organising systems etc
functions functions theory
Probabilities Decision Objectives Constraints

Consequences Alternatives

certainty / risk / uncertainty
© Thompson
information
Systems
Consulting Limited

9. Conclusions &
way forward
• Until reliability models get cleverer / simpler, and more
metrics collected, we can’t predict bug rates after go-live
• So main alternatives for managing handovers seem still:
– rigid criteria
– these Testing Review Board & Risk Management Tree techniques
– something even more agile
• To progress to something more scientific, may need to
confront the following complexity…

© Thompson
information
Systems
Consulting Limited

Potential way forward
(cont’d): benefits
most likely
relative probability of
50% point

actual go-live
being achieved
this benefit

earliest latest
of go-live this date
total gross benefit

1 Jan 1 Apr 1 May 31 Dec date

estimated
estimated

min gross benefit per day
most likely of go-live this date

max (area under curve)

1 Jan

1 Apr

1 May

© Thompson Extended from material by DeMarco & Lister, Waltzing with bears (Dorset House 2003)
information
Systems
Consulting Limited

Potential way forward
(cont’d): risk-costs
most likely
50% point

actual go-live
being incurred
this risk-cost

earliest latest
of go-live this date

1 Jan 1 Apr 1 May 31 Dec date
total risk-cost

estimated
• even more
estimated

min risk-cost per day
most likely of go-live this date difficult to
max (area under curve) estimate
1 Jan
than
benefits?
• much
more
“spiky”!
1 Apr • (note:
excludes
fixed costs)

1 May

© Thompson Extended from material by DeMarco & Lister, Waltzing with bears (Dorset House 2003)
information
Systems
Consulting Limited

Summary
Systems sciences

Decision
Theory
value / utility / game Risks & benefits
functions functions theory forecast

Risks & benefits now:
1 - bugs visible now 1 - cost of future live bugs
2 - stated target benefits 2 - benefit of going live
3 - cost of longer project
4 - penalty payment costs


Usable reliability theory
S-curves
Risk Mitigation Trees decision
Bayesian networks
support

© Thompson
information
Systems
Consulting Limited

References &
acknowledgements
• Main references:
– Beizer, Boris: Software system testing & quality assurance
– Pirsig, Robert M: Lila – an enquiry into morals
– Bach, James: Good enough quality etc
– DeMarco & Lister: Waltzing with bears
– Principia Cybernetica Web
– Rapoport, Anatol: N-person game theory
– Kaner, Cem: Software testing as a social science
– (for others, see associated paper in conference proceedings)
• Acknowledgements:
– to Pat, Rob & Rupert for their very considerable input to the Risk
Mitigation Trees method
– to all the team on that programme and at that client
© Thompson
information
Systems
Consulting Limited

Contact details
Neil Thompson

NeilT@TiSCL.com
Questions?
www.TiSCL.com

23 Oast House Crescent
Farnham, Surrey, England
GU9 0NP, United Kingdom

phone +44 (0)7000 NeilTh (634584) or
+44 (0)7710 305907

© Thompson
information
Systems
Consulting Limited

Risk Mitigation Trees - Review test handovers with stakeholders (2004)

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to Risk Mitigation Trees - Review test handovers with stakeholders (2004)

Similar to Risk Mitigation Trees - Review test handovers with stakeholders (2004) (20)

More from Neil Thompson

More from Neil Thompson (16)

Recently uploaded

Recently uploaded (20)

Risk Mitigation Trees - Review test handovers with stakeholders (2004)