Доклад Eric Riou du Cosquer на конференции SQA Days-20. 24-26 ноября 2016. Минск www.sqadays.com

1. 05/12/2016 1
Eric RIOU du COSQUER
Minsk, November 24th 2015
Put Risk Based Testing in
place, right now!

2. 05/12/2016 2
You took the risk to attend my presentation
Busines Analyst / Product Owner
Project Manager
Test Manager
Functional or Technical Tester
Software Quality and Testing consultant
Sales person
About you

3. 05/12/2016 3
Eric RIOU du COSQUER, erdc@certilogtest.com
• Business Analysis www.iqbba.org
• Member of the executive committee
• Requirements Engineering www.reqb.org
• Member of the executive committee
• International Software Testing www.istqb.org
• General Secretary from 2011 to 2015, France
Representative afterwards
• French Software Testing Qualification Board www.cftl.fr
• Manager since 2013
• Test organizations assessment www.tmmi.org
• Lead Assessor since 2015
About me

4. 05/12/2016 4
The goal is to explain how to implement a Risk Based Testing
approach based on PRISMA® (Product RIsk MAnagement)
Introduction
Risk Management Basics
RBT approach
What next?
Summary
Agenda

5. 05/12/2016 5
Testing, Risk, and Risk Based Testing
Introduction

6. 05/12/2016 6
Main activities (after ISTQB)
What is testing ?
Planning
Control
Closure
Acceptance
System
Integration
Component 1
Analysis and
Design
Implementation and
Execution
Evaluation &
Reporting
Planification
Closure
Control

7. 05/12/2016 7
Definitions (ISTQB)
Risk
• A factor that could result in future negative
consequences; usually expressed as impact and
likelihood
Product Risk
• A risk directly related to the test object
Project Risk
• A risk related to management and control of the (test)
project, e.g. lack of staffing, strict deadlines, changing
requirements…
What is a risk ?

8. 05/12/2016 8
Definition
Risk Based Testing
• An approach to testing to reduce
the level of product risks and
inform stakeholders of their status
(…). It involves the identification
of product risks and the use of
risk levels to guide the process
What is « RBT » ?
(Risk Based Testing)

9. 05/12/2016 9
A general risk management approach applied to product risks
Risk Management Basics

10. 05/12/2016 10
A process with 4 main activities
Risk
Management
Risk
assessment
Identification
Analysis
Risk control
Mitigation
Monitoring
What does the general risk management
approach consist in ?

11. 05/12/2016 11
The result is a list of risks
• Advice: 30 risks max !
1/4 Risk Identification
Risks Type
Risk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk 4: Reliability
…

12. 05/12/2016 12
Define Likelihood and Impact for each risk, and then a risk
level
• Risk Level = Probability * Impact
2/4 Risk Analysis
Risks Type Likelihood Impact Level
Risk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk 4: Reliability
… … … … …

13. 05/12/2016 13
The risk level calculation may be supported by a table
2/4 Risk Analysis

14. 05/12/2016 14
Implement actions to reduce the risks
• Four mains options
1. Mitigate the risk through preventive measures to reduce likelihood
and/or impact
2. Make contingency plans to reduce impact if the risk becomes an
actuality
3. Transfer the risk to some other party to handle
4. Ignore and accept the risk, which means doing nothing but wait and
see whether the problem occurs or not.
• Mitigation with testing
• Associate test cases to the risks
3/4 Risk Mitigation

15. 05/12/2016 15
Periodically review the risk status , identify new risks and
communicate
4/4 Risk monitoring
Risks Type Proba. Impact Action Status Level
Risk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk
4:
Reliability
… … … … …
New
Risk

16. 05/12/2016 16
A practical approach, step by step
RBT approach
based on PRISMA®
(Product RISk Management)

17. 05/12/2016 17
The decision to implement an RBT
approach must be made
#1
RBT Selection

18. 05/12/2016 18
Possible insights
Exhaustive testing is impossible
The allocated test design and execution time and
budget is always reduced
The specifications and requirements may not
cover the overall set of expected caracteristics
The quality and success of a product depend on
the final users and customers view
How to (be) convice(d) to implement an
RBT approach ?

19. 05/12/2016 19
The right people to be involved must
be identified
#2
Stakeholders
identification

20. 05/12/2016 20
The Test Manager must select different kind of stakeholders
Who should be involved in the RBT
process ?
On the vendor
side
On the
customer
side
External
stakeholder
• End user (client of the customer)
• Other organizations (regulatory
entities,…)
• Customer representatives (called
“Business”)
• Project sponsors
• End users (from the customer company)
• Installation and Operations personnel
• Testers and Quality Assurance staff
• Project managers
• Business and System Analysts
• Developers and architects
• DBA
• GUI designers
• Technical writers
• Testers and Quality Assurance staff

21. 05/12/2016 21
PRISMA provides a checklist for stakeholders identification
Who should be involved in the RBT
process ?
- Project manager - Business experts
- Designers - Testers
- Client / sponsor - End users
- Usability experts - Operations
- Maintenance team - Security
- Safety services - Inspectors
- Support / helpdesk - Manufacturing
- Marketing - Legal
- Professional bodies - Special interest groups
- Technology experts - Marketing
- Customers - System development
- Quality assurance - Regulatory bodies

22. 05/12/2016 22
A first list of risks must be created
#3
Risk identification

23. 05/12/2016 23
Different techniques can be combined
How to involve the selected stakeholders in
the risk identification ?
• Requirements based
• Interviews
• Workshops and Brainstorming sessions
Risks Type
Risk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk 4: Reliability
…
Same
result as
above

24. 05/12/2016 24
The initial set of product risks must
be improved
#4 Risk triage or
extended
identification

25. 05/12/2016 25
Review the list and check against requirements
• Remove the less relevant risk from the list
• What to do with
• A risk but no requirement
• A requirement but no risk
How to keep the most relevant risks in the
list ?
Product Risk Requirement
ID Product Risk Risk Type Requirement
01 Customer cannot start the
transaction at another bank
Functionality Customer shall be able to
perform a transaction at another
bank
02 Customer not issued with receipt
at the end of the transaction
Functionality Customer shall receive a receipt
at the end of the transaction
03 The system is unavailable to the
customer for longer than two
hours
Reliability System shall be available to
customers 24/7
…
…
Example of a set
of product risks
for an after
Pinkster]

26. 05/12/2016 26
The impact of each risk needs to be rated
#5
Impact Rating

27. 05/12/2016 27
PRISMA® suggested factors
1. Critical areas (damage, cost and consequences of failure)
2. Visible areas (external visibility of a failure)
3. Most used areas
4. Business importance
5. Cost of rework
Which factors shall we consider to rate the
impact ?
Impact
Factor Criticity Visibility …
Weight 2 1 …
Risk 1 5 3 …
Risk 2 3 5 …
Risk 3 3 2 …
… … … …

28. 05/12/2016 28
The likelihood of each risk needs to be rated
#6
Likelihood Rating

29. 05/12/2016 29
PRISMA® suggested factors
1. Complexity
2. Size
3. Number of changes
4. New technology and methods
5. Inexperience
6. New development vs. re-use
7. Interfacing
8. …
Which factors shall we consider to rate the
likelihood ?
Impact LIkelihood
Factor Criticity Visibility … Complexity Size …
Weight 2 1 … 1 2 …
Risk 1 5 3 … 3 5 …
Risk 2 3 5 … 4 1 …
Risk 3 3 2 … 2 4 …
… … … … … … …

30. 05/12/2016 30
Once impact and likelihood are scored,
the risks are included in a Matrix
#7
Risk Matrix creation

31. 05/12/2016 31
Impact and Likelihood are scored for each risk
• Each risk may be rated by different profiles
• Impact: business skills
• Likelihood: technical skills
How to visualize the risk distribution ?
Impact Probabilité
Factor Criticity Visibility VALUE Complexity Size VALUE
Weight 2 1 na 1 2 na
Risk 1 5 3 13 3 5 13
Risk 2 3 5 11 4 1 6
… … … … … … …

32. 05/12/2016 32
Each risk will be positioned in a matrix
What is the Product Risk Matrix ?
IIV
II IIII
IIIII
LikelihoodofDefects
(TechnicalRisks)
Impact of Defects
(Business Risks)
3
3
15
15
R1
R2
R3
R4
R5

33. 05/12/2016 33
IIV
Consider the following advice
1. Avoid the central circle
2. Try not to have all the risks in the same areas
3. Add a fifth area for safety-critical applications
How to ensure a right distribution of the
risks ?
IIV
II IIII
IIIII
Likelihood
Impact3
3
15
15
R1
R2
R4
R5
R5 R7
R6

34. 05/12/2016 34
The test approach will be based
on the risk distribution
#8
Test approach and Test
techniques selection

35. 05/12/2016 35
Impact and Likelihood help you focus on the right level(s)
How to allocate the test effort on the
different levels ?
IIV
II IIII
IIIII
Likelihood
Impact3
3
15
15
component and
Integration level
test (focus on
technical risk)
system
and
acceptance
level test
(focus on
business
risk)

36. 05/12/2016 36
This question should be adressed for each test level
How to select the right techniques and
define the associated coverage goals ?
IIV
II IIII
IIIII
Likelihood
Impact3
3
15
15
Example for the component level
Decision
coverage
(90%)
Code
inspection
Instruction
coverage
(90%)
Instruction
coverage
(70%)

37. 05/12/2016 37
This question should be adressed for each test level
How to select the right techniques and
define the associated coverage goals ?
IIV
II IIII
IIIII
Probabilité
Impact3
3
15
15
Use Case
(incl alternative
paths)
Decision
table
Use Case
(main path)
Equivalence
partitioning
Use Case
(incl alternative
paths)
Equivalence
partitioning
Use Case
(main path
Exploratory
testing
Example for the acceptance level

38. 05/12/2016 38
The traceability from risks to test cases
is implemented
# 9
Test Design… and
Execution

39. 05/12/2016 39
Use the traceability
How to reach the final Risk Based Test
Execution step ?
Product Risk Requirement Test Cases
Test
Execution
Results
Defects

40. 05/12/2016 40
The risk likelihood and impact
must be reviewed based on
the test execution results
#10
Risk Based reporting
and Defect correction

41. 05/12/2016 41
Update it !
What to do with the Product Risk Matrix
Product Risk Requirement Test Cases
Test
Execution
Results
Defects
Defects  Likelihood is increased
Passed test cases  Likelihood is decreased
New
risks ?

42. 05/12/2016 42
Increase your knowledge in RBT and implement it right now!
What next ?

43. 05/12/2016 43
And at any time!
RBT is everywhere in the test
process

44. 05/12/2016 44
The Best seller about RBT
• ISBN 9789490986070
Sources

45. 05/12/2016 45
With your own Excel file or the PRISMA® tool
The method can be tooled

46. 05/12/2016 46
ISTQB Advanced Level Test Analyst Syllabus
Additional Sources
• http://www.istqb.org/downloa
ds/send/10-advanced-level-
syllabus-2012/53-advanced-
level-syllabus-2012-test-
analyst.html

47. 05/12/2016 47
TMMi
• http://www.tmmi.org/wp-
content/uploads/2016/09/T
MMi.Framework.pdf
Additional Sources

48. 05/12/2016 48
Eric RIOU du COSQUER
erdc@certilogtest.com
www.certilogtest.com
Thank you !