2. Risk Based Testing
and
Random Testing
• Use of Risk Analysis and Metrics
for Software Testing
• Focus Testing to Save Time and
Money while maintaining quality
• How to develop metrics to manage
and organise large test projects
4. Risk Analysis and Testing
Test Plan
Test Item Tree
Risk
Strategy
Risk
Identification
Testing,
Inspection etc.
Risk
Assessment
Matrix: Cost
and Probability
Risk
Mitigation
Test Metrics
Risk
Reporting
Risk
Prediction
5. Risk Based Testing - Theory
• The Formula
Re(f) P(f)*C(f)
– Re(f) Risk Exposure of function f
– P(f) - Probability of a fault in function f
– C(f) - Cost related to a fault in function f
6. Risk Based Testing - Approach
• Plan: Identify Elements to be Tested
– Logical or physical Functions, Modules etc.
• Identify Risk Indicators
– What is important to predict the probability of faults?
• Identify Cost of faults
• Identify Critical Elements
– I.e. functions, tasks, activities etc. based on Risk Analysis
(Indicators and Cost)
• Execute: Improve the Test Process and Organization:
Schedule and Track
7. Simple Test Metrics
• Test Planning
– Number of test cases per function
– Number of hours testing per function
• Progress Tracking
–
–
–
–
Number of tests planned, executed and completed
Number of faults per function
Number of hours used for test and fix
Estimated to Complete
• Probability of faults - Indicators
–
–
–
–
New functionality
Size
Complexity
Quality of previous phases and documents
• Cost of Faults
8. Risk Based Testing - Metrics
• Identify Areas with “High Risk Exposure”
– Probability and Cost
• All functions/modules should be tested to a
“minimum level”
• “Extra Testing” in areas with high risk exposure
• Establish Test Plan and Schedule
– Monitor Quality
• Number of Faults per function and time
– Monitor Progress
• Number of hours in test and fix -> ETC
9. Risk Based Testing - Example
Ranking the functions based on Risk Exposure
The Probability of a Fault
The Cost of a Fault
C(c) C(s)
Re(f) P(f)*
2
Example:
Cost
Func.
Probability
New Design
Func. Quality Size
5
5
1
Risk
Com- Weighted Exp.
plexity Averag func.
3
f
C( s )
C(c)
Avr.
Interest
Calc.
3
3
3
2
3
3
3
37
111
Close
Account
1
3
2
2
2
2
3
31
62
Cust.
Profitab.
2
1
1,5
3
3
2
3
41
61,5
Other Probability Factors might include: Function Points, Frequency of Use etc.
10. High
Probability
Risk Based Testing
- Reporting
1
2
3
4
TECHNICAL
INTERFACE
RISK
10
1
Low
Low
440
Low Medium
Probability
High
510
439
11
2
2
370
369
5
302
Low
Low
Medium
Medium
BUSINESS RISK
High
High
Consequence
Consequence
High
11. Risk Based Testing - Practice
Prior to test execution:
identify critical transactions
1
Test Execution
identifies
“bad” transactions
“Top-20”
2
Extra Testing:
- Additional testing by product specialist
- Automated regression testing
3
12. Planning and Progress Tracking
Number of Test
Cases
On-line Test Cases Completed
Planned
Executed
QAed
Date
Started
Planned
Actual
13. Progress Indicators - “To be vs. Actual”
“To be
Retested”
vs.
“Actually
Retested”
To Be Fixed
Actually fixed
To Be Restested, Actually Retested and Rejected
Number of Faults
• “To be fixed”
vs.
“Actually fixed”
Number of Faults
To Be Fixed and Actually Fixed
To be retested
Act. retested
Rejected
14. Progress Indicators - Hours Used
Number of hours
for finding one
fault and for
fixing one
Hours per Fault for Test and Fix
Hours per Fault
Online
Batch
Fix
Hours per Fault for Test and Fix
Hours per Fault
Number of hours
for finding one
fault and for
fixing one
Test
Fix
Test
Date
15. “Estimated to Complete”
• ETC for system test based on:
Hours ETC
– Number of hours testing per fault found
– Number of hours fixing per fault
– Number of
faults found
Calculated ETC and Actual Hours
per function
Actual to Complete
– Number of
at Time t
fixes being
rejected
– Number of
remaining
tests
(functions
to be tested)
Date
Estimated to Complete at Time t
16. Benefits of Risk Based Testing
• Improved Quality?
– all critical functions tested
• Reduced Time and Money in Testing
– effort not wasted on non critical or low risk
functions
• Improved customer confidence
– due to customer involvement and good reporting
and progress tracking
17. Test Process Work Flow
Risk Identification
Risk Assessment
Basic Test
Data
PD
LD
Case Quality
Standards
Test Exec.
Procedure
Test Case
Case Build
Procedure
Test
Completed
QC / QA
Good
Test Exec
Risk Mitigation
Good/
Bad
Bad
PTDs
Raised
Risk Reporting
Retest
Risk Prediction
Problem Mngmnt.
Procedure
Fix
Fix Procedure
CR
Change Mngmnt.
Procedure
Regression
Test
ProAte
18. Summary
• Risk Based Test Approach
– Focused Testing
• Reduced Resources
• Improved Quality
– Metrics are fundamental
• Process and Organization must support the
new strategy
– Metrics must support the organization and
process
19. Random testing
– Start off with a practical look, and some useful
ideas to get you started on the project: random
testing for file systems
– Then take a deeper look at the notion of feedback
and why it is useful: method for testing OO
systems from ICSE a couple of years
• Then back out to take a look at the general idea of
random testing, if time permits
20. A Little Background
– Generate program inputs at random
– Drawn from some (possibly changing) probability
distribution
“Throw darts at the state space, without drawing a
bullseye”
– May generate the same test (or equivalent tests)
many times
– Will perform operations no sane human would
ever perform
21. Random Testing
• Millions of operations and scenarios, automatically
generated
• Run on fast & inexpensive workstations
• Results checked automatically by a reference oracle
• Hardware simulation for fault injection and reset
simulation
(x 100000)
A day (& night) of testing
(x 100000)
(x 100000)
(x 100000)
22. The Goals
• Randomize early testing (since it is not possible
to be exhaustive)
– We don’t know where the bugs are
Nominal
Scenario Tests
Randomized
Testing
23. The Goals
• Make use of desktop hardware for early testing – vs.
expensive (sloooow) flight hardware testbeds
– Many faults can be exposed without full bit-level
hardware simulation
24. The Goals
• Automate early testing
– Run tests all the time, in the background, while continuing
development efforts
• Automate test evaluation
– Using reference systems for fault detection and diagnosis
– Automated test minimization techniques to speed
debugging and increase regression test effectiveness
• Automate fault injection
– Simulate hardware failures in a controlled test
environment
25. Random testing
• Simulated flash hardware layer allows random
fault injection
• Most development/early testing can be done
on workstations
• Lots of available compute power – can cover
many system behaviors
• Will stress software in ways nominal testing will
not
26. Differential Testing
• How can we tell if a test succeeds?
– POSIX standard for file system operations
• IEEE produced, ANSI/ISO recognized standard for file
systems
• Defines operations and what they should do/return,
including nominal and fault behavior
File system
/
POSIX operation
Result
mkdir (“/eng”, …)
mkdir (“/data”, …)
creat (“/data/image01”, …)
creat (“/eng/fsw/code”, …)
mkdir (“/data/telemetry”, …)
unlink (“/data/image01”)
SUCCESS
SUCCESS
SUCCESS
ENOENT
SUCCESS
SUCCESS
/eng
/data
image01
/telemetry
27. Differential Testing
• How can we tell if a test succeeds?
– The POSIX standard specifies (mostly) what correct
behavior is
– We have heavily tested implementations of the
POSIX standard in every flavor of UNIX, readily
available to us
– We can use UNIX file systems (ext3fs, tmpfs, etc.) as
reference systems to verify the correct behavior of
flash
– First differential approach (published)
was McKeeman’s testing for compilers
28. Random Differential Testing
Choose (POSIX) operation F
Perform F on NVFS
Perform F on Reference
(if applicable)
Compare return values
Compare error codes
Compare file systems
Check invariants
(inject a fault?)
29. Don’t Use Random Testing for Everything!
• Why not test handing read a null pointer?
– Because (assuming the code is correct) it guarantees
some portion of test operations will not induce failure
– But if the code is incorrect, it’s easier and more
efficient to write a single test
– The file system state doesn’t have any impact (we
hope!) on whether there is a null check for the buffer
passed to read
• But we have to remember to actually do these
non-random fixed tests, or we may miss critical,
easy-to-find bugs!
30. Principles Used
•
•
•
•
Random testing (with feedback)
Test automation
Hardware simulation & fault injection
Use of a well-tested reference implementation
as oracle (differential testing)
• Automatic test minimization (delta-debugging)
• Design for testability
– Assertions
– Downward scalability (small model property)
– Preference for predictability
31. Synopsis
• Random testing is sometimes a powerful
method and could likely be applied more
broadly in other missions
– Already applied to four file system-related
development efforts
– Part or all of this approach is applicable to
other critical components (esp. with better
models to use as references)
