This document summarizes an approach to improve fault localization for Simulink models using search-based testing and prediction models. The approach generates a small set of additional test cases to diversify an existing test suite using different test objectives. Predictor models are used to determine when adding more test cases is unlikely to improve fault localization rankings, allowing test generation to stop. An evaluation on 60 faulty industrial Simulink models found the approach significantly improved fault localization accuracy while reducing the number of new test cases generated by over half compared to generating test cases without the predictor models.

1. .lusoftware verification & validation
VVS
Improving Fault Localization for
Simulink Models
using Search-Based Testing and Prediction Models
Bing Liu, Lucia, Shiva Nejati, Lionel Briand
SnT Centre, University of Luxembourg
SANER 2017, Klagenfurt, Austria

2. Simulation
2

3. Simulink
• Is a data flow-driven block diagram language
• Is widely used in the automotive domain
• Is executable and enables simulation and early testing
• Supports automated code generation
3

4. Simulink Fault Localization
4
✔
A
Test
case

5. Background: Statistical
Debugging
5
b4
b2 b3 b5
b7b6
b8
b10 b11 b12
b14b13
b9
b1
Block
Rankings
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
b14
input1
input2
input3
output

6. 6
Ranking Score
b1 0
b2 0
b3 0
b4 0
b5 0
b6 0
b7 0
b8 0
b9 0
b10 0
b11 0
b12 0
b13 0
b14 0
input1
input2
input3
output
Execute test case 1
b4
b2 b3 b5
b7b6
b8
b10 b11 b12
b14b13
b9
b1
Background: Statistical
Debugging

7. 7
Ranking Score
b6 1
b2 0.5
b3 0.5
b5 0.5
b7 0.5
b8 0.5
b9 0.5
b10 0.5
b12 0.5
b13 0.5
b14 0.5
b1 0
b4 0
b11 0
input1
input2
input3
output
Execute test case 2
b4
b2 b3 b5
b7b6
b8
b10 b11 b12
b14b13
b9
b1
Background: Statistical
Debugging

8. Problem: Performance
Limitations
8
Ranking Score
b6 1
b2 0.5
b3 0.5
b5 0.5
b7 0.5
b8 0.5
b9 0.5
b10 0.5
b12 0.5
b13 0.5
b14 0.5
b1 0
b4 0
b11 0
• Faulty blocks may not be
ranked high
• Many blocks may have the
same score
• Engineers may have to
inspect many blocks until
they find the faulty block(s)

9. Goal: Improving Statistical
Debugging
• Statistical debugging can be improved by using larger test
suites
• But, adding test cases is not cost-free in some contexts
because
* test oracles need to be developed manually
* running test cases might be expensive
• We need to generate a small but effective set of test cases
9

10. Our Approach
10
Model Test Suite + Test Oracle Ranking
Generate a small but
effective set of test cases
No
YesRegenerate rankings
Are existing rankings
likely to be improved by
adding more test cases?
Static analysis
+
Predictor models
By diversifying
test cases

11. Test Case Generation
• We use a Single-State search technique to generate new test cases
• Our search strategy is guided by three alternative test objectives that aim
to increase the diversity of the test suite
• Dynamic basic blocks; Baudry et al. [ICSE’06]
• Coverage Density; Campos et al. [ASE’13]
• Coverage dissimilarity; Jiang et al. [ASE’09]
• None of the above test objectives require test oracles
11

12. Stop Test Generation Criteria
• We stop the test generation when adding test cases is unlikely
to improve the rankings
• We rely on
* Simulink Super Blocks: based on static analysis of Simulink
models
* Predictor models built using a supervised learning
technique and historical data
12

13. Decision Trees
• Input features:
• Current Round index
• Set Distance
• Ordering Distance
13

14. Experiment Evaluation
• We applied our approach to 60 single-fault-seeded versions of
our three industrial Simulink models
• We compared our three different test objectives
• We evaluated the effectiveness of predictor models
• The experiment results (except for original industry models)
are available at
https://github.com/Avartar/TCGenForFL/)
14

15. Results (1)
15
• [Accuracies of different test objectives]
ü The generated test cases significantly improved the fault localization accuracy
ü The three test objectives outperform Random test generation
ü No significant difference in the three test objectives

16. Results (2)
16
• [Impact of adding test cases]
ü Adding test cases may not always improve fault localization accuracy

17. Results (3)
17
• [Effectiveness of predictor model]
ü Our approach is able to maintain almost the same fault localization accuracy
while reducing the average number of newly generated test cases by more
than half.

18. Conclusion
• We proposed a new approach to improve fault localization accuracy
for Simulink models by extending an existing test suite with a small
number of high quality test cases.
• We evaluated our approach on 60 faulty versions of industry
models.
• Our approach significantly improves the accuracy of fault
localization for small test suite sizes
• Our approach is able to maintain almost the same fault
localization accuracy while reducing the average number of
newly generated test cases by more than half
18