OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
Mutate and Test your Tests
1. Mutate and Test Your Tests
Benoit Baudry
KTH, Sweden
1
baudry@kth.se
2. Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
Benoit Baudry, KTH, EclipseCon'17
2
3. Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
Benoit Baudry, KTH, EclipseCon'17
3
5. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
5
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
Benoit Baudry, KTH, EclipseCon'17
6. Coverage
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
6
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
Benoit Baudry, KTH, EclipseCon'17
7. Coverage
7
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Benoit Baudry, KTH, EclipseCon'17
8. 8
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
Benoit Baudry, KTH, EclipseCon'17
9. 9
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
Let’s mutate our code to see.
Benoit Baudry, KTH, EclipseCon'17
10. Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
10
Benoit Baudry, KTH, EclipseCon'17
11. Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
•Mutation operators
•Mutant = Program with one mutation
11
Benoit Baudry, KTH, EclipseCon'17
12. Mutation analysis
Inputs : P, TS, Ops
Output : score, coverage
M <- generateMutants (P, OPs)
forAll (m in M)
run (TS,m)
if (one-test-fail)
then killed <- m
else alive <- m
score = killed / size(M)
12
Benoit Baudry, KTH, EclipseCon'17
13. PIT
•Open source
•Actively developed and maintained
•State of the art
•For Java
•Integrates with major build systems
13
Benoit Baudry, KTH, EclipseCon'17
15. long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
15
Benoit Baudry, KTH, EclipseCon'17
16. long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
16
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
Benoit Baudry, KTH, EclipseCon'17
17. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
17
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
Benoit Baudry, KTH, EclipseCon'17
18. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
18
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
Benoit Baudry, KTH, EclipseCon'17
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
19. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
19
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Benoit Baudry, KTH, EclipseCon'17
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
20. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
20
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Benoit Baudry, KTH, EclipseCon'17
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
21. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Mutation reveals bugs in the test suite
21
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Benoit Baudry, KTH, EclipseCon'17
Bugs in the test suite:
- Weak oracle
- Missing input
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
22. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
22
@Test
factorialWith5Test() {
long obs = fact(5);
assertEqual(120, obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
@Test
factorialWith1Test() {
assertEqual(1, fact(1));}
Benoit Baudry, KTH, EclipseCon'17
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
26. Descartes
•Mutation operators: extreme mutation
•Active development
•Open source
•Compared to PIT
• Less mutants
• Different type of feedback
• Same framework
26
Benoit Baudry, KTH, EclipseCon'17
27. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
27
Benoit Baudry, KTH, EclipseCon'17
28. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
28
long fact(int n){
return 0;}
long fact(int n){
return 1;}
Benoit Baudry, KTH, EclipseCon'17
PIT : 7 mutants
Descartes : 2 mutants
46. Descartes – future
•Incremental Descartes in the CI
•Github hook
•Support multi module Maven projects
•Enhanced diagnosis
•Integrate into a test augmentation pipeline
Benoit Baudry, KTH, EclipseCon'17
46
47. Conclusion
•Mutation analysis
• Automatic generation of mutants
• Evaluate the test suite
•Bugs in test suites
• Oracle
• Input space coverage
• Testability
• Indirectly tested code
47
Benoit Baudry, KTH, EclipseCon'17