Benoit Baudry (KTH and H2020 STAMP project scientific leader) introduces and illustrates the benefits of mutation testing to assess the quality of test suites.

1. Mutation testing in Java and GitHub
Oscar Vera Perez, Caroline Landry, Martin Monperrus,
Benoit Baudry
Breizh JUG, Rennes. June 2018.
1

2. Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
2

3. Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
3

4. 4
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}

5. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
5
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}

6. Coverage
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
6
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}

7. Coverage
7
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}

8. 8
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?

9. 9
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
Let’s mutate our code to see.

10. Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
10

11. Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
•Mutation operators = types of bugs
•Mutant = Program with one seeded bug
11

12. Mutation analysis
Input : P, TS, Ops
Output : score, coverage
M <- generateMutants (P, OPs)
forAll (m in M)
run (TS,m)
if (one-test-fail)
then killed <- m
else alive <- m
score = killed / size(M)
12

13. PITest mutation operators
•Conditions
•Constants
•Return
•Delete method calls
•Constructor call
13

14. long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
14

15. long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
15
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

16. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
16
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

17. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
17
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

18. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
18
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

19. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Mutation reveals bugs in the test suite
19
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Bugs in the test suite:
- Weak oracle
- Missing input
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

20. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
20
@Test
factorialWith5Test() {
long obs = fact(5);
assertEqual(120, obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
@Test
factorialWith1Test() {
assertEqual(1, fact(1));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

21. Project #Mutants Time (h) Score (%)
Amazon Web Services SDK 2141690 04:25:35 76.28
XWiki Rendering Engine 112609 01:59:55 50.89
Apache Commons Math 104786 03:22:18 83.81
JFreeChart 89592 00:41:28 58.04
Apache PdfBox 79763 06:20:25 58.89
Java Git 78316 16:02:00 73.86
SCIFIO 62768 03:12:11 45.92
Joda-Time 31233 00:16:32 81.65
Apache Commons Lang 30361 00:21:02 86.17
Apache Commons Collections 20394 00:05:41 85.94
Urban Airship Client Library 17345 00:11:31 82.26
SAT4J 17067 11:11:04 68.58
ImageJ Common 15592 00:29:09 54.77
jsoup 14054 00:12:49 78.34
Jaxen XPath Engine 12210 00:24:40 67.13
Apache Commons Codec 9233 00:07:57 87.82
Apache Commons IO 8809 00:12:48 84.73
Google Gson 7353 00:05:34 81.76
AuthZForce PDP Core 7296 01:23:50 88.18
21

22. Descartes
•Mutation operators: extreme mutation
•Active, open source development
•Pitest plugin
•Compared to PIT
• Less mutants
• Different type of feedback
• Same framework
22

23. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
23

24. long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
24
long fact(int n){
return 0;}
long fact(int n){
return 1;}
PIT : 7 mutants
Descartes : 2 mutants

25. Scalability of extreme mutation
25
public static boolean isValidXmlChar(int ch){
return (ch == 0x9)
|| (ch == 0xA)
|| (ch == 0xD)
|| (ch >= 0x20 && ch <= 0xD7FF)
|| (ch >= 0xE000 && ch <= 0xFFFD
|| (ch >= 0x10000 && ch <= 0x10FFFF);
}
PIT : 45 mutants
Descartes : 2 mutants

26. 26
Number of mutants
1
10
100
1000
10000
100000
1000000
10000000
Amazon Web Services SDK JFreeChart SCIFIO Apache Commons CollecAonsImageJ Common Apache Commons Codec AuthZForce PDP Core
Pit Descartes

27. 27
0 %
25 %
50 %
75 %
00 %
authzforce commons-codec commons-lang jaxen joda-ime sat4j-core spoon
Descartes Gregor

28. Coarser grain than PIT
28
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
long fact(int n){
return 0;}
long fact(int n){
return 1;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}

29. 29
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new
SdkTLSSocketF(SSLContext.getDefault(),null);
...
f.prepareSocket(new TestSSLSocket() {
...
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, expected));
});
}
protected final void prepareSocket(final SSLSocket socket) { }}

30. 30
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new
SdkTLSSocketF(SSLContext.getDefault(),null);
...
f.prepareSocket(new TestSSLSocket() {
...
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, expected));
});
}
protected final void prepareSocket(final SSLSocket socket) { }}

31. 31
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new
SdkTLSSocketF(SSLContext.getDefault(),null);
...
f.prepareSocket(new TestSSLSocket() {
...
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, expected));
});
}
protected final void prepareSocket(final SSLSocket socket) { }}
Missing oracle

32. 32
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}

33. 33
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}

34. 34
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}

35. Bug in test
35
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}

36. 36
0 %
25 %
50 %
75 %
100 %
authzforce commons-codec commons-lang imagej-common jgit jsoup scifio
78 %
94 %
76 %77 %
73 %
93 %96 %95 %
83 %81 %
96 %
72 %
93 %96 %95 %93 %93 %93 %
97 %
81 %
94 %
9 %
3 %
4 %3 %
2 %
2 %
1 %1 %
3 %
3 %
2 %
4 %
2 %
1 %1 %3 %2 %3 %1 %
4 %
1 %
12 %
3 %
21 %20 %
25 %
5 %3 %3 %
14 %16 %
2 %
24 %
5 %3 %3 %4 %4 %3 %2 %
15 %
5 %
Strong pseudo-tested
Weak pseudo-tested
Tested

37. Descartes in Maven
37

38. <plugins>
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.4.0</version>
<dependencies>
<dependency>
<groupId>eu.stamp-project</groupId>
<artifactId>descartes</artifactId>
<version>1.2</version>
</dependency>
</dependencies>
<configuration>
<verbose>true</verbose>
<mutationEngine>descartes</mutationEngine>
<mutators>
<mutator>true</mutator>
<mutator>”A string”</mutator>
<mutator>null</mutator>
<mutator>empty</mutator>
</mutators>
<outputFormats>
<value>ISSUES</value>
<value>METHODS</value>
<value>JSON</value>
</outputFormats>
</configuration>
</plugin>
</plugins>
PITest as a plugin
Descartes to be used by PITest
Regular PITest configuration
Descartes as the mutation engine
Values to use in extreme transforms. Supports literals for
primitive types, String, null and empty for empty arrays.
Reporting options. Custom reports ISSUES and METHODS for
testing issues and method categorization. JSON, HTML, XML for
regular PITest report.

39. Descartes in Jenkins
39

40. 40

41. 41

42. 42

43. Descartes in the GitHub check UI
43

44. Pull request based development
44
update
build
feedback
✗✔
review
Merge if OK

45. GitHub checks API
45

46. Pull request based development
46
update
build
feedback
review
Merge if OK

47. 47

48. 48

49. 49

50. 50

51. Descartes under the hood
51

52. Compiled files
Compiled test files
Source files
Test source files
Dependencies
PIT
Descartes
Operators to use
Number of execution threads
…

53. . . .
. . .
. . .
. . .
. . .
Compiled files
Compiled test files
Source files
Test source files
Dependencies
Configuration
PIT
Descartes
class A {
…
public void voidMethod() {}
public int intMethod() {}
public String strMethod() {}
…
}
void 3 “S” null
Report
Execution Unit
Operators to use
Number of execution threads
…
. . .
53

54. Descartes
•Current
• Compatible with JUnit5 and latest Pitest
• Integrates with Maven and Gradle
•Future: Descartes and CI
• Incremental Descartes on a commit
• Descartes for pull requests
54

55. Conclusion
•Mutation analysis
• Automatic generation of mutants
• Evaluate the test suite
•Bugs in test suites
• Oracle
• Input space coverage
• Testability
• Indirectly tested code
55

56. Feedback welcome!
• https://github.com/STAMP-project/pitest-descartes
• https://github.com/hcoles/pitest
• http://stamp-project.eu/
baudry@kth.se
56