As metrics go, Test Coverage is a real bad one. Not only is it easy to game on purpose, 100% does not guarantee anything. Fortunately, Mutation Testing can help!

1. @nicolas_frankel
Mutation Testing to the
rescue of your tests

2. @nicolas_frankel
• Developer, team lead, architect,
whatever it takes
• Recent Developer Advocate
• Kotlin fanboy
Me, myself and I

3. @nicolas_frankel
Hazelcast
HAZELCAST IMDG is an operational,
in-memory, distributed computing
platform that manages data using
in-memory storage, and performs
parallel execution for breakthrough
application speed and scale.
HAZELCAST JET is the ultra fast,
application embeddable, 3rd
generation stream processing
engine for low latency batch
and stream processing.

4. @nicolas_frankel
• Unit Testing
• Integration Testing
• End-to-end Testing
• Performance Testing
• Penetration Testing
• Exploratory Testing
• etc.
Many kinds of testing

5. @nicolas_frankel
Ensure the quality of the production
code
Their only single goal
5

6. @nicolas_frankel
How to check the quality of the
testing code?
The problem
6

7. @nicolas_frankel
“Code coverage is a measure used to
describe the degree to which the
source code of a program is tested”
--Wikipedia
http://en.wikipedia.org/wiki/Code_coverage
Code coverage
7

8. @nicolas_frankel
Check whether a source code line is
executed during a test
• Or Branch Coverage
Measuring Code Coverage
@nicolas_frankel #mutationtesting #devexperience18
8

9. @nicolas_frankel
Computing Code Coverage
CC =
Lexecuted
Ltotal
*100
• CC: Code
Coverage
(in percent)
• Lexecuted: Number
of executed lines
of code
• Ltotal: Number of
total lines of
code

10. @nicolas_frankel
• JaCoCo
• Clover
• Cobertura
• etc.
Java Tools for Code Coverage
10

11. @nicolas_frankel
“Is 100% code coverage realistic? Of
course it is. If you can write a line of
code, you can write another that
tests it.”
Robert Martin (Uncle Bob)
https://twitter.com/unclebobmartin/status/55966
620509667328
100% Code Coverage?
11

12. @nicolas_frankel
@Test
public void add_should_add() {
new Math().add(1, 1);
}
Assertless testing
But, where is the assert?
As long as the Code Coverage is OK…

13. @nicolas_frankel
• Any metric can be gamed!
• Code coverage is a metric…
⇒ Code coverage can be gamed
• On purpose
• Or by accident
Code coverage as a measure of quality
13

14. @nicolas_frankel
• Code Coverage lulls you into a false
sense of security…
Code coverage as a measure of quality
14

15. @nicolas_frankel
• Code coverage cannot ensure test
quality
• Is there another way?
• Mutation Testing to the rescue!
The problem still stands
15

16. @nicolas_frankel
The Cast
William Stryker
Original Source Code
Jason Stryker
Modified Source Code
a.k.a “The Mutant”

17. @nicolas_frankel
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}

18. @nicolas_frankel
Standard testing
✔Execute Test

19. @nicolas_frankel
Mutation testing
?Execute SAME Test
Mutation

20. @nicolas_frankel
Mutation testing
✗
✔Execute SAME Test
Execute SAME Test
Mutant Killed
Mutant Survived

21. @nicolas_frankel
• Surviving means changing the
source code did not change the
test result  Bad!
• Killed means changing the source
code changed the test result 
Good!
Killed or Surviving?
21

22. @nicolas_frankel
Test the code
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
✔Execute Test
@Test
public void add_should_add() {
new Math().add(1, 1);
}

23. @nicolas_frankel
✔Execute Test
Surviving mutant
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
}

24. @nicolas_frankel
Test the code
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}

25. @nicolas_frankel
Killed mutant
✗Execute SAME Test
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}

26. @nicolas_frankel
• PIT is a tool for Mutation testing
• Available as
• Command-line tool
• Ant target
• Maven plugin
Mutation Testing in Java
26

27. @nicolas_frankel
Mutators are patterns applied to
source code to produce mutations
Mutators
27

28. @nicolas_frankel
PIT mutators sample
Name Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=
Remove Conditionals foo == bar true
Math + -
Increments foo++ foo--
Invert Negatives -foo foo
Inline Constant static final FOO= 42 static final FOO = 43
Return Values return true return false
Void Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis() long t = 0
Constructor Call Date d = new Date() Date d = null;

29. @nicolas_frankel
Conditionals Boundary
• Potential serious bug hiding there
• if (foo > bar)
Important mutators
29

30. @nicolas_frankel
Void Method Call
• Assert.checkNotNull()
• connection.close()
Important mutators
30

31. @nicolas_frankel
It’s not because the IDE generates
code safely that it will never change
• equals()
• hashCode()
Remember
31

32. @nicolas_frankel

33. @nicolas_frankel
Sh… (bad stuff) happens
• False positives
• Imperfect
• Sloooooooooooooooow

34. @nicolas_frankel
if (p < 0)
...
// changed condition boundary -> survived:
if (p > 0)
...
return 0;
Pit is imperfect

35. @nicolas_frankel
void rebootMachine() {
// removed method call:
checkUserPermissions();
Runtime.getRuntime().exec("reboot");
}
Pit is imperfect

36. @nicolas_frankel
• Analyze test code
• For each class under test
• For each mutator
• Create mutation
• For each mutation
• Run test
• Analyze result
• Aggregate results
Why so slow?
36

37. @nicolas_frankel
• Increase number of threads  
• Set a limited a set of mutators
• Limit scope of target classes
• Limit number of tests
• Limit dependency distance
• Use incremental analysis  
• Don’t bind to the test phase  
• Use scmMutationCoverage
Workarounds
37

38. @nicolas_frankel
• Metadata stored between runs
• Mutant will not be checked again, if:
• killed, and neither class nor test have changed
• survived, and there are no new/changed tests for it
Incremental analysis

39. @nicolas_frankel
• Checks the relevance of your unit
tests
• Points out potential bugs
Is Mutation Testing the Silver Bullet?
39

40. @nicolas_frankel
• Validate the assembled application
• Integration Testing
• Check the performance
• Performance Testing
• Look out for display bugs
• End-to-end testing
• Etc.
The rest is up to you
40

41. @nicolas_frankel
• Don’t test to achieve 100%
coverage
• Test because it saves money in the
long run
• Prioritize:
• Business-critical code
• Complex code
Testing is about ROI
41

42. @nicolas_frankel
• https://blog.frankel.ch/
• @nicolas_frankel
• https://git.io/vznQK
Thanks a lot!