1. Mutants to the rescue:
How effective are your
unit tests?

2. @DevPaco
Paco van Beckhoven
- Software engineer
- Automated warehouse team @ Picnic
About me

3. @DevPaco
“Quality”

4. @DevPaco
“The degree to which a system, component, or process meets
specified requirements” – IEEE
Software quality

5. @DevPaco
ISO/IEC 25010

6. @DevPaco
ISO/IEC 25010

7. @DevPaco

8. @DevPaco
Unit test example
public class Calculator {
public int multiply(int a, int b) {
return a * b;
}
}

9. @DevPaco
Unit test example
public class Calculator {
public int multiply(int a, int b) {
return a * b;
}
}
class CalculatorTest {
@Test
void testMultiply() {
assertEquals(20, new Calculator().multiply(4, 5));
}
@Test
void testMultiplyWithZero() {
assertEquals(0, new Calculator().multiply(0, 5));
}
}

10. @DevPaco
Maintenance

11. @DevPaco
Documentation

12. @DevPaco
Quick feedback
https://www.rainerhahnekamp.com/en/tag/agile/

13. @DevPaco
“ Always code as if the guy who ends up maintaining
your code will be a violent psychopath who knows
where you live ”
John F. Woods

14. @DevPaco

15. @DevPaco

16. @DevPaco
• Projects evolve, grow
• Tests evolve, sometimes missed when refactoring
• Test code often not monitored
The problem

17. @DevPaco
Code coverage
“The degree to which the source code of a program is
executed when a particular test suite runs”
Monitoring Tests

18. @DevPaco
Example
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

19. @DevPaco
Method coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

20. @DevPaco
Method coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));

21. @DevPaco
Statement coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));

22. @DevPaco
Statement coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));

23. @DevPaco
Condition coverage
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

24. @DevPaco
Condition coverage
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));
testZ: submit(proposal, 0, Instant.now().minusSeconds(999));
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}

25. @DevPaco
Pros:
• Helps you write more/better tests
• Easy/cheap to measure
• Shows what you didn’t test
• Shows that what you did test, didn’t crash
Code coverage

26. @DevPaco
But:
• Can be misleading
• Doesn’t give any guarantees
Code coverage

27. @DevPaco
But:
• Can be misleading
• Doesn’t give any guarantees
Code coverage
@Test
public void shouldReturnSuccessOnValidSubmit() {
proposalService.submit(randomProposal(), 0, tomorrow());
}

28. @DevPaco
• Testing the tests
• “Bug detection ability”
Test effectiveness
https://shirt.woot.com/offers/bug-hunt

29. @DevPaco
• Simulating bugs
• Typical programming errors
Mutation testing

30. @DevPaco
Step 1. Generating mutants
MUTATOR
Production code The mutant
Altered version of the
production code

31. @DevPaco
Example
boolean isAllowedToApplyForDriversLicense(int age) {
return age < 17;
}
boolean isAllowedToApplyForDriversLicense(int age) {
return age >= 17;
}

32. @DevPaco
Example
return age < 17; // negate conditional
return age > 17; // change conditional boundaries
return true; // true returns
return false; // false returns
boolean isAllowedToApplyForDriversLicense(int age) {
return age >= 17;
}

33. @DevPaco
Math mutators
int getTotalNumberOfLayers() {
return layers.size() + numberOfAdditionalLayers;
}
int getTotalNumberOfLayers() {
return layers.size() - numberOfAdditionalLayers;
}

34. @DevPaco
Removing void statements
MetaData createMetaData(Instant eventTimestamp) {
MetaData meta = new MetaData();
meta.setPublished(eventTimestamp);
return meta;
}
MetaData createMetaData(Instant eventTimestamp) {
MetaData meta = new MetaData();
return meta;
}

35. @DevPaco
Step 2. Mutation analysis
❌ Mutant survived
✅ Mutant killed
+ Tests

36. @DevPaco
The first generation mutation tools (for Java)
- Jester
- Jumble
- MuJava
Mutation tools

37. @DevPaco
• Created by Henry Coles in 2011
• Byte code mutation
• Optimized for coverage
• Easy to use
• Actively maintained/supported
PIT / Pitest

38. @DevPaco
1. Add the plugin
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.7.5</version>
</plugin>
2. Run pitest
> mvn clean verify pitest:mutationCoverage
Getting started

39. @DevPaco
Demo

40. @DevPaco
Reality

41. @DevPaco
• Limit number of mutants
<targetClasses>
<param>org.jfree.chart.util.*</param>
</targetClasses>
• Limit number of executed tests
<targetTests>
<param>org.jfree.chart.util.*</param>
</targetTests>
• Increase Threads
<threads>8</threads>
Start small!

42. @DevPaco
Tweaking performance

43. @DevPaco
• Reduce number of tests to run
<configuration>
<excludedTestClasses>foo.bar.acceptance.*</excludedTestClasses>
<excludedTestClasses>foo.bar.database.*</excludedTestClasses>
<excludedTestClasses>foo.bar.slooow.*</excludedTestClasses>
</configuration>
Tweaking performance

44. @DevPaco
• Exclude code from mutation
<configuration>
<excludedClasses>foo.bar.generated.*</excludedClasses>
<excludedClasses>*pdfgenerator.*</excludedClasses>
<excludedClasses>foo.bar.database.*</excludedClasses>
<excludedMethods>*ThreadManager.run*</excludedMethods>
</configuration>
Tweaking performance

45. @DevPaco
• Incremental analysis
• Experimental feature
• Reuse results from last run
Tweaking performance

46. @DevPaco
Keeping your reports clean
<configuration>
<avoidCallsTo>
<avoidCallsTo>java.util.logging</avoidCallsTo>
<avoidCallsTo>org.apache.log4j</avoidCallsTo>
<avoidCallsTo>org.slf4j</avoidCallsTo>
<avoidCallsTo>org.apache.commons.logging</avoidCallsTo>
</avoidCallsTo>
</configuration>
log.info(”Execution took {} seconds", end – start);
Tweaking performance

47. @DevPaco
Tweaking Mutators

48. @DevPaco
Tweaking Mutators

49. @DevPaco
Tweaking Mutators

50. @DevPaco
Equivalent mutants
MUTATOR
Replace == with >=
int i = 0;
while (true) {
i++;
if (i == MAX) {
break;
}
}
int i = 0;
while (true) {
i++;
if (i >= MAX) {
break;
}
}

51. @DevPaco
• Add the maven goal to your build step
mvn org.pitest:pitest-maven:mutationCoverage
• Plugins available for Jenkins/Sonar
• Pitest can be configured to break the pipeline
Adding mutation testing to CI

52. @DevPaco
When to consider mutation testing?

53. @DevPaco
Are there lives at stake?
Are you building software for a rocket?
Is it a self driving car?
When to consider mutation testing?

54. @DevPaco
Are you using code coverage?
What is the cost of fixing a bug?
Does the team think it’s a good idea?
When to consider mutation testing?

55. @DevPaco
• Code coverage: “How much of the code is tested”
• Mutation testing: “How proper is the code tested”
• Start small
• Tweak for performance!
Summary

56. @DevPaco
Thank you!
https://pitest.org/
Mutation testing for Java, Kotlin(paid)
https://stryker-mutator.io/
Supports JavaScript (typescript), C#, Scala
https://github.com/boxed/mutmut
Mutation testing tool for Python
@DevPaco
For questions, feedback, suggestions