12. Quis custodiet ipsos custodes?
Who will guard the guards themselves?
How do you tests your tests?
Changing code should cause test failures
Enter the mutants!
14. 1. Competent Programmer
● Code is close to correct
● Most faults are simple mistakes
2. Coupling Effects
● Simple faults are coupled to complex faults
● Fixing simple faults will fix complex faults
● Tests that detect mutants would also detect real bugs
Two Hypotheses
15. Fault based testing - intentionally break code
1. Test original class
2. Create mutant class - change one aspect
3. Test mutant class
4. Compare test results
“Open Box” testing - knows about internals
Mutation testing
R. Lipton, “Fault Diagnosis of Computer Programs”, Student Report, Carnegie Mellon University, 1971
17. Mutant Killed
● Changing code breaks tests
● Mutant is covered by test case
Mutant Survived
● Changing code did not break test
● Mutant is not covered by test case, or
● Mutant is equivalent to original class
Mutation Testing
21. One change per mutant
Avoid multiple mutations of the same line
Limit mutations of a single class
Large test suite means fewer mutants
● Target specific classes
Generating Mutants
22. Generate new mutants for each cycle
● They can’t get any deader
● Avoids maintenance overhead
Chance of regenerating identical mutants
● Change tactics
● Target only updated files
Generating Mutants
23. Computationally expensive
For each mutant
● Replace original code
● Deploy mutant
● Run entire test suite
● Restore original code
Execute, evaluate, repeat
● Execute every mutant, every time
Executing Mutants
24. Some mutants will not deploy
● Not a weakness of tests
● Developers should spot this!
● Continuous integration will catch this
Suppress (aka delete) them
Executing Mutants
25. Expensive
● Human Oracle required
● Manual checks for all survivors
Suppress equivalents
Calculate mutation score
● Indicator of test quality
Fix tests
● Ensure original code is still green
Evaluating, or Counting the Bodies
killed mutants
total mutants
Mutation Score
* 100
26. Well suited to automation
● What changes is less important than something changed
● Simple “mistakes” introduced
● Coupling effect says real bugs would be found
Repeated deploy/execute tests
Automation
27. New scratch org for cycle
Source tracking detects changes
● Replace production class with mutant on filesystem
● Reverse after testing
CLI to push source and run tests
SFDX is Key
34. Allow sufficient time
Easier with smaller files
● Fewer mutants
Mutation scores won’t surprise you!
● Lots of test effort = high score
● Focus on coverage = low score
Key Learnings
35. Mutation test at the end of development
Automatically generated perform as well as hand crafted
Constants are less helpful than expected
● Tests typically reference production values
Removing DML didn’t break tests
● Tendency to check returned value over database contents
Key Learnings
36. Key Learnings
Pros Cons
Detect untested code Vast number of potential mutants
Measurable mutation score Manual verification of surviving mutants
Improve test suite quality Full test run for each mutant
Tests how the code doesn’t work Knowledge of code internals required
37. CLI Plug-in on NPM - https://bobbuzz.me.uk/mutationplugin
sfdx plugins:install bbmutation
Source on Github - https://bobbuzz.me.uk/mutationsrc
Resources