Mutation test is a way to put a "mutation" into your code, run the test and then see if the test fails or not. A mutation is a change to production code which should make it behave in a different way. The idea is, if the code is just enough to pass the test, any mutation should make the test failed due to the behavior change. Although this statement is not always true, it should be correct in most cases.
By TDD, we should get just enough and the simplest code that passing all the test. But, how do we know this? Sometimes, since writing test and code is so interactive in TDD, we may not choose the smallest baby step to pass the test. This means we may write some code which is not driven by current test. On the other side, regardless doing TDD or not, we hope those unit tests can avoid future regression when we changing the code, right? To make it more clear, our expectation is that any "bad" code change (aka. mutation) can be detected by those tests.
5. Truth of Test Coverage
I expect a high level of coverage. Sometimes
managers require one. There's a subtle difference.
-- Brian Marick
http://martinfowler.com/bliki/TestCoverage.html
Tuesday, July 9, 13
8. Function Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Satisfied Test: foo(2, 2)
Tuesday, July 9, 13
9. Function Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Tuesday, July 9, 13
10. Statement Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Satisfied Test: foo(2, 2)
Tuesday, July 9, 13
11. Statement Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Unsatisfied Test: foo(2, -1)
Tuesday, July 9, 13
12. Decision Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Satisfied Test: foo(2, 2), foo(-1, 2)
Tuesday, July 9, 13
13. Decision Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Unsatisfied Test: foo(2, 2)
Tuesday, July 9, 13
14. Condition Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Satisfied Test: foo(2, 2), foo(2, -1), foo(-1, -1)
T/F
Tuesday, July 9, 13
15. Condition Coverage
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
T/F
Unsatisfied Test: foo(2, 2), foo(-1, 2)
T/F
Tuesday, July 9, 13
22. Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
Tuesday, July 9, 13
23. Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
Tuesday, July 9, 13
24. Redundant Code?
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
sideEffect(z);
return z;
}
n Missing test to prove the sideEffect call is needed
n If code is redundant, it can’t be learnt via any
classical test coverage
Tuesday, July 9, 13
25. Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
26. Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>=0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
27. Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>=0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
Test to cover this Mutator:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, 0))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
28. Conditional Boundary Mutator
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0)) {
z = x;
}
return z;
}
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>=0)) {
z = x;
}
return z;
}
Condition Coverage Satisfied Test:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, -1))
assertEquals(0, foo(-1, 2))
n Test not prevent “bad” code change to happen
Test to cover this Mutator:
assertEquals(2, foo(2, 2))
assertEquals(0, foo(2, 0))
assertEquals(0, foo(-1, 2))
Tuesday, July 9, 13
31. TDD and Test Coverage
n You can most likely ignore Classical Test Coverage
n You can’t ignore Mutation Test Coverage
n If not TDD, then you can learn a lot from both
Classical and Mutation Test Coverage
Tuesday, July 9, 13
36. Reference
n Martin Fowler’s blog http://martinfowler.com/bliki/
TestCoverage.html
n Mutation Test Wikipedia http://en.wikipedia.org/
wiki/Mutation_testing
n Mutation Test Analysis Report http://
crestweb.cs.ucl.ac.uk/resources/
mutation_testing_repository/TR-09-06.pdf
Tuesday, July 9, 13