SlideShare una empresa de Scribd logo
1 de 47
Descargar para leer sin conexión
Mutate and Test your Tests
Benoit Baudry (KTH, baudry@kth.se)
Oscar Vera Perez (INRIA), Martin Monperrus (KTH)
Spotify, Stockholm. May 2018.
1
Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
2
Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs
3
4
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
5
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
Coverage
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
6
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
Coverage
7
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
8
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
9
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Are these test cases good at
detecting bugs?
Let’s mutate our code to see.
Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
10
Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
•Mutation operators = types of bugs
•Mutant = Program with one seeded bug
11
Mutation analysis
Input : P, TS, Ops
Output : score, coverage
M <- generateMutants (P, OPs)
forAll (m in M)
run (TS,m)
if (one-test-fail)
then killed <- m
else alive <- m
score = killed / size(M)
12
PITest mutation operators
•Conditions
•Constants
•Return
•Delete method calls
•Constructor call
13
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
14
long fact(int n) {
if(n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
15
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
16
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
17
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
18
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Mutation reveals bugs in the test suite
19
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
Bugs in the test suite:
- Weak oracle
- Missing input
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
20
@Test
factorialWith5Test() {
long obs = fact(5);
assertEqual(120, obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
@Test
factorialWith1Test() {
assertEqual(1, fact(1));}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1
Project #Mutants Time (h) Score (%)
Amazon Web Services SDK 2141690 04:25:35 76.28
XWiki Rendering Engine 112609 01:59:55 50.89
Apache Commons Math 104786 03:22:18 83.81
JFreeChart 89592 00:41:28 58.04
Apache PdfBox 79763 06:20:25 58.89
Java Git 78316 16:02:00 73.86
SCIFIO 62768 03:12:11 45.92
Joda-Time 31233 00:16:32 81.65
Apache Commons Lang 30361 00:21:02 86.17
Apache Commons Collections 20394 00:05:41 85.94
Urban Airship Client Library 17345 00:11:31 82.26
SAT4J 17067 11:11:04 68.58
ImageJ Common 15592 00:29:09 54.77
jsoup 14054 00:12:49 78.34
Jaxen XPath Engine 12210 00:24:40 67.13
Apache Commons Codec 9233 00:07:57 87.82
Apache Commons IO 8809 00:12:48 84.73
Google Gson 7353 00:05:34 81.76
AuthZForce PDP Core 7296 01:23:50 88.18
Apache Commons CLI 2560 00:01:26 88.71
JOpt Simple 2271 00:01:36 93.52
21
Descartes
•Mutation operators: extreme mutation
•Active, open source development
•Pitest plugin
•Compared to PIT
• Less mutants
• Different type of feedback
• Same framework
22
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
23
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
Descartes - example
24
long fact(int n){
return 0;}
long fact(int n){
return 1;}
PIT : 7 mutants
Descartes : 2 mutants
Scalability of extreme mutation
25
public static boolean isValidXmlChar(int ch){
return (ch == 0x9)
|| (ch == 0xA)
|| (ch == 0xD)
|| (ch >= 0x20 && ch <= 0xD7FF)
|| (ch >= 0xE000 && ch <= 0xFFFD
|| (ch >= 0x10000 && ch <= 0x10FFFF);
}
PIT : 45 mutants
Descartes : 2 mutants
Project Mutants PIT Mutants Descartes Time PIT Time Descartes
Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30
XWiki Rendering Engine 112609 5534 01:59:55 00:10:50
Apache Commons Math 104786 7150 03:22:18 00:08:30
JFreeChart 89592 7210 00:41:28 00:05:26
Apache PdfBox 79763 7559 06:20:25 00:42:11
Java Git 78316 7152 16:02:00 00:56:07
SCIFIO 62768 3627 03:12:11 00:15:26
Joda-Time 31233 4525 00:16:32 00:04:13
Apache Commons Lang 30361 3872 00:21:02 00:02:18
Apache Commons Collections 20394 3558 00:05:41 00:01:48
Urban Airship Client Library 17345 3082 00:11:31 00:09:38
SAT4J 17067 2296 11:11:04 00:56:42
ImageJ Common 15592 1947 00:29:09 00:04:08
jsoup 14054 1566 00:12:49 00:02:45
Jaxen XPath Engine 12210 1252 00:24:40 00:01:34
Apache Commons Codec 9233 979 00:07:57 00:02:02
Apache Commons IO 8809 1164 00:12:48 00:02:18
Google Gson 7353 848 00:05:34 00:01:11
AuthZForce PDP Core 7296 626 01:23:50 00:08:45
Apache Commons CLI 2560 271 00:01:26 00:00:09
JOpt Simple 2271 412 00:01:36 00:00:25
26
Coarser grain than PIT
27
long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
long fact(int n){
return 0;}
long fact(int n){
return 1;}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs); }
@Test
factorialWith5Test() {
assertEqual(1, fact(0));}
28
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null);
...
f.prepareSocket(new TestSSLSocket() {
...
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, expected));
});
}
protected final void prepareSocket(final SSLSocket socket) { }}
29
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null);
...
f.prepareSocket(new TestSSLSocket() {
...
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, expected));
});
}
protected final void prepareSocket(final SSLSocket socket) { }}
30
@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null);
...
f.prepareSocket(new TestSSLSocket() {
...
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, expected));
});
}
protected final void prepareSocket(final SSLSocket socket) { }}
Missing oracle
31
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
32
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}
33
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}
Bug in test
34
bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
bool equals(object other) {
return true;}
bool equals(object other) {
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}
35
1
10
100
1000
10000
100000
1000000
10000000
Number of mutants
Pit Descartes
36
15
285
4 15 54 28 64 132 23
115
14
594 369
92 9 40
158
458 116
65
2592
69
2
14 28 19 27 55
10
17
12
97
85
31 2
14
11
61 20
69
197
274
1516
175 408 1178 610 1840 4245 459
344
585
2931 2237
2596 252 714
456
1726 425
2004
1618
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Tested
Weak pseudo-tested
Strong pseudo-tested
Quick journey in spotify/apollo
37
38
3
2
1
10 2
1
1
2 1
1
1
24
49
11 19
14
50 11 93
29
25
20
18
3
33
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
package com.spotify.apollo.request;
class TrackedOngoingRequestImpl extends ForwardingOngoingRequest {
public void reply(Response<ByteString> message) {
doReply(message);
}
private boolean doReply(Response<ByteString> msg) {
final boolean removed = requestTracker.remove(this);
if (removed) {
super.reply(message);}
return removed;
}
}
39
package com.spotify.apollo.request;
class TrackedOngoingRequestImpl extends ForwardingOngoingRequest {
public void reply(Response<ByteString> message) {
doReply(message);
}
private boolean doReply(Response<ByteString> msg) {
final boolean removed = requestTracker.remove(this);
if (removed) {
super.reply(message);}
return removed;
}
}
40
public void reply(Response<ByteString> message) {}
private boolean doReply(Response<ByteString> msg) {return true;}
private boolean doReply(Response<ByteString> msg) {return false;}
@Test
shouldNotReplyIfNotTracked() {
tOR = new TrackedOngoingRequestImpl(o,t);
tracker.remove(tOR);
tOR.reply(Response.ok());
verifyNoMoreInteractions(ongoingRequest);
}
41
public void reply(Response<ByteString> message) {}
private boolean doReply(Response<ByteString> msg) {return true;}
private boolean doReply(Response<ByteString> msg) {return false;}
package com.spotify.apollo.request;
class TrackedOngoingRequestImpl extends ForwardingOngoingRequest {
public void reply(Response<ByteString> message) {
doReply(message);
}
private boolean doReply(Response<ByteString> msg) {
final boolean removed = requestTracker.remove(this);
if (removed) {
super.reply(message);}
return removed;
}
}
@Test
shouldNotReplyIfNotTracked() {
tOR = new TrackedOngoingRequestImpl(o,t);
tracker.remove(tOR);
tOR.reply(Response.ok());
verifyNoMoreInteractions(ongoingRequest);
}
42
package com.spotify.apollo.environment;
public class ApolloConfig {
public boolean enableMetaApi() {
return optionalBoolean(apolloNode, "metaApi").orElse(true);
}
43
public boolean enableMetaApi() {return true;}
public boolean enableMetaApi() {return false;}
package com.spotify.apollo.environment;
public class ApolloConfig {
public boolean enableMetaApi() {
return optionalBoolean(apolloNode, "metaApi").orElse(true);
}
Covered by 8 test cases which all expect
enableMetaApi()to return true
44
package com.spotify.apollo.environment;
public class ApolloConfig {
public boolean enableMetaApi() {
return optionalBoolean(apolloNode, "metaApi").orElse(true);
}
public boolean enableMetaApi() {return true;}
public boolean enableMetaApi() {return false;}
Descartes
•Current
• Compatible with JUnit5 and latest Pitest
• Integrates with Maven and Gradle
•Future: Descartes and CI
• Incremental Descartes on a commit
• Descartes for pull requests
45
Conclusion
•Mutation analysis
• Automatic generation of mutants
• Evaluate the test suite
•Bugs in test suites
• Oracle
• Input space coverage
• Testability
• Indirectly tested code
46
Feedback welcome!
• https://github.com/STAMP-project/pitest-descartes
• https://github.com/hcoles/pitest
• http://stamp-project.eu/
baudry@kth.se
47

Más contenido relacionado

La actualidad más candente

Java practical(baca sem v)
Java practical(baca sem v)Java practical(baca sem v)
Java practical(baca sem v)mehul patel
 
Java Puzzle
Java PuzzleJava Puzzle
Java PuzzleSFilipp
 
Java practice programs for beginners
Java practice programs for beginnersJava practice programs for beginners
Java practice programs for beginnersishan0019
 
Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*Intel® Software
 
12. Java Exceptions and error handling
12. Java Exceptions and error handling12. Java Exceptions and error handling
12. Java Exceptions and error handlingIntro C# Book
 
Java_practical_handbook
Java_practical_handbookJava_practical_handbook
Java_practical_handbookManusha Dilan
 
Core java pract_sem iii
Core java pract_sem iiiCore java pract_sem iii
Core java pract_sem iiiNiraj Bharambe
 
JDays 2016 - Beyond Lambdas - the Aftermath
JDays 2016 - Beyond Lambdas - the AftermathJDays 2016 - Beyond Lambdas - the Aftermath
JDays 2016 - Beyond Lambdas - the AftermathDaniel Sawano
 
Dealing with combinatorial explosions and boring tests
Dealing with combinatorial explosions and boring testsDealing with combinatorial explosions and boring tests
Dealing with combinatorial explosions and boring testsAlexander Tarlinder
 
Transferring Software Testing and Analytics Tools to Practice
Transferring Software Testing and Analytics Tools to PracticeTransferring Software Testing and Analytics Tools to Practice
Transferring Software Testing and Analytics Tools to PracticeTao Xie
 
Spotify 2016 - Beyond Lambdas - the Aftermath
Spotify 2016 - Beyond Lambdas - the AftermathSpotify 2016 - Beyond Lambdas - the Aftermath
Spotify 2016 - Beyond Lambdas - the AftermathDaniel Sawano
 
EdSketch: Execution-Driven Sketching for Java
EdSketch: Execution-Driven Sketching for JavaEdSketch: Execution-Driven Sketching for Java
EdSketch: Execution-Driven Sketching for JavaLisa Hua
 
Computer java programs
Computer java programsComputer java programs
Computer java programsADITYA BHARTI
 

La actualidad más candente (20)

Java practical(baca sem v)
Java practical(baca sem v)Java practical(baca sem v)
Java practical(baca sem v)
 
Java Puzzle
Java PuzzleJava Puzzle
Java Puzzle
 
Java practical
Java practicalJava practical
Java practical
 
Java practice programs for beginners
Java practice programs for beginnersJava practice programs for beginners
Java practice programs for beginners
 
Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*Tools and Techniques for Understanding Threading Behavior in Android*
Tools and Techniques for Understanding Threading Behavior in Android*
 
Java puzzles
Java puzzlesJava puzzles
Java puzzles
 
12. Java Exceptions and error handling
12. Java Exceptions and error handling12. Java Exceptions and error handling
12. Java Exceptions and error handling
 
Testability for Developers
Testability for DevelopersTestability for Developers
Testability for Developers
 
Java Puzzlers
Java PuzzlersJava Puzzlers
Java Puzzlers
 
PRACTICAL COMPUTING
PRACTICAL COMPUTINGPRACTICAL COMPUTING
PRACTICAL COMPUTING
 
Java_practical_handbook
Java_practical_handbookJava_practical_handbook
Java_practical_handbook
 
Core java pract_sem iii
Core java pract_sem iiiCore java pract_sem iii
Core java pract_sem iii
 
DCN Practical
DCN PracticalDCN Practical
DCN Practical
 
JDays 2016 - Beyond Lambdas - the Aftermath
JDays 2016 - Beyond Lambdas - the AftermathJDays 2016 - Beyond Lambdas - the Aftermath
JDays 2016 - Beyond Lambdas - the Aftermath
 
Dealing with combinatorial explosions and boring tests
Dealing with combinatorial explosions and boring testsDealing with combinatorial explosions and boring tests
Dealing with combinatorial explosions and boring tests
 
Transferring Software Testing and Analytics Tools to Practice
Transferring Software Testing and Analytics Tools to PracticeTransferring Software Testing and Analytics Tools to Practice
Transferring Software Testing and Analytics Tools to Practice
 
Spotify 2016 - Beyond Lambdas - the Aftermath
Spotify 2016 - Beyond Lambdas - the AftermathSpotify 2016 - Beyond Lambdas - the Aftermath
Spotify 2016 - Beyond Lambdas - the Aftermath
 
Stamp breizhcamp 2019
Stamp breizhcamp 2019Stamp breizhcamp 2019
Stamp breizhcamp 2019
 
EdSketch: Execution-Driven Sketching for Java
EdSketch: Execution-Driven Sketching for JavaEdSketch: Execution-Driven Sketching for Java
EdSketch: Execution-Driven Sketching for Java
 
Computer java programs
Computer java programsComputer java programs
Computer java programs
 

Similar a Mutation @ Spotify

Mutate and Test your Tests
Mutate and Test your TestsMutate and Test your Tests
Mutate and Test your TestsSTAMP Project
 
The CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestionsThe CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestionsCaroline Landry
 
How to add an optimization for C# to RyuJIT
How to add an optimization for C# to RyuJITHow to add an optimization for C# to RyuJIT
How to add an optimization for C# to RyuJITEgor Bogatov
 
Mutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The BugsMutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The BugsAri Waller
 
2012 JDays Bad Tests Good Tests
2012 JDays Bad Tests Good Tests2012 JDays Bad Tests Good Tests
2012 JDays Bad Tests Good TestsTomek Kaczanowski
 
(Www.entrance exam.net)-ignou mca solved assignment 2011
(Www.entrance exam.net)-ignou mca  solved assignment 2011(Www.entrance exam.net)-ignou mca  solved assignment 2011
(Www.entrance exam.net)-ignou mca solved assignment 2011swatith
 
Clone Refactoring with Lambda Expressions
Clone Refactoring with Lambda ExpressionsClone Refactoring with Lambda Expressions
Clone Refactoring with Lambda ExpressionsNikolaos Tsantalis
 
An introduction to Google test framework
An introduction to Google test frameworkAn introduction to Google test framework
An introduction to Google test frameworkAbner Chih Yi Huang
 
ALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveSanderSlideShare
 
33rd Degree 2013, Bad Tests, Good Tests
33rd Degree 2013, Bad Tests, Good Tests33rd Degree 2013, Bad Tests, Good Tests
33rd Degree 2013, Bad Tests, Good TestsTomek Kaczanowski
 
Data Structure: Algorithm and analysis
Data Structure: Algorithm and analysisData Structure: Algorithm and analysis
Data Structure: Algorithm and analysisDr. Rajdeep Chatterjee
 
Java™ (OOP) - Chapter 5: "Methods"
Java™ (OOP) - Chapter 5: "Methods"Java™ (OOP) - Chapter 5: "Methods"
Java™ (OOP) - Chapter 5: "Methods"Gouda Mando
 
ch04-conditional-execution.ppt
ch04-conditional-execution.pptch04-conditional-execution.ppt
ch04-conditional-execution.pptMahyuddin8
 
Ruslan Shevchenko - Property based testing
Ruslan Shevchenko - Property based testingRuslan Shevchenko - Property based testing
Ruslan Shevchenko - Property based testingIevgenii Katsan
 
Where the wild things are - Benchmarking and Micro-Optimisations
Where the wild things are - Benchmarking and Micro-OptimisationsWhere the wild things are - Benchmarking and Micro-Optimisations
Where the wild things are - Benchmarking and Micro-OptimisationsMatt Warren
 
Unit testing patterns for concurrent code
Unit testing patterns for concurrent codeUnit testing patterns for concurrent code
Unit testing patterns for concurrent codeDror Helper
 
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...STAMP Project
 
16. Java stacks and queues
16. Java stacks and queues16. Java stacks and queues
16. Java stacks and queuesIntro C# Book
 

Similar a Mutation @ Spotify (20)

Mutate and Test your Tests
Mutate and Test your TestsMutate and Test your Tests
Mutate and Test your Tests
 
The CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestionsThe CI as a partner for test improvement suggestions
The CI as a partner for test improvement suggestions
 
How to add an optimization for C# to RyuJIT
How to add an optimization for C# to RyuJITHow to add an optimization for C# to RyuJIT
How to add an optimization for C# to RyuJIT
 
Mutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The BugsMutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The Bugs
 
2012 JDays Bad Tests Good Tests
2012 JDays Bad Tests Good Tests2012 JDays Bad Tests Good Tests
2012 JDays Bad Tests Good Tests
 
(Www.entrance exam.net)-ignou mca solved assignment 2011
(Www.entrance exam.net)-ignou mca  solved assignment 2011(Www.entrance exam.net)-ignou mca  solved assignment 2011
(Www.entrance exam.net)-ignou mca solved assignment 2011
 
Clone Refactoring with Lambda Expressions
Clone Refactoring with Lambda ExpressionsClone Refactoring with Lambda Expressions
Clone Refactoring with Lambda Expressions
 
An introduction to Google test framework
An introduction to Google test frameworkAn introduction to Google test framework
An introduction to Google test framework
 
ALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra deriveALE2014 let tests drive or let dijkstra derive
ALE2014 let tests drive or let dijkstra derive
 
Best practices unit testing
Best practices unit testing Best practices unit testing
Best practices unit testing
 
33rd Degree 2013, Bad Tests, Good Tests
33rd Degree 2013, Bad Tests, Good Tests33rd Degree 2013, Bad Tests, Good Tests
33rd Degree 2013, Bad Tests, Good Tests
 
Data Structure: Algorithm and analysis
Data Structure: Algorithm and analysisData Structure: Algorithm and analysis
Data Structure: Algorithm and analysis
 
Java™ (OOP) - Chapter 5: "Methods"
Java™ (OOP) - Chapter 5: "Methods"Java™ (OOP) - Chapter 5: "Methods"
Java™ (OOP) - Chapter 5: "Methods"
 
ch04-conditional-execution.ppt
ch04-conditional-execution.pptch04-conditional-execution.ppt
ch04-conditional-execution.ppt
 
Golang dot-testing-lite
Golang dot-testing-liteGolang dot-testing-lite
Golang dot-testing-lite
 
Ruslan Shevchenko - Property based testing
Ruslan Shevchenko - Property based testingRuslan Shevchenko - Property based testing
Ruslan Shevchenko - Property based testing
 
Where the wild things are - Benchmarking and Micro-Optimisations
Where the wild things are - Benchmarking and Micro-OptimisationsWhere the wild things are - Benchmarking and Micro-Optimisations
Where the wild things are - Benchmarking and Micro-Optimisations
 
Unit testing patterns for concurrent code
Unit testing patterns for concurrent codeUnit testing patterns for concurrent code
Unit testing patterns for concurrent code
 
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
 
16. Java stacks and queues
16. Java stacks and queues16. Java stacks and queues
16. Java stacks and queues
 

Más de STAMP Project

Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenSTAMP Project
 
STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project
 
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP Project
 
20181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday201820181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday2018STAMP Project
 
1803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum20181803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum2018STAMP Project
 
Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17STAMP Project
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP Project
 

Más de STAMP Project (7)

Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, Sweden
 
STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018
 
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
 
20181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday201820181106 arie van_deursen_testday2018
20181106 arie van_deursen_testday2018
 
1803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum20181803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum2018
 
Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017
 

Último

Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdftimtebeek1
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypseTomasz Kowalczewski
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmuxevmux96
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Conceptsthomashtkim
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...Neo4j
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConNatan Silnitsky
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaNeo4j
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Maxim Salnikov
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...OnePlan Solutions
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jNeo4j
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringPrakhyath Rai
 
Test Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdfTest Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdfkalichargn70th171
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio, Inc.
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Andreas Granig
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksJinanKordab
 

Último (20)

Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
Abortion Clinic Pretoria ](+27832195400*)[ Abortion Clinic Near Me ● Abortion...
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
 
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements Engineering
 
Test Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdfTest Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdf
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with Links
 

Mutation @ Spotify

  • 1. Mutate and Test your Tests Benoit Baudry (KTH, baudry@kth.se) Oscar Vera Perez (INRIA), Martin Monperrus (KTH) Spotify, Stockholm. May 2018. 1
  • 2. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 2
  • 3. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 3
  • 4. 4 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  • 5. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 5 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  • 6. Coverage long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 6 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  • 7. Coverage 7 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}
  • 8. 8 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs?
  • 9. 9 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs? Let’s mutate our code to see.
  • 10. Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests 10
  • 11. Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests •Mutation operators = types of bugs •Mutant = Program with one seeded bug 11
  • 12. Mutation analysis Input : P, TS, Ops Output : score, coverage M <- generateMutants (P, OPs) forAll (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(M) 12
  • 14. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 14
  • 15. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 15 n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 16. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 16 n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 17. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 17 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 18. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 18 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 19. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Mutation reveals bugs in the test suite 19 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} Bugs in the test suite: - Weak oracle - Missing input n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 20. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 20 @Test factorialWith5Test() { long obs = fact(5); assertEqual(120, obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} @Test factorialWith1Test() { assertEqual(1, fact(1));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  • 21. Project #Mutants Time (h) Score (%) Amazon Web Services SDK 2141690 04:25:35 76.28 XWiki Rendering Engine 112609 01:59:55 50.89 Apache Commons Math 104786 03:22:18 83.81 JFreeChart 89592 00:41:28 58.04 Apache PdfBox 79763 06:20:25 58.89 Java Git 78316 16:02:00 73.86 SCIFIO 62768 03:12:11 45.92 Joda-Time 31233 00:16:32 81.65 Apache Commons Lang 30361 00:21:02 86.17 Apache Commons Collections 20394 00:05:41 85.94 Urban Airship Client Library 17345 00:11:31 82.26 SAT4J 17067 11:11:04 68.58 ImageJ Common 15592 00:29:09 54.77 jsoup 14054 00:12:49 78.34 Jaxen XPath Engine 12210 00:24:40 67.13 Apache Commons Codec 9233 00:07:57 87.82 Apache Commons IO 8809 00:12:48 84.73 Google Gson 7353 00:05:34 81.76 AuthZForce PDP Core 7296 01:23:50 88.18 Apache Commons CLI 2560 00:01:26 88.71 JOpt Simple 2271 00:01:36 93.52 21
  • 22. Descartes •Mutation operators: extreme mutation •Active, open source development •Pitest plugin •Compared to PIT • Less mutants • Different type of feedback • Same framework 22
  • 23. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 23
  • 24. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 24 long fact(int n){ return 0;} long fact(int n){ return 1;} PIT : 7 mutants Descartes : 2 mutants
  • 25. Scalability of extreme mutation 25 public static boolean isValidXmlChar(int ch){ return (ch == 0x9) || (ch == 0xA) || (ch == 0xD) || (ch >= 0x20 && ch <= 0xD7FF) || (ch >= 0xE000 && ch <= 0xFFFD || (ch >= 0x10000 && ch <= 0x10FFFF); } PIT : 45 mutants Descartes : 2 mutants
  • 26. Project Mutants PIT Mutants Descartes Time PIT Time Descartes Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30 XWiki Rendering Engine 112609 5534 01:59:55 00:10:50 Apache Commons Math 104786 7150 03:22:18 00:08:30 JFreeChart 89592 7210 00:41:28 00:05:26 Apache PdfBox 79763 7559 06:20:25 00:42:11 Java Git 78316 7152 16:02:00 00:56:07 SCIFIO 62768 3627 03:12:11 00:15:26 Joda-Time 31233 4525 00:16:32 00:04:13 Apache Commons Lang 30361 3872 00:21:02 00:02:18 Apache Commons Collections 20394 3558 00:05:41 00:01:48 Urban Airship Client Library 17345 3082 00:11:31 00:09:38 SAT4J 17067 2296 11:11:04 00:56:42 ImageJ Common 15592 1947 00:29:09 00:04:08 jsoup 14054 1566 00:12:49 00:02:45 Jaxen XPath Engine 12210 1252 00:24:40 00:01:34 Apache Commons Codec 9233 979 00:07:57 00:02:02 Apache Commons IO 8809 1164 00:12:48 00:02:18 Google Gson 7353 848 00:05:34 00:01:11 AuthZForce PDP Core 7296 626 01:23:50 00:08:45 Apache Commons CLI 2560 271 00:01:26 00:00:09 JOpt Simple 2271 412 00:01:36 00:00:25 26
  • 27. Coarser grain than PIT 27 long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} long fact(int n){ return 0;} long fact(int n){ return 1;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}
  • 28. 28 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); ... f.prepareSocket(new TestSSLSocket() { ... @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, expected)); }); } protected final void prepareSocket(final SSLSocket socket) { }}
  • 29. 29 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); ... f.prepareSocket(new TestSSLSocket() { ... @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, expected)); }); } protected final void prepareSocket(final SSLSocket socket) { }}
  • 30. 30 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); ... f.prepareSocket(new TestSSLSocket() { ... @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, expected)); }); } protected final void prepareSocket(final SSLSocket socket) { }} Missing oracle
  • 31. 31 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}
  • 32. 32 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  • 33. 33 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  • 34. Bug in test 34 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  • 36. 36 15 285 4 15 54 28 64 132 23 115 14 594 369 92 9 40 158 458 116 65 2592 69 2 14 28 19 27 55 10 17 12 97 85 31 2 14 11 61 20 69 197 274 1516 175 408 1178 610 1840 4245 459 344 585 2931 2237 2596 252 714 456 1726 425 2004 1618 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Tested Weak pseudo-tested Strong pseudo-tested
  • 37. Quick journey in spotify/apollo 37
  • 38. 38 3 2 1 10 2 1 1 2 1 1 1 24 49 11 19 14 50 11 93 29 25 20 18 3 33 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
  • 39. package com.spotify.apollo.request; class TrackedOngoingRequestImpl extends ForwardingOngoingRequest { public void reply(Response<ByteString> message) { doReply(message); } private boolean doReply(Response<ByteString> msg) { final boolean removed = requestTracker.remove(this); if (removed) { super.reply(message);} return removed; } } 39
  • 40. package com.spotify.apollo.request; class TrackedOngoingRequestImpl extends ForwardingOngoingRequest { public void reply(Response<ByteString> message) { doReply(message); } private boolean doReply(Response<ByteString> msg) { final boolean removed = requestTracker.remove(this); if (removed) { super.reply(message);} return removed; } } 40 public void reply(Response<ByteString> message) {} private boolean doReply(Response<ByteString> msg) {return true;} private boolean doReply(Response<ByteString> msg) {return false;} @Test shouldNotReplyIfNotTracked() { tOR = new TrackedOngoingRequestImpl(o,t); tracker.remove(tOR); tOR.reply(Response.ok()); verifyNoMoreInteractions(ongoingRequest); }
  • 41. 41 public void reply(Response<ByteString> message) {} private boolean doReply(Response<ByteString> msg) {return true;} private boolean doReply(Response<ByteString> msg) {return false;} package com.spotify.apollo.request; class TrackedOngoingRequestImpl extends ForwardingOngoingRequest { public void reply(Response<ByteString> message) { doReply(message); } private boolean doReply(Response<ByteString> msg) { final boolean removed = requestTracker.remove(this); if (removed) { super.reply(message);} return removed; } } @Test shouldNotReplyIfNotTracked() { tOR = new TrackedOngoingRequestImpl(o,t); tracker.remove(tOR); tOR.reply(Response.ok()); verifyNoMoreInteractions(ongoingRequest); }
  • 42. 42 package com.spotify.apollo.environment; public class ApolloConfig { public boolean enableMetaApi() { return optionalBoolean(apolloNode, "metaApi").orElse(true); }
  • 43. 43 public boolean enableMetaApi() {return true;} public boolean enableMetaApi() {return false;} package com.spotify.apollo.environment; public class ApolloConfig { public boolean enableMetaApi() { return optionalBoolean(apolloNode, "metaApi").orElse(true); } Covered by 8 test cases which all expect enableMetaApi()to return true
  • 44. 44 package com.spotify.apollo.environment; public class ApolloConfig { public boolean enableMetaApi() { return optionalBoolean(apolloNode, "metaApi").orElse(true); } public boolean enableMetaApi() {return true;} public boolean enableMetaApi() {return false;}
  • 45. Descartes •Current • Compatible with JUnit5 and latest Pitest • Integrates with Maven and Gradle •Future: Descartes and CI • Incremental Descartes on a commit • Descartes for pull requests 45
  • 46. Conclusion •Mutation analysis • Automatic generation of mutants • Evaluate the test suite •Bugs in test suites • Oracle • Input space coverage • Testability • Indirectly tested code 46
  • 47. Feedback welcome! • https://github.com/STAMP-project/pitest-descartes • https://github.com/hcoles/pitest • http://stamp-project.eu/ baudry@kth.se 47