Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Mutation Testing at BzhJUG

Ad

Mutation	testing	in	Java	and	GitHub
Oscar Vera Perez, Caroline Landry, Martin Monperrus,
Benoit Baudry
Breizh JUG, Rennes....

Ad

Test	Your	Tests
•What	do	you	expect	from	test	cases?	
• Cover	requirements	
• Stress	the	application	
• Prevent	regression...

Ad

Test	Your	Tests
•What	do	you	expect	from	test	cases?	
• Cover	requirements	
• Stress	the	application	
• Prevent	regression...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Próximo SlideShare
Stamp breizhcamp 2019
Stamp breizhcamp 2019
Cargando en…3
×

Eche un vistazo a continuación

1 de 56 Anuncio
1 de 56 Anuncio

Más Contenido Relacionado

Similares a Mutation Testing at BzhJUG (20)

Mutation Testing at BzhJUG

  1. 1. Mutation testing in Java and GitHub Oscar Vera Perez, Caroline Landry, Martin Monperrus, Benoit Baudry Breizh JUG, Rennes. June 2018. 1
  2. 2. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 2
  3. 3. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs 3
  4. 4. 4 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  5. 5. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 5 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  6. 6. Coverage long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 6 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  7. 7. Coverage 7 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}
  8. 8. 8 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs?
  9. 9. 9 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs? Let’s mutate our code to see.
  10. 10. Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests 10
  11. 11. Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests •Mutation operators = types of bugs •Mutant = Program with one seeded bug 11
  12. 12. Mutation analysis Input : P, TS, Ops Output : score, coverage M <- generateMutants (P, OPs) forAll (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(M) 12
  13. 13. PITest mutation operators •Conditions •Constants •Return •Delete method calls •Constructor call 13
  14. 14. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 14
  15. 15. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 15 n != 0 return 1+1 < --!(i<=n) result/i result+1
  16. 16. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 16 n != 0 return 1+1 < --!(i<=n) result/i result+1
  17. 17. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 17 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } n != 0 return 1+1 < --!(i<=n) result/i result+1
  18. 18. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 18 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  19. 19. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Mutation reveals bugs in the test suite 19 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} Bugs in the test suite: - Weak oracle - Missing input n != 0 return 1+1 < --!(i<=n) result/i result+1
  20. 20. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 20 @Test factorialWith5Test() { long obs = fact(5); assertEqual(120, obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} @Test factorialWith1Test() { assertEqual(1, fact(1));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  21. 21. Project #Mutants Time (h) Score (%) Amazon Web Services SDK 2141690 04:25:35 76.28 XWiki Rendering Engine 112609 01:59:55 50.89 Apache Commons Math 104786 03:22:18 83.81 JFreeChart 89592 00:41:28 58.04 Apache PdfBox 79763 06:20:25 58.89 Java Git 78316 16:02:00 73.86 SCIFIO 62768 03:12:11 45.92 Joda-Time 31233 00:16:32 81.65 Apache Commons Lang 30361 00:21:02 86.17 Apache Commons Collections 20394 00:05:41 85.94 Urban Airship Client Library 17345 00:11:31 82.26 SAT4J 17067 11:11:04 68.58 ImageJ Common 15592 00:29:09 54.77 jsoup 14054 00:12:49 78.34 Jaxen XPath Engine 12210 00:24:40 67.13 Apache Commons Codec 9233 00:07:57 87.82 Apache Commons IO 8809 00:12:48 84.73 Google Gson 7353 00:05:34 81.76 AuthZForce PDP Core 7296 01:23:50 88.18 21
  22. 22. Descartes •Mutation operators: extreme mutation •Active, open source development •Pitest plugin •Compared to PIT • Less mutants • Different type of feedback • Same framework 22
  23. 23. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 23
  24. 24. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 24 long fact(int n){ return 0;} long fact(int n){ return 1;} PIT : 7 mutants Descartes : 2 mutants
  25. 25. Scalability of extreme mutation 25 public static boolean isValidXmlChar(int ch){ return (ch == 0x9) || (ch == 0xA) || (ch == 0xD) || (ch >= 0x20 && ch <= 0xD7FF) || (ch >= 0xE000 && ch <= 0xFFFD || (ch >= 0x10000 && ch <= 0x10FFFF); } PIT : 45 mutants Descartes : 2 mutants
  26. 26. 26 Number of mutants 1 10 100 1000 10000 100000 1000000 10000000 Amazon Web Services SDK JFreeChart SCIFIO Apache Commons CollecAonsImageJ Common Apache Commons Codec AuthZForce PDP Core Pit Descartes
  27. 27. 27 0 % 25 % 50 % 75 % 00 % authzforce commons-codec commons-lang jaxen joda-ime sat4j-core spoon Descartes Gregor
  28. 28. Coarser grain than PIT 28 long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} long fact(int n){ return 0;} long fact(int n){ return 1;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}
  29. 29. 29 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); ... f.prepareSocket(new TestSSLSocket() { ... @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, expected)); }); } protected final void prepareSocket(final SSLSocket socket) { }}
  30. 30. 30 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); ... f.prepareSocket(new TestSSLSocket() { ... @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, expected)); }); } protected final void prepareSocket(final SSLSocket socket) { }}
  31. 31. 31 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); ... f.prepareSocket(new TestSSLSocket() { ... @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, expected)); }); } protected final void prepareSocket(final SSLSocket socket) { }} Missing oracle
  32. 32. 32 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}
  33. 33. 33 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  34. 34. 34 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  35. 35. Bug in test 35 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  36. 36. 36 0 % 25 % 50 % 75 % 100 % authzforce commons-codec commons-lang imagej-common jgit jsoup scifio 78 % 94 % 76 %77 % 73 % 93 %96 %95 % 83 %81 % 96 % 72 % 93 %96 %95 %93 %93 %93 % 97 % 81 % 94 % 9 % 3 % 4 %3 % 2 % 2 % 1 %1 % 3 % 3 % 2 % 4 % 2 % 1 %1 %3 %2 %3 %1 % 4 % 1 % 12 % 3 % 21 %20 % 25 % 5 %3 %3 % 14 %16 % 2 % 24 % 5 %3 %3 %4 %4 %3 %2 % 15 % 5 % Strong pseudo-tested Weak pseudo-tested Tested
  37. 37. Descartes in Maven 37
  38. 38. <plugins> <plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <version>1.4.0</version> <dependencies> <dependency> <groupId>eu.stamp-project</groupId> <artifactId>descartes</artifactId> <version>1.2</version> </dependency> </dependencies> <configuration> <verbose>true</verbose> <mutationEngine>descartes</mutationEngine> <mutators> <mutator>true</mutator> <mutator>”A string”</mutator> <mutator>null</mutator> <mutator>empty</mutator> </mutators> <outputFormats> <value>ISSUES</value> <value>METHODS</value> <value>JSON</value> </outputFormats> </configuration>
 </plugin> </plugins> PITest as a plugin Descartes to be used by PITest Regular PITest configuration Descartes as the mutation engine Values to use in extreme transforms. Supports literals for primitive types, String, null and empty for empty arrays. Reporting options. Custom reports ISSUES and METHODS for testing issues and method categorization. JSON, HTML, XML for regular PITest report.
  39. 39. Descartes in Jenkins 39
  40. 40. 40
  41. 41. 41
  42. 42. 42
  43. 43. Descartes in the GitHub check UI 43
  44. 44. Pull request based development 44 update build feedback ✗✔ review Merge if OK
  45. 45. GitHub checks API 45
  46. 46. Pull request based development 46 update build feedback review Merge if OK
  47. 47. 47
  48. 48. 48
  49. 49. 49
  50. 50. 50
  51. 51. Descartes under the hood 51
  52. 52. Compiled files Compiled test files Source files Test source files Dependencies PIT Descartes Operators to use Number of execution threads …
  53. 53. . . . . . . . . . . . . . . . Compiled files Compiled test files Source files Test source files Dependencies Configuration PIT Descartes class A { … public void voidMethod() {} public int intMethod() {} public String strMethod() {} … } void 3 “S” null Report Execution Unit Operators to use Number of execution threads … . . . 53
  54. 54. Descartes •Current • Compatible with JUnit5 and latest Pitest • Integrates with Maven and Gradle •Future: Descartes and CI • Incremental Descartes on a commit • Descartes for pull requests 54
  55. 55. Conclusion •Mutation analysis • Automatic generation of mutants • Evaluate the test suite •Bugs in test suites • Oracle • Input space coverage • Testability • Indirectly tested code 55
  56. 56. Feedback welcome! • https://github.com/STAMP-project/pitest-descartes • https://github.com/hcoles/pitest • http://stamp-project.eu/ baudry@kth.se 56

×