Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Mutate and Test your Tests

Ad

Mutate and Test Your Tests
Benoit Baudry
KTH, Sweden
1
baudry@kth.se

Ad

Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
•...

Ad

Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
•...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Eche un vistazo a continuación

1 de 48 Anuncio
1 de 48 Anuncio

Más Contenido Relacionado

Mutate and Test your Tests

  1. 1. Mutate and Test Your Tests Benoit Baudry KTH, Sweden 1 baudry@kth.se
  2. 2. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs Benoit Baudry, KTH, EclipseCon'17 2
  3. 3. Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs Benoit Baudry, KTH, EclipseCon'17 3
  4. 4. 4 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } Benoit Baudry, KTH, EclipseCon'17
  5. 5. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 5 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } Benoit Baudry, KTH, EclipseCon'17
  6. 6. Coverage long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 6 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } Benoit Baudry, KTH, EclipseCon'17
  7. 7. Coverage 7 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Benoit Baudry, KTH, EclipseCon'17
  8. 8. 8 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs? Benoit Baudry, KTH, EclipseCon'17
  9. 9. 9 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs? Let’s mutate our code to see. Benoit Baudry, KTH, EclipseCon'17
  10. 10. Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests 10 Benoit Baudry, KTH, EclipseCon'17
  11. 11. Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests •Mutation operators •Mutant = Program with one mutation 11 Benoit Baudry, KTH, EclipseCon'17
  12. 12. Mutation analysis Inputs : P, TS, Ops Output : score, coverage M <- generateMutants (P, OPs) forAll (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(M) 12 Benoit Baudry, KTH, EclipseCon'17
  13. 13. PIT •Open source •Actively developed and maintained •State of the art •For Java •Integrates with major build systems 13 Benoit Baudry, KTH, EclipseCon'17
  14. 14. PIT mutation operators •Conditions •Constants •Return •Delete method calls •Constructor call 14 Benoit Baudry, KTH, EclipseCon'17
  15. 15. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 15 Benoit Baudry, KTH, EclipseCon'17
  16. 16. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 16 n != 0 return 1+1 < --!(i<=n) result/i result+1 Benoit Baudry, KTH, EclipseCon'17
  17. 17. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 17 n != 0 return 1+1 < --!(i<=n) result/i result+1 Benoit Baudry, KTH, EclipseCon'17
  18. 18. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 18 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } Benoit Baudry, KTH, EclipseCon'17 n != 0 return 1+1 < --!(i<=n) result/i result+1
  19. 19. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 19 @Test factorialWith5Test() { assertEqual(1, fact(0));} Benoit Baudry, KTH, EclipseCon'17 n != 0 return 1+1 < --!(i<=n) result/i result+1
  20. 20. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 20 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} Benoit Baudry, KTH, EclipseCon'17 n != 0 return 1+1 < --!(i<=n) result/i result+1
  21. 21. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Mutation reveals bugs in the test suite 21 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} Benoit Baudry, KTH, EclipseCon'17 Bugs in the test suite: - Weak oracle - Missing input n != 0 return 1+1 < --!(i<=n) result/i result+1
  22. 22. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 22 @Test factorialWith5Test() { long obs = fact(5); assertEqual(120, obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} @Test factorialWith1Test() { assertEqual(1, fact(1));} Benoit Baudry, KTH, EclipseCon'17 n != 0 return 1+1 < --!(i<=n) result/i result+1
  23. 23. PIT – reporting 23 Benoit Baudry, KTH, EclipseCon'17
  24. 24. PIT – reporting 24 Benoit Baudry, KTH, EclipseCon'17
  25. 25. Project #LOC #Mutants Time (h) Score (%) Amazon Web Services SDK 1703892 2141690 04:25:35 76.28 XWiki Rendering Engine 104727 112609 01:59:55 50.89 Apache Commons Math 180951 104786 03:22:18 83.81 JFreeChart 134353 89592 00:41:28 58.04 Apache PdfBox 137099 79763 06:20:25 58.89 Java Git 128874 78316 16:02:00 73.86 SCIFIO 55347 62768 03:12:11 45.92 Joda-Time 85911 31233 00:16:32 81.65 Apache Commons Lang 60733 30361 00:21:02 86.17 Apache Commons Collections 51632 20394 00:05:41 85.94 Urban Airship Client Library 40885 17345 00:11:31 82.26 SAT4J 26415 17067 11:11:04 68.58 ImageJ Common 23868 15592 00:29:09 54.77 jsoup 17839 14054 00:12:49 78.34 Jaxen XPath Engine 21204 12210 00:24:40 67.13 Apache Commons Codec 17267 9233 00:07:57 87.82 Apache Commons IO 24334 8809 00:12:48 84.73 Google Gson 20068 7353 00:05:34 81.76 AuthZForce PDP Core 16602 7296 01:23:50 88.18 Apache Commons CLI 7005 2560 00:01:26 88.71 JOpt Simple 9214 2271 00:01:36 93.52Benoit Baudry, KTH, EclipseCon'17 25
  26. 26. Descartes •Mutation operators: extreme mutation •Active development •Open source •Compared to PIT • Less mutants • Different type of feedback • Same framework 26 Benoit Baudry, KTH, EclipseCon'17
  27. 27. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 27 Benoit Baudry, KTH, EclipseCon'17
  28. 28. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 28 long fact(int n){ return 0;} long fact(int n){ return 1;} Benoit Baudry, KTH, EclipseCon'17 PIT : 7 mutants Descartes : 2 mutants
  29. 29. Scalability of extreme mutation Benoit Baudry, KTH, EclipseCon'17 29 public static boolean isValidXmlChar(int ch){ return (ch == 0x9) || (ch == 0xA) || (ch == 0xD) || (ch >= 0x20 && ch <= 0xD7FF) || (ch >= 0xE000 && ch <= 0xFFFD || (ch >= 0x10000 && ch <= 0x10FFFF); } PIT : 45 mutants Descartes : 2 mutants
  30. 30. Project Mutants PIT Mutants Descartes Time PIT Time Descartes Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30 XWiki Rendering Engine 112609 5534 01:59:55 00:10:50 Apache Commons Math 104786 7150 03:22:18 00:08:30 JFreeChart 89592 7210 00:41:28 00:05:26 Apache PdfBox 79763 7559 06:20:25 00:42:11 Java Git 78316 7152 16:02:00 00:56:07 SCIFIO 62768 3627 03:12:11 00:15:26 Joda-Time 31233 4525 00:16:32 00:04:13 Apache Commons Lang 30361 3872 00:21:02 00:02:18 Apache Commons Collections 20394 3558 00:05:41 00:01:48 Urban Airship Client Library 17345 3082 00:11:31 00:09:38 SAT4J 17067 2296 11:11:04 00:56:42 ImageJ Common 15592 1947 00:29:09 00:04:08 jsoup 14054 1566 00:12:49 00:02:45 Jaxen XPath Engine 12210 1252 00:24:40 00:01:34 Apache Commons Codec 9233 979 00:07:57 00:02:02 Apache Commons IO 8809 1164 00:12:48 00:02:18 Google Gson 7353 848 00:05:34 00:01:11 AuthZForce PDP Core 7296 626 01:23:50 00:08:45 Apache Commons CLI 2560 271 00:01:26 00:00:09 JOpt Simple 2271 412 00:01:36 00:00:25Benoit Baudry, KTH, EclipseCon'17 30
  31. 31. Coarser grain than PIT Benoit Baudry, KTH, EclipseCon'17 31 long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} long fact(int n){ return 0;} long fact(int n){ return 1;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}
  32. 32. Benoit Baudry, KTH, EclipseCon'17 32 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}
  33. 33. Benoit Baudry, KTH, EclipseCon'17 33 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  34. 34. Benoit Baudry, KTH, EclipseCon'17 34 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  35. 35. Wrong test Benoit Baudry, KTH, EclipseCon'17 35 bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  36. 36. Benoit Baudry, KTH, EclipseCon'17 36 protected final void prepareSocket(final SSLSocket socket) { }}
  37. 37. Benoit Baudry, KTH, EclipseCon'17 37 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] {"TLSv1.2", "TLSv1.1", "T }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  38. 38. Benoit Baudry, KTH, EclipseCon'17 38 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] {"TLSv1.2", "TLSv1.1", "T }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  39. 39. Missing oracle Benoit Baudry, KTH, EclipseCon'17 39 @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] {"TLSv1.2", "TLSv1.1", "T }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  40. 40. Benoit Baudry, KTH, EclipseCon'17 40 private boolean isLongOption(final String token){ if (...){return false;} //Covered 27 times ... if (...){return true;} //Not covered else if (...){return true;} //Covered once return false; //Covered 7 times }
  41. 41. Benoit Baudry, KTH, EclipseCon'17 41 private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  42. 42. Benoit Baudry, KTH, EclipseCon'17 42 ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  43. 43. Benoit Baudry, KTH, EclipseCon'17 43 ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  44. 44. Benoit Baudry, KTH, EclipseCon'17 44 ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;} Testability issue
  45. 45. 45 15 285 4 15 54 28 64 132 23 115 14 594 369 92 9 40 158 458 116 65 2592 69 2 14 28 19 27 55 10 17 12 97 85 31 2 14 11 61 20 69 197 274 1516 175 408 1178 610 1840 4245 459 344 585 2931 2237 2596 252 714 456 1726 425 2004 1618 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Tested Weak pseudo-tested Strong pseudo-tested Benoit Baudry, KTH, EclipseCon'17
  46. 46. Descartes – future •Incremental Descartes in the CI •Github hook •Support multi module Maven projects •Enhanced diagnosis •Integrate into a test augmentation pipeline Benoit Baudry, KTH, EclipseCon'17 46
  47. 47. Conclusion •Mutation analysis • Automatic generation of mutants • Evaluate the test suite •Bugs in test suites • Oracle • Input space coverage • Testability • Indirectly tested code 47 Benoit Baudry, KTH, EclipseCon'17
  48. 48. Feedback welcome! • https://github.com/STAMP-project/pitest-descartes • https://github.com/hcoles/pitest • http://stamp-project.eu/ baudry@kth.se 48 Benoit Baudry, KTH, EclipseCon'17

×