Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017

Ad

Mutate	and	Test	Your	Tests
Author: Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
1
baudry@kth.se car...

Ad

POSS	17	- 2 Caroline Landry, INRIABenoit Baudry, KTH
Test	Your	Tests
•What	do	you	expect	from	test	cases?
• Cover	requirem...

Ad

POSS	17	- 3 Caroline Landry, INRIABenoit Baudry, KTH
Test	Your	Tests
•What	do	you	expect	from	test	cases?
• Cover	requirem...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Próximo SlideShare
Mutate and Test your Tests
Mutate and Test your Tests
Cargando en…3
×

Eche un vistazo a continuación

1 de 51 Anuncio
1 de 51 Anuncio
Anuncio

Más Contenido Relacionado

Similares a Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017 (20)

Anuncio
Anuncio

Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017

  1. 1. Mutate and Test Your Tests Author: Benoit Baudry Project Coordinator and Scientific Leader KTH, Sweden 1 baudry@kth.se caroline.landry@inria.fr Speaker: Caroline Landry Project Technical Manager INRIA, France
  2. 2. POSS 17 - 2 Caroline Landry, INRIABenoit Baudry, KTH Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs
  3. 3. POSS 17 - 3 Caroline Landry, INRIABenoit Baudry, KTH Test Your Tests •What do you expect from test cases? • Cover requirements • Stress the application • Prevent regressions • Reveal bugs
  4. 4. POSS 17 - 4 Caroline Landry, INRIABenoit Baudry, KTH @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  5. 5. POSS 17 - 5 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  6. 6. POSS 17 - 6 Caroline Landry, INRIABenoit Baudry, KTH Coverage long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); }
  7. 7. POSS 17 - 7 Caroline Landry, INRIABenoit Baudry, KTH Coverage @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}
  8. 8. POSS 17 - 8 Caroline Landry, INRIABenoit Baudry, KTH @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs?
  9. 9. POSS 17 - 9 Caroline Landry, INRIABenoit Baudry, KTH @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs? Let’s mutate our code to see.
  10. 10. POSS 17 - 10 Caroline Landry, INRIABenoit Baudry, KTH Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests
  11. 11. POSS 17 - 11 Caroline Landry, INRIABenoit Baudry, KTH Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests •Mutation operators •Mutant = Program with one mutation
  12. 12. POSS 17 - 12 Caroline Landry, INRIABenoit Baudry, KTH Mutation analysis Inputs : P, TS, Ops Output : score, coverage M <- generateMutants (P, OPs) forAll (m in M) run (TS,m) if (one-test-fail) then killed <- m else alive <- m score = killed / size(M)
  13. 13. POSS 17 - 13 Caroline Landry, INRIABenoit Baudry, KTH PIT •Open source •Actively developed and maintained •State of the art •For Java •Integrates with major build systems
  14. 14. POSS 17 - 14 Caroline Landry, INRIABenoit Baudry, KTH PIT mutation operators •Conditions •Constants •Return •Delete method calls •Constructor calls
  15. 15. POSS 17 - 15 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;}
  16. 16. POSS 17 - 16 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0
  17. 17. POSS 17 - 17 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1
  18. 18. POSS 17 - 18 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 <
  19. 19. POSS 17 - 19 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 < --!(i<=n) result/i result+1
  20. 20. POSS 17 - 20 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} n != 0 return 1+1 < --!(i<=n) result/i result+1
  21. 21. POSS 17 - 21 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } n != 0 return 1+1 < --!(i<=n) result/i result+1
  22. 22. POSS 17 - 22 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith0Test() { assertEqual(1, fact(0));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  23. 23. POSS 17 - 23 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  24. 24. POSS 17 - 24 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Mutation reveals bugs in the test suite @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} Bugs in the test suite: - Weak oracle - Missing input n != 0 return 1+1 < --!(i<=n) result/i result+1
  25. 25. POSS 17 - 25 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertEqual(120, obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));} @Test factorialWith1Test() { assertEqual(1, fact(1));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  26. 26. POSS 17 - 26 Caroline Landry, INRIABenoit Baudry, KTH PIT – reporting
  27. 27. POSS 17 - 27 Caroline Landry, INRIABenoit Baudry, KTH PIT – reporting
  28. 28. POSS 17 - 28 Caroline Landry, INRIABenoit Baudry, KTH Project #LOC #Mutants Time (h) Score (%) Amazon Web Services SDK 1703892 2141690 04:25:35 76.28 XWiki Rendering Engine 104727 112609 01:59:55 50.89 Apache Commons Math 180951 104786 03:22:18 83.81 JFreeChart 134353 89592 00:41:28 58.04 Apache PdfBox 137099 79763 06:20:25 58.89 Java Git 128874 78316 16:02:00 73.86 SCIFIO 55347 62768 03:12:11 45.92 Joda-Time 85911 31233 00:16:32 81.65 Apache Commons Lang 60733 30361 00:21:02 86.17 Apache Commons Collections 51632 20394 00:05:41 85.94 Urban Airship Client Library 40885 17345 00:11:31 82.26 SAT4J 26415 17067 11:11:04 68.58 ImageJ Common 23868 15592 00:29:09 54.77 jsoup 17839 14054 00:12:49 78.34 Jaxen XPath Engine 21204 12210 00:24:40 67.13
  29. 29. POSS 17 - 29 Caroline Landry, INRIABenoit Baudry, KTH Descartes •Mutation operators: extreme mutation •Active development •Open source •Compared to PIT • Less mutants • Different type of feedback • Same framework
  30. 30. POSS 17 - 30 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example
  31. 31. POSS 17 - 31 Caroline Landry, INRIABenoit Baudry, KTH long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example long fact(int n){ return 0;} long fact(int n){ return 1;} PIT : 7 mutants Descartes : 2 mutants
  32. 32. POSS 17 - 32 Caroline Landry, INRIABenoit Baudry, KTH Scalability of extreme mutation public static boolean isValidXmlChar(int ch){ return (ch == 0x9) || (ch == 0xA) || (ch == 0xD) || (ch >= 0x20 && ch <= 0xD7FF) || (ch >= 0xE000 && ch <= 0xFFFD || (ch >= 0x10000 && ch <= 0x10FFFF); } PIT : 45 mutants Descartes : 2 mutants
  33. 33. POSS 17 - 33 Caroline Landry, INRIABenoit Baudry, KTH Project Mutants PIT Mutants Descartes Time PIT Time Descartes Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30 XWiki Rendering Engine 112609 5534 01:59:55 00:10:50 Apache Commons Math 104786 7150 03:22:18 00:08:30 JFreeChart 89592 7210 00:41:28 00:05:26 Apache PdfBox 79763 7559 06:20:25 00:42:11 Java Git 78316 7152 16:02:00 00:56:07 SCIFIO 62768 3627 03:12:11 00:15:26 Joda-Time 31233 4525 00:16:32 00:04:13 Apache Commons Lang 30361 3872 00:21:02 00:02:18 Apache Commons Collections 20394 3558 00:05:41 00:01:48 Urban Airship Client Library 17345 3082 00:11:31 00:09:38 SAT4J 17067 2296 11:11:04 00:56:42 ImageJ Common 15592 1947 00:29:09 00:04:08 jsoup 14054 1566 00:12:49 00:02:45
  34. 34. POSS 17 - 34 Caroline Landry, INRIABenoit Baudry, KTH Coarser grain than PIT long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} long fact(int n){ return 0;} long fact(int n){ return 1;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith0Test() { assertEqual(1, fact(0));}
  35. 35. POSS 17 - 35 Caroline Landry, INRIABenoit Baudry, KTH bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;}
  36. 36. POSS 17 - 36 Caroline Landry, INRIABenoit Baudry, KTH bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  37. 37. POSS 17 - 37 Caroline Landry, INRIABenoit Baudry, KTH bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  38. 38. POSS 17 - 38 Caroline Landry, INRIABenoit Baudry, KTH Wrong test bool equals(object other) { return other instanceof AClass &&((AClass) other).aField==aField; } bool equals(object other) { return true;} bool equals(object other) { return false;} test() { AClass a = new AClass(3); AClass b = new AClass(3); AClass c = new AClass(4); assertEquals(a, b); assertFalse(a == c); }
  39. 39. POSS 17 - 39 Caroline Landry, INRIABenoit Baudry, KTH protected final void prepareSocket(final SSLSocket socket) { }}
  40. 40. POSS 17 - 40 Caroline Landry, INRIABenoit Baudry, KTH @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] { "TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" })); }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  41. 41. POSS 17 - 41 Caroline Landry, INRIABenoit Baudry, KTH @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] { "TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" })); }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  42. 42. POSS 17 - 42 Caroline Landry, INRIABenoit Baudry, KTH Missing oracle @Test void typical() throws NoSuchAlgorithmException { SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null); f.prepareSocket(new TestSSLSocket() { @Override public void setEnabledProtocols(String[] protocols) { assertTrue(Arrays.equals(protocols, new String[] { "TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" })); }});}}} protected final void prepareSocket(final SSLSocket socket) { }}
  43. 43. POSS 17 - 43 Caroline Landry, INRIABenoit Baudry, KTH private boolean isLongOption(final String token){ if (...){return false;} //Covered 27 times ... if (...){return true;} //Not covered else if (...){return true;} //Covered once return false; //Covered 7 times }
  44. 44. POSS 17 - 44 Caroline Landry, INRIABenoit Baudry, KTH private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  45. 45. POSS 17 - 45 Caroline Landry, INRIABenoit Baudry, KTH ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  46. 46. POSS 17 - 46 Caroline Landry, INRIABenoit Baudry, KTH ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;}
  47. 47. POSS 17 - 47 Caroline Landry, INRIABenoit Baudry, KTH ... if (condition && isLongOption(value)){ action1(value);} else{ action2(value);} ... private boolean isLongOption(final String token){ return false;} private boolean isLongOption(final String token){ return true;} Testability issue
  48. 48. POSS 17 - 48 Caroline Landry, INRIABenoit Baudry, KTH 15 285 4 15 54 28 64 132 23 115 14 594 369 92 9 40 158 458 116 65 2592 69 2 14 28 19 27 55 10 17 12 97 85 31 2 14 11 61 20 69 197 274 1516 175 408 1178 610 1840 4245 459 344 585 2931 2237 2596 252 714 456 1726 425 2004 1618 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Tested Weak pseudo-tested Strong pseudo-tested
  49. 49. POSS 17 - 49 Caroline Landry, INRIABenoit Baudry, KTH Descartes – future •Incremental Descartes in the CI •Github hook •Support multi module Maven projects •Enhanced diagnosis •Integrate into a test augmentation pipeline
  50. 50. POSS 17 - 50 Caroline Landry, INRIABenoit Baudry, KTH Conclusion •Mutation analysis • Automatic generation of mutants • Evaluate the test suite •Bugs in test suites • Oracle • Input space coverage • Testability • Indirectly tested code
  51. 51. Feedback welcome! • https://github.com/STAMP-project/pitest-descartes • https://github.com/hcoles/pitest • http://stamp-project.eu/ baudry@kth.se caroline.landry@inria.fr

×