Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

STAMP Descartes Presentation

Ad

1

Ad

Software Testing AMPlification
H2020 LEIT RIA - ICT-10-2016 – Software Technology
2016/12/01 – 2019/11/30
2

Ad

DevOps
3

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Próximo SlideShare
Mutation Testing at BzhJUG
Mutation Testing at BzhJUG
Cargando en…3
×

Eche un vistazo a continuación

1 de 57 Anuncio
1 de 57 Anuncio

STAMP Descartes Presentation

Descargar para leer sin conexión

Benoit Baudry, Professor at KTH University, offered this presentation of STAMP software testing tools Descartes and DSpot, to Orange DevOps teams, using a remote webinar on October 2, 2019.

Benoit Baudry, Professor at KTH University, offered this presentation of STAMP software testing tools Descartes and DSpot, to Orange DevOps teams, using a remote webinar on October 2, 2019.

Más Contenido Relacionado

Presentaciones para usted (19)

Similares a STAMP Descartes Presentation (20)

STAMP Descartes Presentation

  1. 1. 1
  2. 2. Software Testing AMPlification H2020 LEIT RIA - ICT-10-2016 – Software Technology 2016/12/01 – 2019/11/30 2
  3. 3. DevOps 3
  4. 4. DevOps – automatic development unit perf. fuzzing loggingdep. inj. UI CI pertur- bation fault recov. IDEs libraries container IDS VMs cluster config. 4
  5. 5. DevOps – continuous testing unit perf. fuzzing logging UI pertur- bation fault recov. IDS cluster config. 5
  6. 6. DevOps – STAMP focus unit logging config. 6
  7. 7. Human – bot collaboration Collaboration platform pull req. 7
  8. 8. Human – bot collaboration Collaboration platform pull req. code 8
  9. 9. Human – bot collaboration Collaboration platform pull req. code analyses 9
  10. 10. Human – bot collaboration Collaboration platform pull req. code analyses feedback 10
  11. 11. Human – bot collaboration pull req. code analyses feedback 11
  12. 12. Human – bot collaboration Collaboration platform pull req. code analyses feedback 12 STAMP tools
  13. 13. Human – bot collaboration Collaboration platform pull req. code analyses feedback STAMP tools 13
  14. 14. DevOps – STAMP context •Test automation •Human – bot collaboration 14
  15. 15. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something https://dictionary.cambridge.org/dictionary/english/amplify 15
  16. 16. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something  Test amplification: Increase the effect of test assets 16 https://dictionary.cambridge.org/dictionary/english/amplify
  17. 17. STAMP’s concept: amplification  Amplify (v.): to increase the size or effect of something  Test amplification: Increase the effect of test assets  Test assets: test cases, configuration files, production logs  Effect metrics: mutation score, feature interactions  Automatic amplification 17 https://dictionary.cambridge.org/dictionary/english/amplify
  18. 18. Unit test amplification •Premise: unit test cases exist • Can run automatically • Developpers capture essential behaviors and expectations • They are not sufficient •Intuition: unit test cases can be automatically amplified • Automatic analysis and transformations • Automatic assessment and selection • In the continuous integration pipeline 18
  19. 19. 19 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}
  20. 20. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 20 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}
  21. 21. Coverage 21 long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));}
  22. 22. 22 long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} Are these test cases good at detecting bugs?
  23. 23. 23 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Are these test cases good at detecting bugs? Let’s mutate our code to see.
  24. 24. Mutation analysis •Tests are good if they can detect bugs •Principle: generate bugs and test the tests • Conditions • Constants • Return • Delete method calls • Constructor call 24
  25. 25. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 25
  26. 26. long fact(int n) { if(n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 26 n != 0 return 1+1 < --!(i<=n) result/i result+1
  27. 27. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 27 n != 0 return 1+1 < --!(i<=n) result/i result+1
  28. 28. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 28 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } n != 0 return 1+1 < --!(i<=n) result/i result+1
  29. 29. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 29 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  30. 30. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Mutation reveals bugs in the test suite 30 @Test factorialWith5Test() { long obs = fact(5); assertTrue(5 < obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} Bugs in the test suite: - Weak oracle - Missing input n != 0 return 1+1 < --!(i<=n) result/i result+1
  31. 31. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} 31 @Test factorialWith5Test() { long obs = fact(5); assertEqual(120, obs); } @Test factorialWith5Test() { assertEqual(1, fact(0));} @Test factorialWith1Test() { assertEqual(1, fact(1));} n != 0 return 1+1 < --!(i<=n) result/i result+1
  32. 32. Descartes – extreme Java mutation 32
  33. 33. Descartes •Mutation operators: extreme mutation •Active, open source development •Pitest plugin •Compared to PIT • Less mutants • Different type of feedback • Same framework 33
  34. 34. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 34
  35. 35. long fact(int n) { if (n == 0)return 1; long result = 1; for(int i=2; i<=n; i++) result = result * i; return result;} Descartes - example 35 long fact(int n){ return 0;} long fact(int n){ return 1;} PIT : 7 mutants Descartes : 2 mutants
  36. 36. Assess test cases effectiveness 36Apache Commons Collections class SingletonListIterator implements Iterator<Node> { ... void add() { throw new UnsupportedOperationException(); } ... }
  37. 37. Assess test cases effectiveness 37Apache Commons Collections @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} class SingletonListIterator implements Iterator<Node> { void add() { throw new UnsupportedOperationException(); } }
  38. 38. Assess test cases effectiveness 38Apache Commons Collections @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} class SingletonListIterator implements Iterator<Node> { void add() { throw new UnsupportedOperationException(); } } ✔
  39. 39. Descartes 39Apache Commons Collections @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} class SingletonListIterator implements Iterator<Node> { void add() { } } Remove the whole body of method
  40. 40. Descartes 40Apache Commons Collections @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} class SingletonListIterator implements Iterator<Node> { void add() { } } ✔
  41. 41. Descartes 41Apache Commons Collections @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} class SingletonListIterator implements Iterator<Node> { void add() { } } ✔Pseudo-tested method
  42. 42. Descartes 42Apache Commons Collections @Test void testAdd() { SingletonListIterator it = ...; try { it.add(value); } catch(Exception ex) {} class SingletonListIterator implements Iterator<Node> { void add() { } } ✔Pseudo-tested method No exception is thrown A fail is needed here
  43. 43. Descartes 43
  44. 44. <plugins> <plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <version>1.4.0</version> <dependencies> <dependency> <groupId>eu.stamp-project</groupId> <artifactId>descartes</artifactId> <version>1.2</version> </dependency> </dependencies> <configuration> <verbose>true</verbose> <mutationEngine>descartes</mutationEngine> <mutators> <mutator>true</mutator> <mutator>”A string”</mutator> <mutator>null</mutator> <mutator>empty</mutator> </mutators> <outputFormats> <value>ISSUES</value> <value>METHODS</value> <value>JSON</value> </outputFormats> </configuration> </plugin> </plugins> PITest as a plugin Descartes to be used by PITest Regular PITest configuration Descartes as the mutation engine Values to use in extreme transforms. Supports literals for primitive types, String, null and empty for empty arrays. Reporting options. Custom reports ISSUES and METHODS for testing issues and method categorization. JSON, HTML, XML for regular PITest report.
  45. 45. 45
  46. 46. public void testIterationOrder() { for (int size = 1; size < 1000; size++) { List<Integer> other = new ArrayList<Integer>(size); for (int i = 0; i < size; i++) {other.add(i);} TreeList<Integer> l = new TreeList<Integer>(other); ListIterator<Integer> it = l.listIterator(); int i = 0; while (it.hasNext()) { Integer val = it.next(); assertEquals(i++, val.intValue()); } } } Original test case input (data and functions) oracle (expected properties) Benoit Baudry, KTH, ICES DevOps 46
  47. 47. Unit testing Benoit Baudry, KTH, ICES DevOps P state Input space Observation space 47
  48. 48. Unit test amplification Benoit Baudry, KTH, ICES DevOps P state Input space Observation space 48
  49. 49. 49 @Test public void html() { Attribute attr = new Attribute("key", "value &"); assertEquals("key="value &"", attr.html()); assertEquals(attr.html(), attr.toString()); }
  50. 50. 50 @Test public void html() { Attribute attr = new Attribute("key", "value &"); assertEquals("key="value &"", attr.html()); assertEquals(attr.html(), attr.toString()); } @Test public void html_add33() throws Exception { Attribute attr = new Attribute("key", "value &"); Assert.assertEquals("key="value &"", attr.html()); Assert.assertEquals("key="value &"", attr.toString()); Assert.assertEquals("key", attr.getKey()); Assert.assertEquals("value &", attr.getValue()); }
  51. 51. 51
  52. 52. 52
  53. 53. 53 DSpot Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.
  54. 54. Test amplification •Descartes: spots weak tests •Dspot: suggests improvements of existing tests •Available as Maven plugins •Method • Developer machine • In the CI •Future work • Integration in PR-based development • Develop work on advanced Maven dependency management 54
  55. 55. Open source tooling •https://github.com/STAMP-project/pitest-descartes •https://github.com/STAMP-project/dspot •Two Maven plugins 55
  56. 56. More information • An Approach and Benchmark to Detect Behavioral Changes of Commits in Continuous Integration • https://arxiv.org/pdf/1902.08482 • Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects • https://arxiv.org/pdf/1811.08330 • A Comprehensive Study of Pseudo-tested Methods • https://arxiv.org/pdf/1807.05030 • http://stamp-project.eu/ • https://github.com/KTH/devops-course/ • baudry@kth.se 56
  57. 57. •4 res. institutions •5 companies •1 open source consortium •516 p.m •To increase test automation in DevOps

×