Caroline Landry, Research engineer at Inria, presented the STAMP project at OW2con'19, June 12-13, 2019 in Paris. This talk presents two automatic techniques that can be used in the CI to reveal weaknesses and suggest possible improvements of your test cases. The first one is based on mutation testing, while the second one is based on test amplification.

1. 1
Improve your tests in the CI with STAMP
Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
Caroline Landry
Project Technical Manager
INRIA, France
19-Apr-2019

2. This work was partially supported by the EU Project
STAMP ICT-16-10 No.731529
•4 research institutions
•5 companies
•1 open source
consortium
•Automated Testing in
DevOps

3. DevOps
3

4. DevOps
4
Automation
& Continuous testing

5. DevOps
5
unit
perf.
fuzzing
UI
chaos
Logs
analysis
A/B testing
Config.
IDEs
CI
completion linter
coverage
mutation Crash
analysis
CD
Automation
& Continuous testing

6. 6

7. STAMP’s concept: amplification
 Test developers capture essential behaviors and
expectations
 But tests are uncomplete
 Test amplification: Increase the effect of test assets
 Automatic analysis and transformations
 Automatic assessment and selection
 In the continuous integration pipeline
7https://dictionary.cambridge.org/dictionary/english/amplify

8. DevOps – STAMP focus
8

9. DevOps – STAMP focus
9

10. DevOps – STAMP focus
10

11. DevOps – STAMP focus
11

12. Unit test amplification
12
Descartes

13. Descartes I mutate therefore I am
• A mutation engine for PIT
• Implement extreme mutation
• Compute code coverage & mutation score
• Identify weaknesses in your tests
• Find pseudo-tested methods

14. Pseudo-tested methods
•A method executed by the test suite
•No extreme mutation is detected
•Found in well tested projects

15. class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
15
Apache Commons Collections

16. A pseudo-tested method
16
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
}
Apache Commons Collections

17. class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
17
Pseudo-tested
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
}
Apache Commons Collections

18. class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
18
No exception is thrown
A fail is needed here
Pseudo-tested
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
}
Apache Commons Collections

19. Descartes results
19

20. Unit test amplification
20
Descartes

21. Unit test amplification
21
Descartes &
DSpot

22. DSpot
•Automatically enhances existing JUnit test suites
•Generate new assertions or new test cases
22

23. x
Dspot principle
23
Test criterion
(Mutation score)

24. 24
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
Example
Original
test case

25. 25
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
Example
Original
test case
Remove a method call

26. @Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
assertTrue(benjamin.isHungry());
assertFalse(benjamin.isHappy());
} 26
Example
Original
test case
Amplified
test case
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
Remove a method call
Instrument the test and
run it again

27. 27
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());
}
Original
test case
DSpot

28. 28
@Test
public void html_add33() throws Exception {
Attribute attr = new Attribute("key", "value &");
Assert.assertEquals("key="value &"", attr.html());
Assert.assertEquals("key="value &"", attr.toString());
Assert.assertEquals("key", attr.getKey());
Assert.assertEquals("value &", attr.getValue());
}
@Test
public void html() {
Attribute attr = new Attribute("key", "value &");
assertEquals("key="value &"", attr.html());
assertEquals(attr.html(), attr.toString());
}
Original
test case
Amplified
test case
New assertions
DSpot

29. 29
DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.

30. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
30

31. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
31
Novel analyses

32. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
32
Novel analyses

33. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
33
Novel analyses

34. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
34
Novel analyses

35. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
35
Novel analyses

36. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
36
Novel analyses

37. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
37
Novel analyses

38. More
•Open Source tools
• https://github.com/STAMP-project
•Public website
• https://www.stamp-project.eu
•Medium on Descartes
• https://medium.com/@almyre/short-circuiting-method-
executions-to-assess-test-quality-2d3fda45bc7f
•Publications
• https://www.stamp-project.eu/view/main/publications
38

39. Contacts
https://github.com/STAMP-project/
http://stamp-project.eu/
baudry@kth.se
caroline.landry@inria.fr
http://www.diverse-team.fr/
barais@irisa.fr
39

40. 40