DevOps strongly encourages the development of test suites and test scripts that can run automatically at every build in order to detect regressions as soon as possible. However, the manual development of strong test suites is extremely time consuming. In this talk I will introduce test amplification, as an original solution to address this pain point. This approach consists in leveraging existing test cases and scripts in order to produce new ones through automatic transformation processes. I will present initial results about unit test amplification, from the STAMP project.

1. Software Testing Amplification
Benoit Baudry, INRIA

2. DevOps
● Software development practices
– To reduce time between releases
– To favor feedback from production into development
● Emphasizes automation
– Testing, build, deployment, configuration,
monitoring

3. Test automation
● Test execution
– Unit, UI, performance
● Quality metrics
– Static metrics, test coverage
● Improve quality
– Reduce regressions, detect bugs early

4. Test automation
● Test development
– Hard, time consuming
– Unit tests have limited coverage and some are flaky
● Configuration testing
– Little support for automation
– Very limited sampling among all configurations

5. Test amplification
Automatically transform
existing test assets in
order to detect regression
bugs before production
and drive down the cost
of software testing.

6. Example
6
public void testSearch() {
final ArrayStack<E> stack = makeObject();
stack.push(((E)("First Item")));
stack.push(((E)("Second Item")));
assertEquals("Top item is 'Second Item'", 1, stack.search("Second Item"));
assertEquals("Next Item is 'First Item'", 2, stack.search("First Item"));
assertEquals("Cannot find 'Missing Item'", -1, stack.search("Missing Item"));}

7. Example
7
public void testSearch() {
final ArrayStack<E> stack = makeObject();
stack.push(((E)("First Item")));
stack.push(((E)("Second Item")));
assertEquals("Top item is 'Second Item'", 1, stack.search("Second Item"));
assertEquals("Next Item is 'First Item'", 2, stack.search("First Item"));
assertEquals("Cannot find 'Missing Item'", -1, stack.search("Missing Item"));}
public void testSearch_cf5() {
final ArrayStack<E> stack = makeObject();
stack.push(((E)("First Item")));
stack.push(((E)("Second Item")));
E o_5 = stack.peek();
assertFalse(((java.lang.String)o_5).isEmpty());
E o_6 = stack.pop();
assertFalse(((java.lang.String)o_6).isEmpty());
Object o_7_0 = stack.search("Second Item");
Object o_9_0 = stack.search("First Item");
assertEquals("Cannot find 'Missing Item'", -1, stack.search("Missing Item"));
assertEquals(((java.lang.String)o_5).length(), 11);
assertEquals(((java.lang.String)o_6).length(), 11);}
new calls and asserts
extracted from asserts
new asserts

8. Focus on regressions
● Amplified tests
– Embedd regression oriented
oracles
– Detect behavioral differences
between two versions of a program

9. 9
STAMP in DevOps
● Focus on regression bugs in the continuous integration phase
● Reduce configuration bugs before application deployment
● Identify more operation bugs in edge cases with semantic
logging

10. STAMP objectives
● Reduce time for manual writing of test
cases, while increasing coverage
● Test the application in multiple
deployment configurations
● Analyze program behavior to re-inject
production test cases in continuous
testing

11. 11
DSpot: JUnit Test Suites
Amplification
Amplified Test
Suite
Original Test Suite
for Program P
Update in the Program
Under Test

12. 12
P
TS
P'
TS+
search-based
input space
ampli cafi tion
observation
space
ampli cafi tion
runandlook
for differences
TS++
DSpot: JUnit Test Suites
Amplification
github.com/STAMP-project/dspot

13. 13

14. 14
Descartes Engine for PIT
● Mutation Testing
– Inject faults into a program to determine if the test cases can
detect them.
● PIT: mutation testing system for Java
– Many fine-grained fault models
– But does not scale to large code bases
● Descartes
– STAMP’s adaption of PIT with extreme mutation operators
– Empty the body of methods github.com/STAMP-project/pitest-descartes

15. Example
15
public void setValue(int x) {
if(x<10){
a:=x;
}
}
public void setValue(int x) {
}
●
Original method
●
Mutant

16. 16
STAMP Use Cases
ProActive Workflows and
Scheduling
ATOS FIWARE Smart City
Ecosystem
TelluCloud e-health
XWiki hybrid Open
Source business/project
OW2 Software
Quality Platform

17. 17
● Project Long Name: Software Testing AMPlification
– A consortium of nine European members
● The STAMP project has received funding from the
EU H2020 research and innovation programme
– Start: December 1, 2016
– End: November 31, 2019
– EU program: Horizon 2020 ICT-10-2016-RIA
STAMP at a Glance

18. 18
STAMP Useful Links
Web Site
www.stamp-project.eu
Twitter
@stamp_project
SlideShare
www.slideshare.net/stamp-project
Web Site
www.stamp-project.eu
Forge
gitlab.ow2.org/stamp
github.com/STAMP-project