Presentation for BARCO and the EFFECTS Project ---Abstract--- With the rise of agile development, software teams all over the world embrace faster release cycles as *the* way to incorporate customer feedback into product development processes. Yet, faster release cycles imply rethinking the traditional notion of software quality: agile teams must balance reliability (minimize known defects) against agility (maximize ease of change). This talk will explore the state-of-the-art in software test automation and the opportunities this may present for maintaining this balance. We will address questions like: Will our test suite detect critical defects early? If not, how can we improve our test suite? Where should we fix a defect?

1. Finding Bugs, Fixing Bugs, Preventing Bugs
Exploiting Automated Tests to Increase Reliability
March 2022 — © Prof. Serge Demeyer

3. in vitro
in vivo

4. Ericsson, Bombardier,
Saab, System Veri
fi
cation,
Empear, Verifyter, KTH,
MDH, RISE Comiq, E
fi
Code,
Ponsse, Siili,
Qentinel, Symbio,
Uni.Oulu, VTT
Axini, Testwerk,
TNO, Open Uni. AKKA, Expleo,
EKS, FFT,
Fraunhofer,
IFAK, OFFIS,
Parasoft
Alerion,
Prodevelop,,
Uni.Mandragon Kuveyt Bank,
Saha BT
The TESTOMAT project will allow software teams to increase the
development speed without sacri
fi
cing quality.
To achieve this goal, the project will advance the state-of-the-art in test
automation for software teams moving towards a more agile development
process.

6. Six decades into the computer revolution, four
decades since the invention of the microprocessor,
and two decades into the rise of the modern Internet,
all of the technology required to transform industries
through software
fi
nally works and can be widely
delivered at global scale.

9. Finding Bugs, Fixing Bugs, Preventing Bugs
Exploiting Automated Tests to Increase Reliability
March 2022 — © Prof. Serge Demeyer

10. © Serge Demeyer
Continuous Integration Pipeline
10
<<Breaking the Build>>
version
control
build
developer
tests
deploy
scenario
tests
deploy to
production
measure &
validate

11. Finding Bugs, Fixing Bugs, Preventing Bugs
Exploiting Automated Tests to Increase Reliability
March 2022 — © Prof. Serge Demeyer

12. c++
java

13. © Serge Demeyer
Test Suite
13
TEST(FindLastTests, emptyVector) {
EXPECT_EQ(-1, findLast({}, 3));
}
TEST(FindLastTests, doubleOccurrence) {
EXPECT_EQ(3, findLast({1, 2, 42, 42, 63}, 42));
}
TEST(FindLastTests, noOccurrence) {
EXPECT_EQ(-1, findLast({1, 2, 42, 42, 63}, 99));
int findLast(std::vector<int> x, int y) {
if (x.size() == 0)
return -1;
for (int i = x.size() - 1; i >= 0; i--)
if (x[i] == y)
return i;
return -1;
}

14. 100% line coverage
100% statement coverage
100% branch coverage
all tests passed

15. © Serge Demeyer
Inject Mutant (Killed)
15
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i < 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08 }
[ PASSED ] 2 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] FindLastTests.doubleOccurrence
Relational Operator Replacement (ROR)
“i >= 0” becomes “i < 0”

16. © Serge Demeyer
Inject Mutant (Survived - Live)
16
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i > 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08 }
[==========] 3 tests from 1 test suite ran. (0 ms total)
[ PASSED ] 3 tests.
Relational Operator Replacement (ROR)
“i >= 0” becomes “i > 0”

17. © Serge Demeyer
TEST(FindLastTests, occurrenceOnBoundary) {
EXPECT_EQ(0, findLast({1, 2, 42, 42, 63}, 1));
}
[==========] 4 tests from 1 test suite ran. (0 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] FindLastTests.occurrenceOnBoundary
Strengthening the Test Suite
17
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i > 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08 }

18. © Serge Demeyer
Industrial Case Study
18
• 83K lines of code
• Complicated structure
• Lots of legacy code
• Lots of black-box tests
Ali Parsai, Serge Demeyer; “Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting”.
Software Tools for Technology Transfer

19. Unit tests only !
Segmentation
Percentage
0
20
40
60
80
100
Mutation Coverage
Branch Coverage

20. CI
D
evelop
Bu
ild
Test
W
ay
too
slow
We witnessed 48 hours of mutation testing time on a
test suite comprising 272 unit tests and 5,258 lines of
test code for testing a project with 48,873 lines of
production code.
Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the
Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018

21. Master
1) Initial test Build
2) ∀ files to mutate:
queue file names
3a) Generate mutants
4a) Execute mutants
3b) Store mutants
4b) Store results
3c) Queue mutant
references
5) Process results

22. 0
1h
2h
3h
LittleDarwin
1 worker
2 workers
4 workers
8 workers
16 workers
0
12h
1d
1d 12h
2d
2d 12h
LittleDarwin
1 worker
2 workers
4 workers
8 workers
16 workers

23. https://github.com/joakim-brannstrom/dextool

24. © Serge Demeyer
DO-178C = Avionics Safety Standard
24
Mutation Testing =
Actionable !

25. Finding Bugs, Fixing Bugs, Preventing Bugs
Exploiting Automated Tests to Increase Reliability
March 2022 — © Prof. Serge Demeyer

26. © Serge Demeyer
4 Quadrants
26
Supporting
Team
Functional Tests
Examples
Story Tests
Prototypes
Simulation
Exploratory Testing
Scenarios
Usability Testing
Acceptance Testing
Alpha / Beta
Unit Tests
Integration Tests
Performance Testing
Load Testing
Security Testing
“ility” Testing
Technology Facing
Business Facing
Critique
Product
Automated
& Manual
Manual
Autom
ated
Tools

27. © Serge Demeyer
Flipping the V
27
70%
20%
10%
10%
20%
70%
Acceptance Tests
(GUI Tests)
System Tests
Integration Tests
Component Tests
Unit Tests
Manual Automated

28. © Serge Demeyer
Flipping the V?
28

29. © Serge Demeyer
Flakey Tests
29
version
control
build
developer
tests
deploy
scenario
tests
deploy to
production
measure &
validate
Non-determistic Result

30. © Serge Demeyer
Flakey Tests: Taxonomy
30
CAUSES
Avoidable
Async Wait Concurrency
I/O
Memory
Arithmetic
Randomness
Collections
Inevitable
Time
Test Order
External
Dependencies

31. © Serge Demeyer
Flakey Tests: Management
31
Zero Tolerance Manage

32. Finding Bugs, Fixing Bugs, Preventing Bugs
Exploiting Automated Tests to Increase Reliability
March 2022 — © Prof. Serge Demeyer

33. Advance the state-of-the-art in
autonomous (= zero-touch) testing
for DevOps software teams.
Arti
fi
cial
Intelligence
Inside

34. © Serge Demeyer
Testing
34
Program
Under
Test
Input
Expected output
Software Testing is the process of executing a program or system with
the intent of finding errors.
(Myers, Glenford J., The art of software testing. Wiley, 1979

35. © Serge Demeyer
Coverage
35
Program
Under
Test
Input Expected output
(mutation)
coverage

36. © Serge Demeyer
Test Ampli
fi
cation
36
Program
Under
Test
Input Expected output
(mutation)
coverage
+coverage
Extra Input
Extra Input
+Extra Input +Extra output
Arti
fi
cial
Intelligence
Inside

37. © Serge Demeyer
Example - testDeposit
37
1 def testDeposit ( self ) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.get_balance(), 10)
5 self.b.deposit(100)
6 self.b.deposit(100)
7 self.assertEqual(self.b.get_balance() , 210)
I
n
p
u
t
Expected
output

38. © Serge Demeyer
Example - testDeposit_ampli
fi
ed (1/2)
38
1 def testDeposit_amplified ( self ) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse (self .b. is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
…
Assertion
Am
plification

39. © Serge Demeyer
Example - testDeposit_ampli
fi
ed (2/2)
39
1 def testDeposit_amplified ( self ) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse (self .b. is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
9 with self.assertRaises(Exception):
10 self .b. deposit(−56313)
11 self.b.deposit(100)
12 self.b.set_owner(’Guido van Rossum’)
13 self.assertEqual(self.b.
14 get_transactions(), [10, 100])
…
Input Am
plification

40. 11 pull requests
8 merged
3 pending

41. © Serge Demeyer
Research Agenda
41
Program
Under
Test
(mutation)
coverage
+coverage
Arti
fi
cial
Intelligence
Inside
(Intention Revealing)
Test Names
Explainable
Test Cases
Visualisation
(Dashboards)
Work
fl
ow
(Github Actions)

42. Finding Bugs, Fixing Bugs, Preventing Bugs
Exploiting Automated Tests to Increase Reliability
March 2022 — © Prof. Serge Demeyer

43. Description text Mining
Stack Traces Link to source code
Product/Component
Speci
fi
c vocabulary
Suggestions?

44. Question Cases Precision Recall
Who should
fi
x this bug? Eclipse, Firefox, gcc
eclipse: 57%
fi
refox: 64%
gcc: 6%
—
How long will it take to
fi
x
this bug? (*)
JBoss
depends on the component
many similar reports: off by one hour
few similar reports: off by 7 hours
What is the severity of this
bug? (**)
Mozilla, Eclipse, Gnome
mozilla, eclipse:67% -
73%
gnome:
75%-82%
mozilla, eclipse:50% -
75%
gnome:
68%-84%
(*) In CSMR2012 Proceedings
Who should
fi
x this bug? Eclipse, Firefox, gcc
eclipse: 57%
fi
refox: 64%
gcc: 6%
—
Irrelevant for
Practitioners
(**) In CSMR2011; MSR 2010 Proceedings
Arti
fi
cial
Intelligence
Inside
Internal vs.
External
Bug
Reports

45. © Serge Demeyer
Story Points (Planning Poker)
45
1/2 1 2 3 5 8 13 20 40 100 ♾
Public Domain

46. Human
MMRE: 0.48
(*) Mean Magnitude
of Relative Error
Learning Curve

47. © Serge Demeyer
Arti
fi
cial
Intelligence
Inside
“in vivo” Validation
47
Explainable!

48. Finding Bugs, Fixing Bugs, Preventing Bugs
Exploiting Automated Tests to Increase Reliability
March 2022 — © Prof. Serge Demeyer

49. © Serge Demeyer
Q&A support
49

50. In CHI2016 Proceedings
Arti
fi
cial
Intelligence
Inside

51. © Serge Demeyer
Hype Cycle
51
Hype Cycle © Gartner
Visibility
Maturity
Technology
Trigger
Peak of
Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity

52. © Serge Demeyer
The Future ?
52
Personal
Opinion
Peak of
Inflated
Expectations
Hype Cycle © Gartner
Visibility
Maturity
Technology
Trigger
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
IBM
Microsoft
Google
FaceBook
…
… for an experiment nearby