TAROT 2013 9th International Summer School on Training And Research On Testing, Volterra, Italy, 9-13 July, 2013
These slides summarize Myra Cohen's presentation about "Sampling, Re-use and Incremental Testing in Software Product Lines"
Why Teams call analytics are critical to your entire business
TAROT2013 Testing School - Myra Cohen presentation
1. Sampling, Re-use and Incremental Testing in
Software Product Lines
Myra B. Cohen
myra@cse.unl.edu
University of Nebraska-Lincoln
2. Agenda
SPL System Testing: the combinatorics
SPL Integration Testing
Summary
Continuous Test Suite Augmentation
Introduction to Software Product Lines
4. Software Product Lines
• A software product line (SPL) is a set of programs that share a
significant common (managed) set of features satisfying a
market segment or mission developed from a common set of
core assets in a prescribed way
Product 1
Commonality
Variability
..
.Product 2 Product n
Core Assets
5. Feature Oriented Domain
Analysis
• SEI FODA Project in Late 1980s
• Identified features (variability) as the key to software
product lines
• Identified the need for artifact-independent modeling of the
features in an SPL
•
• Introduced the feature diagram which led to what we know
today a the feature model
6. Feature Models
• Representation of the features and
dependencies/constraints in an SPL
• Defines what is (and is not) allowable in
the SPL
• Can be written as a logic equation and can
use SAT solvers to reason about model
• Several different notations for feature
models
7. Software Product Line
Engineering
• Consists of two parts:
– Domain engineering
• Determining what the re-usable components/
architecture is for the SPL
• Defines the commonality and variability of the
SPL
– Application engineering
• Deriving products from the platform created in
the domain engineering step
• Makes use of re-use and exploits variability/
commonality
8. Domain Testing
• Validate and verify reusable
components
• No running application exists at this
time - only components
• This differs from system/application
testing where the entire system is tested
as a unit
10. Examples of Variability
• Electric bulb can be lit or unlit
• Software applications support different
languages
• A triple band mobile phone supports
three different network standards
• Can load Choco vs. MiniSat to solve
constraints
13. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
14. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
15. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
16. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
17. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
18. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
19. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
20. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Mobile Phone SPL
Display
Type
VP
21. Testing a Product Line
• SPL engineering decreases time to market
• With shorter life cycles, the percentage of
time spent in testing has increased – the new
bottleneck
• Much of the current work on testing SPLs
focuses on testing individual instances and
reuse of specific test cases
22. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V V
Camera
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Validating at the Unit Level
Display
Type
VP
23. V V V
16MC 8MC BW
Validating at the Unit Level
Display
Type
VP
V
16MC
24. V V V
16MC 8MC BW
Validating at the Unit Level
Display
Type
VP
V
16MC
V
8MC
25. V V V
16MC 8MC BW
Validating at the Unit Level
Display
Type
VP
V
16MC
V
8MC
V
BW
Focus is on reuse of artifacts during testing
Views testing from the feature level rather
than the system level
26. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
System Level Testing
Display
Type
VP
27. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
System Level Testing
Viewer
Type
VP
Test Case: Open email in HTML format
28. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
System Level Testing
Display
Type
VP
Test Case: Open email in HTML format
29. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Feature Interactions
Viewer
Type
VP
Failure: caused by interaction of data passed
between 16MC display and textual viewer
31. Real Interaction Failure
• NASA Detected failed communications on
two independent channels
• Determined that the rover was continually
re-booting – risked running out of energy
and/or overheating
• Fault caused by assumptions made
between two different features: (NASA component
expected third party file system to deallocate memory)
– when one feature was removed the reboots
stopped
32. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Possible Interactions
Display
Type
VP
33. Fault Map of Configurations
V4
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14
C0
C10
C20
C30
C40
C50
C60
F1 F2 F3 F4 F5
C0
C10
C20
C30
C40
C50
C60
Faults
C
o
n
f
I
g
u
r
a
t
I
o
n
s
34. V V V
16MC 8MC BW
V V
GV TV
V V
2MP 1MP
Viewer
Type
VP
Camera
Type
VP
Phone
VP
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp requires_v_vp
Testing the Interaction Space
Viewer
Type
VP
Number of possible interactions grows
exponentially with number of variation points
35. Some Other Approaches to
SPL Testing
• Extended use cases for variability [Bertolino et al., SPFE 2003]
• Integration testing a SPL based on UML activity
diagrams [Reis et al., FASE 2007]
• Generate test cases incrementally for a SPL
[Uzuncaova et al., ISSRE 2008]
• Applying test cases for a subset of products only
[Kim et al., ASE 2010]
• Applying monitoring properties for a subset of
products only [Kim et al., RV 2010]
36. Some SPL Resources
• http://splot-research.org/
• http://www.isa.us.es/fama/
• http://sir.unl.edu
37. Agenda
SPL System Testing: the combinatorics
SPL Integration Testing
Summary
Continuous Test Suite Augmentation
Introduction to Software Product Lines
38. A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels Yes Yes
8 Million Colors Text 1 Megapixel No No
Black and White None None
39. A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels Yes Yes
8 Million Colors Text 1 Megapixel No No
Black and White None None
3
40. A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels Yes Yes
8 Million Colors Text 1 Megapixel No No
Black and White None None
3 x 3
41. A Combinatorial Problem
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels Yes Yes
8 Million Colors Text 1 Megapixel No No
Black and White None None
• 33x22=108 feasible SPL instances
3 x 3 x 3 x 2 x 2
42. A Combinatorial Problem
• 33x22=108 feasible SPL instances
• 10 features each with 5 values = 510 or
9,765,625 possible instances
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels Yes Yes
8 Million Colors Text 1 Megapixel No No
Black and White None None
43. A Combinatorial Problem
• 33x22=108 feasible SPL instances
• 10 features each with 5 values = 510 or
9,765,625 possible instances
• 4 hours to run test suite = approximately 4,459 years
to test
GCC optimizer: configuration space ~1061
Display Email Viewer Camera Video
Camera
Video
Ringtones
16 Million Colors Graphical 2 Megapixels Yes Yes
8 Million Colors Text 1 Megapixel No No
Black and White None None
44. Combinatorial Interaction Testing
(CIT)
• Test all pairs or t-way combinations of
feature-values, where t is a defined
strength of testing
• Uses a mathematical structure called a
covering array
45. Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
2-way CIT
46. Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
2-way CIT
47. Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
2-way CIT
48. Combinatorial Interaction Testing
(CIT)
• Based on statistical design of experiments (DOE)
– Manufacturing
– Drug test interactions
– Chemical interactions
• For software testing
– Mandl – compiler testing
– Brownlie, Prowse, Phadke – OATS system
– D. Cohen, Dalal, Fredman, Patton, Parelius –
AETG
– Williams, Probert – network node interfaces
– Yilmaz, Cohen, Porter- ACE/TAO
49. Empirical Results on CIT
for System Inputs
• CIT has been well studied on testing input
parameters for programs
• [Brownlie, Prowse, Phadke 92], [Burr,Young
98],[Burroughs, Jain, Erickson 94] [Cohen,
Dalal, Fredman, Patton 97], [Dalal, Jain,
Karunanithi, Leaton, Lott 98], [Dunietz, Erlich,
Szablak, Mallows, Iannino 97], [Mandl 85]
$ sort -m -c input.txt
50. CIT for Configurable
Systems
• Some studies on configurable
systems
• [Kuhn, Wallace,Gallo 04]
– Medical devices, web browsers, data mgt, http server
• [Yilmaz, Cohen, Porter 04,06]
– Ace/TAO Corba middleware
• [Qu, Cohen, Rothermel 08]
51. CIT vs. Default
Configuration of vim in SIRFaultDetection(%)
020406080100
v2 v3 v4 v5 v6 v7
Version default configuration
52. How do we Generate the
Samples?
• Greedy approaches
– AETG, IPOG
– Search-based (simulated annealing, GAs)
• Use of constraint solvers
55. Meta-heuristic Algorithms
• Provide mechanisms to escape local
optima
• Sometimes the algorithm can accept a
worse choice in the hope we will find a
better route to a good solution
59. • Meant to simulate the coolng of metal
• Physical annealing is a process of slowly cooling
a metal, in baths of decreasing temperatures.
• At each stage the metal molecules are allowed
to come to an equilibrium. Then the bath is
lowered to the next temperature.
• If the initial temperature is not high enough or if
the cooling is done too quickly the metal will be
brittle.
Simulated Annealing
60. • The basic strategy is the same as a hill climb,
but the meta-heuristic creates a gentle decrease
in the probability for making a bad move.
• At the start of our hill climb we often make a lot
of bad moves.
• By the time we are close to an optimal solution
we are almost in a pure hill climb (the
probability drops to almost zero)
Simulated Annealing (cont.)
61. • Start with a solution S
• Select a new solution S’ that is a neighbor of S
• If S’-S ≤ 0 then S’ is accepted
• Otherwise
– Accept S’ with a probability
Where T = temperature and B is the Boltzmann
constant (this is a distribution which we won’t
actually use for simulating the process)
Simulated Annealing
€
e
s−s'
kBT
63. Lower bound L
Binary search for “best” solution
Upper bound U
(L+U)/2
Finding “N”
64. The Real System
Constraints on Valid Configurations:
__________________________________________________________________________________________________
(1) Graphical email viewer requires color display
(2) 2 Megapixel camera requires a color display
(3) Graphical email viewer not supported with 2 Megapixel camera
(4) 8 Million color display does not support a 2 Megapixel camera
(5) Video camera requires a camera and a color display
(6) Video ringtones cannot occur with No video camera
(7) The combination of 16 Million colors, Text and
2 Megapixel camera will not be supported
65. Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
Constraints
66. Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
Constraints
“Video Ringtones cannot occur
without video camera
67. Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
Constraints
“Video Ringtones cannot occur
without video camera
68. Display Email Viewer Camera Video
Camera
Video
Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
Constraints:
(1) Graphical email viewer requires color display
(2) 2 Megapixel Camera requires color display
(3) Graphical email viewer not supported with the 2 Megapixel
camera
Decrease Sample Size
69. Display Email Viewer Camera Video
Camera
Video
Ringtones
1 16 Million Colors None 2 Megapixels No No
2 16 Million Colors Graphical 1 Megapixel No No
3 8 Million Colors None 1 Megapixel Yes Yes
4 16 Million Colors Graphical None Yes Yes
5 16 Million Colors Text 1 Megapixel Yes No
6 Black and White None 1 Megapixel Yes Yes
7 8 Million Colors Graphical 2 Megapixels Yes Yes
8 Black and White Text 2 Megapixels No Yes
9 8 Million Colors Text None No No
10 Black and White None None No No
Constraints:
_____________________________________________________________________
(1) Graphical email viewer requires color display
Increase Sample Size
70. Display Email Viewer Camera Video
Camera
Video
Ringtones
1 16 Million Colors None 2 Megapixels No No
2 16 Million Colors Graphical 1 Megapixel No No
3 8 Million Colors None 1 Megapixel Yes Yes
4 16 Million Colors Graphical None Yes Yes
5 16 Million Colors Text 1 Megapixel Yes No
6 Black and White None 1 Megapixel Yes Yes
7 8 Million Colors Graphical 2 Megapixels Yes Yes
8 Black and White Text 2 Megapixels No Yes
9 8 Million Colors Text None No No
10 Black and White None None No No
Constraints:
_____________________________________________________________________
(1) Graphical email viewer requires color display
Increase Sample Size
71. Display Email Viewer Camera Video
Camera
Video
Ringtones
1 16 Million Colors None 2 Megapixels No No
2 16 Million Colors Graphical 1 Megapixel No No
3 8 Million Colors None 1 Megapixel Yes Yes
4 16 Million Colors Graphical None Yes Yes
5 16 Million Colors Text 1 Megapixel Yes No
6 Black and White None 1 Megapixel Yes Yes
7 8 Million Colors Graphical 2 Megapixels Yes Yes
8 Black and White Text 2 Megapixels No Yes
9 8 Million Colors Text None No No
10 Black and White None None No No
Constraints:
_____________________________________________________________________
(1) Graphical email viewer requires color display
Increase Sample Size
72. Display Email Viewer Camera Video
Camera
Video
Ringtones
1 16 Million Colors None 2 Megapixels No No
2 16 Million Colors Graphical 1 Megapixel No No
3 8 Million Colors None 1 Megapixel Yes Yes
4 16 Million Colors Graphical None Yes Yes
5 16 Million Colors Text 1 Megapixel Yes No
6 Black and White None 1 Megapixel Yes Yes
7 8 Million Colors Graphical 2 Megapixels Yes Yes
8 Black and White Text 2 Megapixels No Yes
9 8 Million Colors Text None No No
10 Black and White None None No No
Constraints:
_____________________________________________________________________
(1) Graphical email viewer requires color display
Increase Sample Size
74. Greedy Worked Well
• Extended both greedy and meta-
heuristic search algorithms to find
constrained CIT samples
• Used off the shelf SAT solver
• Mine information learned during
Boolean propagation to improve
performance of greedy algorithm
M.B. Cohen, M.B. Dwyer and J. Shi, Constructing interaction test suites for
highly-configurable systems in the presence of constraints: a greedy approach,
IEEE Transactions on Software Engineering , 34(5), 2008, pp. 633-650.
75. Limitations
• Greedy algorithms tend to:
• Finish quickly
• Handle constraints well
• Meta-heuristic algorithms tend to:
• Find smaller samples
• Handle constraints poorly
• Software product lines may have long test times per
instance and many dependencies:
• Want a meta-heuristic search algorithm that builds small
samples quickly
76. Reformulated Simulated
Annealing CIT Algorithm
• Modified:
– Neighborhood of search
• Narrows as search progresses
– Global strategy to find minimal size of array
– A set of minor modifications
• B.J. Garvin, M.B. Cohen, and M.B. Dwyer, Evaluating Improvements to a
Meta-Heuristic Search for Constrained Interaction Testing, Empirical Software
Engineering (EMSE), 16(1), 2011, pp.61-102.
• CASA tool: http://cse.unl.edu/~citportal/tools/casa/
77. Major Modifications
• Original implementation uses two
phases:
(1) Finding minimal size of sample
(2) Finding a valid sample within that size
• Modifications:
• New search space (neighborhood)
• Improved method for finding size (global)
78. Reduces infeasible search paths (dotted lines)
Original Search Space may lead to
many infeasible solutions
New Search Space provides a more direct
path to feasible solutions
Modified Search Space
(CASA)1
1. Available at: http://www.cse.unl.edu/citportal/tools/casa/
79. Reduces infeasible search paths (dotted lines)
Original Search Space may lead to
many infeasible solutions
New Search Space provides a more direct
path to feasible solutions
Modified Search Space
(CASA)
80. Reduces infeasible search paths (dotted lines)
Original Search Space may lead to
many infeasible solutions
New Search Space provides a more direct
path to feasible solutions
Modified Search Space
(CASA)
82. Case Studies (t=2)
Average Size Average Time Break
Even Point
Greedy Old SA New SA Greedy Old SA New SA Greedy vs.
New SA
SPINs 27.0 20.2 19.4 0.2 414.9 8.6 1.1
SPINv 42.5 33.2 36.8 11.3 4010.7 102.1 15.9
GCC 24.7 19.8 21.1 204.0 36,801.4 1902.0 471.7
Apac 42.6 32.6 32.3 76.4 23,589.1 109.1 3.2
Bugz 21.8 16.0 16.2 1.9 58.0 9.1 1.3
83. Case Studies (t=2)
Average Size Average Time Break
Even Point
Greedy Old SA New SA Greedy Old SA New SA Greedy vs.
New SA
SPINs 27.0 20.2 19.4 0.2 414.9 8.6 1.1
SPINv 42.5 33.2 36.8 11.3 4010.7 102.1 15.9
GCC 24.7 19.8 21.1 204.0 36,801.4 1902.0 471.7
Apac 42.6 32.6 32.3 76.4 23,589.1 109.1 3.2
Bugz 21.8 16.0 16.2 1.9 58.0 9.1 1.3
84. Case Studies (t=2)
Average Size Average Time Break
Even Point
Greedy Old SA New SA Greedy Old SA New SA Greedy vs.
New SA
SPINs 27.0 20.2 19.4 0.2 414.9 8.6 1.1
SPINv 42.5 33.2 36.8 11.3 4010.7 102.1 15.9
GCC 24.7 19.8 21.1 204.0 36,801.4 1902.0 471.7
Apac 42.6 32.6 32.3 76.4 23,589.1 109.1 3.2
Bugz 21.8 16.0 16.2 1.9 58.0 9.1 1.3
85. Case Studies (t=2)
Average Size Average Time Break
Even Point
Greedy Old SA New SA Greedy Old SA New SA Greedy vs.
New SA
SPINs 27.0 20.2 19.4 0.2 414.9 8.6 1.1
SPINv 42.5 33.2 36.8 11.3 4010.7 102.1 15.9
GCC 24.7 19.8 21.1 204.0 36,801.4 1902.0 471.7
Apac 42.6 32.6 32.3 76.4 23,589.1 109.1 3.2
Bugz 21.8 16.0 16.2 1.9 58.0 9.1 1.3
86. Case Studies (t=2)
Average Size Average Time Break
Even Point
Greedy Old SA New SA Greedy Old SA New SA Greedy vs.
New SA
SPINs 27.0 20.2 19.4 0.2 414.9 8.6 1.1
SPINv 42.5 33.2 36.8 11.3 4010.7 102.1 15.9
GCC 24.7 19.8 21.1 204.0 36,801.4 1902.0 471.7
Apac 42.6 32.6 32.3 76.4 23,589.1 109.1 3.2
Bugz 21.8 16.0 16.2 1.9 58.0 9.1 1.3
87. Case Studies (t=2)
Average Size Average Time Break
Even Point
Greedy Old SA New SA Greedy Old SA New SA Greedy vs.
New SA
SPINs 27.0 20.2 19.4 0.2 414.9 8.6 1.1
SPINv 42.5 33.2 36.8 11.3 4010.7 102.1 15.9
GCC 24.7 19.8 21.1 204.0 36,801.4 1902.0 471.7
Apac 42.6 32.6 32.3 76.4 23,589.1 109.1 3.2
Bugz 21.8 16.0 16.2 1.9 58.0 9.1 1.3
88. Going Beyond CIT
• CIT is a system testing technique:
– Need to instantiate complete products
– Many interactions are low order
• CIT is black box:
– Tests may or may not actually trigger
interactions between features
– No consideration of data/control flow
directions
89. Directed vs. Undirected
Interactions
Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
90. Directed vs. Undirected
Interactions
Display Email Viewer Camera Video V. Ringtones
1 16 Million Colors None 1 Megapixel Yes Yes
2 8 Million Colors Text None No No
3 16 Million Colors Text 2 Megapixels No Yes
4 Black and White None None No Yes
5 8 Million Colors None 2 Megapixels Yes No
6 16 Million Colors Graphical None Yes No
7 Black and White Text 1 Megapixel Yes No
8 8 Million Colors Graphical 1 Megapixel No Yes
9 Black and White Graphical 2 Megapixels Yes Yes
2 ways to test this interaction:
8 million colors à Text
Text à 8 million colors
94. Agenda
SPL System Testing: the combinatorics
SPL Integration Testing
Summary
Continuous Test Suite Augmentation
Introduction to Software Product Lines
95. Our Idea
• Test at the integration level:
• We can test partial products
• Incrementally increase strength of interactions
• Re-use results as we test
• Reduce the interaction space:
• Identify only feasible interactions
• Integrate features using the analysis of feasible
interactions
• Generate tests to exercise these interactions
(future)
96. Approach
s() {!
…!
v();!
if (c) {!
…!
}!
w();!
…!
}!
f1(){!
…!
}!
f2(){!
…!
}!
f3() !
f4() !
J. Shi, Myra B. Cohen and Matthew B. Dwyer, Integration Testing of Software
Product Lines Using Compositional Symbolic Execution, International Conference on
Fundamental Approaches to Software Engineering (FASE), March 2012, pp. 270-284.
97. Approach
s() {!
…!
v();!
if (c) {!
…!
}!
w();!
…!
}!
f1(){!
…!
}!
f2(){!
…!
}!
f3() !
f4() !
CFG
PDG
Code: Program Dependency
and Control Flow Graph
98. Approach
s() {!
…!
v();!
if (c) {!
…!
}!
w();!
…!
}!
f1(){!
…!
}!
f2(){!
…!
}!
f3() !
f4() !
Interaction trees
CFG
PDG
Feature Model: Defines possible
interaction space
111. Testing Method
• Use bounded symbolic execution
• Obtain summaries from individual
features
• Use interaction trees to guide
composition of all t-feature summaries
112. Symbolic Summaries
• We have three outcomes:
1. Complete execution of a path (passing
behavior)
2. Exception (failing behavior)
– may or may not be feasible to reach in full system
3. Depth bound is reach (unknown behavior)
In this work we only accumulate type 1.
113. Results of (Composed)
Summaries
• Constraints can be collected for each region to
perform test generation (in progress)
sum(f1,f2,f4) = {(f1.x0,f4.ret=f1.x+1), …)
complete
exception
Time out
complete
exception
Time out
Feature 1 Feature 2
117. Case Study
We performed a case study on 2 software
product lines to answer these questions:
• RQ1: What is the reduction from our dependency
analysis on the number of interactions that should be
tested in an SPL?
• RQ2: What is the difference in time between using
our compositional symbolic technique versus a
traditional directed technique?
121. Method
RQ1:
• Compared number of (directed/undirected)
interactions
RQ2:
• Compared directed SE vs. incremental
compositional SE
122. Method
• Tools:
– Adapted the IFA in Soot
– Direct SE – use Java PathFinder
– Compositional approach – built a composer
• Depth:
– 20*t
– Explicit for Direct SE
– Implicit for compositional
135. RQ1: Summary
• We see a reduction of over 50% in
directed and over 75% in undirected
interactions when we use the interaction
trees
136. Subject t Direct
SE
Comp
SE
SCARI 1 6.75 6.75
2 14.48 9.63
3 17.67 10.06
4 36.09 10.93
5 35.87 11.70
GPL 1 41.77 41.77
2 67.25 56.28
3 184.76 82.00
4 727.34 216.63
5 3887.23 965.92
Compositional vs.
Directed(seconds)
137. Subject t Direct
SE
Comp
SE
SCARI 1 6.75 6.75
2 14.48 9.63
3 17.67 10.06
4 36.09 10.93
5 35.87 11.70
GPL 1 41.77 41.77
2 67.25 56.28
3 184.76 82.00
4 727.34 216.63
5 3887.23 965.92
Compositional vs.
Directed(seconds)
138. Subject t Direct
SE
Comp
SE
SCARI 1 6.75 6.75
2 14.48 9.63
3 17.67 10.06
4 36.09 10.93
5 35.87 11.70
GPL 1 41.77 41.77
2 67.25 56.28
3 184.76 82.00
4 727.34 216.63
5 3887.23 965.92
Compositional vs.
Directed(seconds)
139. Subject t Direct
SE
Comp
SE
SCARI 1 6.75 6.75
2 14.48 9.63
3 17.67 10.06
4 36.09 10.93
5 35.87 11.70
GPL 1 41.77 41.77
2 67.25 56.28
3 184.76 82.00
4 727.34 216.63
5 3887.23 965.92
Compositional vs.
Directed(seconds)
140. Subject t Direct
SE
Comp
SE
SCARI 1 6.75 6.75
2 14.48 9.63
3 17.67 10.06
4 36.09 10.93
5 35.87 11.70
GPL 1 41.77 41.77
2 67.25 56.28
3 184.76 82.00
4 727.34 216.63
5 3887.23 965.92
Compositional vs.
Directed(seconds)
Same time
141. Subject t Direct
SE
Our
Tech
SCARI 1 6.75 6.75
2 14.48 9.63
3 17.67 10.06
4 36.09 10.93
5 35.87 11.70
GPL 1 41.77 41.77
2 67.25 56.28
3 184.76 82.00
4 727.34 216.63
5 3887.23 965.92
1/3 of the time
1/4 of the time
Compositional vs.
Directed(seconds)
144. RQ2: Summary
• As we increase the interaction strength
we see a greater reduction in time when
using compositional symbolic execution
• We reduce the time by a factor of 4 in
the best case
• We believe this will allow for greater
scalability and testing of higher
strengths
145. Summary
• We developed an incremental
compositional symbolic execution
technique for integration testing of SPLs
• Leverages dependency analysis to
reduce interactions by more than 50%
• Compositional symbolic execution
reduces time by as much as 75%
146. Future Directions
• Use summaries for test generation to
execute specific interaction
• Characterize areas of behavior to drive
further test generation
• Extend to other types of SPLs
147. Agenda
SPL System Testing: the combinatorics
SPL Integration Testing
Summary
Continuous Test Suite Augmentation
Introduction to Software Product Lines
148. COntinuous TEst Suite
Augmentation
• CONTESA
Z. Xu, M. B. Cohen, W. Motycka, G. Rothermel, Continuous Test Suite Augmentation
in Software Product Lines, International Software Product Line Conference (SPLC),
August, 2013, (to appear).
149. Approaches to Testing/
Analyzing SPLS1
• Feature based
– Focusing on individual features
• Product based
– Focusing on products
• Family based
– Testing from family perspective
1. T. Thum, S. Apel, C. Kastner, M. Kuhlemann, I. Schaefer, and G. Saake. Analysis
strategies for software product lines. Technical report, Faukulat fur Informatik,
Otto-von-Guericke-Univeritat, April 2012.
150. CONTESA
• Hybrid family/product based approach
• Aims to cover all code in family by
selecting products to test
• Leverages ideas/techniques from
regression testing
157. Test Cases Change Coverage
Tests
We need to add tests to the test
suite
158. Code-based View
E
int foo (int x, int y)
P1: if(x0)
S1:return
x;
S3:return
x;
P3: if(x3)
X
S2:return
x+2;
S4:return
x+y;
P2: if(xy)
T
T
T
F
F
F
Test Suite:
t1=(x = 2, y = 2) t2=(x = 4, y = 4)
t3=(x = 1, y = 0) t4=(x = 4, y = 3)
t5=(x = −1, y = 0)
159. Code-based View
E
int foo (int x, int y)
P1: if(x0)
S1:return
x;
S3:return
x;
P3: if(x3)
X
S2:return
x+2;
S4:return
x+y;
P2: if(xy)
T
T
T
F
F
F
Test Suite:
t1=(x = 2, y = 2) t2=(x = 4, y = 4)
t3=(x = 1, y = 0) t4=(x = 4, y = 3)
t5=(x = −1, y = 0)
160. Code-based View
E
int foo (int x, int y)
P1: if(x0)
S1:return
x;
S3:return
x;
P3: if(x3)
X
S2:return
x+2;
S4:return
x+y;
P2: if(xy)
T
T
T
F
F
F
Test Suite:
t1=(x = 2, y = 2) t2=(x = 4, y = 4)
t3=(x = 1, y = 0) t4=(x = 4, y = 3)
t5=(x = −1, y = 0)
161. Code-based View
E
int foo (int x, int y)
P1: if(x0)
S1:return
x;
S3:return
x;
P3: if(x3)
X
S2:return
x+2;
S4:return
x+y;
P2: if(xy)
T
T
T
F
F
F
Test Suite:
t1=(x = 2, y = 2) t2=(x = 4, y = 4)
t3=(x = 1, y = 0) t4=(x = 4, y = 3)
t5=(x = −1, y = 0)
162. Code-based View
E
int foo (int x, int y)
P1: if(x0)
S1:return
x;
S3:return
x;
P3: if(x3)
X
S2:return
x+2;
S4:return
x+y;
P2: if(xy)
T
T
T
F
F
F
Test Suite:
t1=(x = 2, y = 2) t2=(x = 4, y = 4)
t3=(x = 1, y = 0) t4=(x = 4, y = 3)
t5=(x = −1, y = 0)
164. Implication
E
int foo (int x, int y)
P1: if(x0)
S1:return
x;
S3:return
x;
P3: if(x3)
X
S2:return
x+2;
S4:return
x+y;
P2: if(xy)
T
T
T
F
F
F
E
int foo’ (int x, int y)
P1: if(x0)
S1:return
x;
S3:return
x;
P3: if(x3)
X
S2:return
x+2;
S4:return
x+y;
P2:if(x=y)
T
T
T
F
F
F
167. 1. Identify affected elements (portions of
the program′ or its specification for
which new tests are needed)
2. Create or guide the creation of tests
that exercise these elements
Test Suite Augmentation
168. Test Suite Augmentation
for SPLs
V1 V2 V3 V4 V5 V6
Non-SPL system evolves
Products of an SPL
A
B C
F
D
E
V V V16MC 8MC BW V VGV TV V V
2MP 1MP
Viewer
Type
V
P
Camera
TypeV
P
Phone
V
P
V
Display
V
Email
Viewer
V
Camera
V
Video
Camera
V
Video
Ringtones
requires_v_vp
requires_v_vp
requires_v_vp
Viewer
Type
V
P
170. CONTESA
• Test a single product
• Use regression impact analysis
(DejaVu) to identify common vs.
variable code between current/next
product
• Generate tests to cover variable code in
the new product (use a GA)
171. Two Orders of Testing
Both select product with largest number of
uncovered branches (variable code)
• Static:
– Determine order at start
• Dynamic:
– Re-calculate after each product tested
172. Case Study
• RQ1: Is continuous test suite augmentation more
effective and efficient than generating test cases
independently for each product?
• RQ2: Does the order used in continuous test suite
augmentation matter in terms of effectiveness and
efficiency?
177. Summary
• Continuous test suite augmentation
uses ideas from regression testing to
drive SPL test generation
• It is a hybrid product and family based
approach
• Improves both effectiveness and
efficiency over an individual product
based approach
178. Agenda
SPL System Testing: the combinatorics
SPL Integration Testing
Summary
Continuous Test Suite Augmentation
Introduction to Software Product Lines
179. Summary of Lecture
• We have seen how combinatorics
impact SPL testing
• We have seen some sampling and re-
use techniques
– System and integration level
• We have seen an augmentation
technique aimed at the entire code-
base
180. Students and Collaborators
on this work
Isis Cabral
Matthew Dwyer
Brady Garvin
Wayne Motycka
Gregg Rothermel
Xiao Qu
Jiangfan Shi
Zhihong Xu
181. Sampling, Re-use and Incremental Testing in
Software Product Lines
Myra B. Cohen
myra@cse.unl.edu
University of Nebraska-Lincoln
This work is supported in part by the National Science Foundation through awards CCF-1161767,
CCF-0747009 and by the Air Force Office of Scientific Research through award FA9550-10-1-0406.