1. Effec%ve
Test
Execu%on
for
SPLs
Sabrina
Souto
(Supervisor:
Marcelo
d’Amorim)
Federal
University
of
Pernambuco
-­‐
UFPE

2. General
Assump%on
• We
assume
that
tests
are
not
designed
for
one
specific
product
– A
test
can
run
against
several
products!
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

3. General
Problem
• Which
products
to
select
for
running
a
par%cular
SPL
test
against?
Soundness
Efficiency
– One
can
miss
fault-­‐revealing
products
in
this
set
– One
can
add
irrelevant
products
in
this
set
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

4. Previous
selec%on
strategies
• Black-­‐box
tes6ng
(e.g.,
CIT,
sampling,
exhaus%ve…)
[Cohen
et.
al.,
ROSETEA
2006];
[Yilmaz
et.
al.,
TSE
2006]
;
[Perrouin
et.
al.,
ICST
2010];
[Cabral
et.
al.,
SPLC
2010]
…
• White-­‐box
tes6ng
[Reisner
et.
al.,
ICSE
2010],
[Kim
et.
al.,
ISSRE
2012];
[Kim
et.
al.,
AOSD
2011]
…
• Gray-­‐box
tes6ng
[Garvin
et.
al.,
ISSRE
2011];
[Shi
et.
al.,
FASE
2012];
[Song
et.
al.,
ICSE
2012]
…
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

5. Hypothesis
Few
valid
configura%ons
are
reachable
from
individual
tests
Substan%ated
from
results
in
literature:
[Reisner
et.
al.,
ICSE
2010]
and
[Kim
et.
al.,
AOSD
2011]
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

6. Ques%ons
(revisited
in
the
end)
• Is
efficiency
important
in
this
context?
• Is
combinatorial
explosion
(scalability)
rela%vely
less
important
for
tes%ng?
• Is
soundness
important
for
tes%ng?
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

7. Ques%ons
(revisited
in
the
end)
• Is
efficiency
important
in
this
context?
• Is
combinatorial
explosion
(scalability)
rela%vely
less
important
for
tes%ng?
• Is
soundness
important
for
tes%ng?
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

8. Ques%ons
(revisited
in
the
end)
• Is
efficiency
important
in
this
context?
• Is
combinatorial
explosion
(scalability)
rela%vely
less
important
for
tes%ng?
• Is
soundness
important
for
tes%ng?
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

9. Ques%ons
(revisited
in
the
end)
• Is
efficiency
important
in
this
context?
• Is
combinatorial
explosion
(scalability)
rela%vely
less
important
for
tes%ng?
• Is
soundness
important
for
tes%ng?
Our
answer
is
YES
to
all
ques%ons!
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

10. Symbolic
Execu%on
of
Features
–
SEF
SEF
Test
Program
schemata
Feature
Model
Configura6ons
Symbolic
Execu%on
has
been
proposed
in
the
70’s
Reisner
et
al.
proposed
SEF
in
2010
(ICSE)
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

11. Program
Schemata
#IFDEF FEATURE_1
x = 10;
#ELIF
x = 20;
#ENDIF
static boolean feature1;
if(feature1){x = 10;} else { x = 20;}
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

12. Symbolic
Execu%on
of
Features
–
SEF
SEF
Test
Program
schemata
Feature
Model
Configura6ons
Symbolic
Execu%on
has
been
proposed
in
the
70’s
Reisner
et
al.
proposed
SEF
in
2010
(ICSE)
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

13. Symbolic
Execu%on
of
Features
–
SEF
[<o1, c1 >, ...]
feature
model
Driver
Test
SAT
solver
test schemata
program
Our
goal
is
to
apply
SEF
to
reach
sound
and
efficient
tes%ng
for
SPLs
Symbolic
Execu%on
has
been
proposed
in
the
70’s
Reisner
et
al.
proposed
SEF
in
2010
(ICSE)
Insight
for
efficient
implementa%on
is
to
par%%on
state
in
two:
one
concrete
and
one
symbolic.
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

14. Example
Valid
paths:
[2
3
8
6
7
5]
[2
3
8
-­‐6
7
5]
Invalid
paths:
[-­‐2]
[2
3
8
-­‐6
-­‐7]
[2
-­‐3
8]
[2
3
8
6
-­‐7]
[2
-­‐3
-­‐8]
[2
3
8
-­‐6
7
-­‐5]
[2
3
-­‐8]
[2
3
8
6
7
-­‐5]
Sudoku
Feature
Model
Test1
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br
Base(2)
Generator(8)
States(3)
Generator(8)
Color(6)
Solver(7)
Solver(7)
Undo(8)
Undo(8)
X
X
X
X
X
X
X
X
✓ ✓
-­‐2
2
-­‐3
3
8
-­‐8
8
-­‐8
-­‐6
6
-­‐7
-­‐7
7
7
5
5
-­‐5
-­‐5

15. Example
Sudoku
Feature
Model
Test1
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br
Base(2)
Generator(8)
States(3)
Generator(8)
Color(6)
Solver(7)
Solver(7)
Undo(8)
Undo(8)
X
X
X
X
X
X
X
X
✓ ✓
-­‐2
2
-­‐3
3
8
-­‐8
8
-­‐8
-­‐6
6
-­‐7
-­‐7
7
7
5
5
-­‐5
-­‐5
Valid
paths:
[2
3
8
6
7
5]
[2
3
8
-­‐6
7
5]
Invalid
paths:
[-­‐2]
[2
3
8
-­‐6
-­‐7]
[2
-­‐3
8]
[2
3
8
6
-­‐7]
[2
-­‐3
-­‐8]
[2
3
8
-­‐6
7
-­‐5]
[2
3
-­‐8]
[2
3
8
6
7
-­‐5]

16. Example
Valid
paths:
[2
3
8
6
7
5]
[2
3
8
-­‐6
7
5]
Invalid
paths:
[-­‐2]
[2
3
8
-­‐6
-­‐7]
[2
-­‐3
8]
[2
3
8
6
-­‐7]
[2
-­‐3
-­‐8]
[2
3
8
-­‐6
7
-­‐5]
[2
3
-­‐8]
[2
3
8
6
7
-­‐5]
Sudoku
Feature
Model
Test1
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br
Feature
Model
Base(2)
Generator(8)
States(3)
Generator(8)
Color(6)
Solver(7)
Solver(7)
Undo(8)
Undo(8)
X
X
X
X
X
X
X
X
✓ ✓
-­‐2
2
-­‐3
3
8
-­‐8
8
-­‐8
-­‐6
6
-­‐7
-­‐7
7
7
5
5
-­‐5
-­‐5

17. Example
Valid
paths:
[2
3
8
6
7
5]
[2
3
8
-­‐6
7
5]
Invalid
paths:
[-­‐2]
[2
3
8
-­‐6
-­‐7]
[2
-­‐3
8]
[2
3
8
6
-­‐7]
[2
-­‐3
-­‐8]
[2
3
8
-­‐6
7
-­‐5]
[2
3
-­‐8]
[2
3
8
6
7
-­‐5]
Sudoku
Feature
Model
Test1
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br
Feature
Model
Test
Base(2)
Generator(8)
States(3)
Generator(8)
Color(6)
Solver(7)
Solver(7)
Undo(8)
Undo(8)
X
X
X
X
X
X
X
X
✓ ✓
-­‐2
2
-­‐3
3
8
-­‐8
8
-­‐8
-­‐6
6
-­‐7
-­‐7
7
7
5
5
-­‐5
-­‐5

18. public class SudokuGenerator{ //…!
public Board generate() {!
if(GENERATOR)){!
Board board = new Board(); !
fillBoard(board); !
! makeSolvable(board, 50) ! !
! return board;!
}!
}!
//…!
private void fillBoard(Board board){!
if(GENERATOR)){ !
BoardManager bm = new BoardManager(); !
! //…!
! bm.undo(); !!
//…!
}!
}!
//…!
}!
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br
public class Test{ //…!
BoardManager bm;!
SudokuGenerator sGen;!
public static void test1(){!
if(BASE){!
this.bm = new BoardManager();!
if(GENERATOR){!
this.sGen = new SudokuGenerator();!
} !
}!
} //…!
}!
public class BoardManager{ //…!
public BoardManager(){!
if(BASE){// …!
if(STATES){/*…*/} !
!} !
}!
//…!
public void undo(){!
if(UNDO){/*…*/}!
}!
//…!
}!
public class Board { //…!
public Board () {!
if(BASE){ //…!
this.board[i] = new Field();!
} !
}//…!
}!
public class Field{ //…!
public Field() {!
if(BASE){ //…!
if(COLOR){!
! !//…!
! } !
! if(SOLVER){/*…*/} !
} !
}//…!
}!
1
2
3
4
5
6

19. Example
Valid
paths:
[2
3
8
6
7
5]
[2
3
8
-­‐6
7
5]
Invalid
paths:
[-­‐2]
[2
3
8
-­‐6
-­‐7]
[2
-­‐3
8]
[2
3
8
6
-­‐7]
[2
-­‐3
-­‐8]
[2
3
8
-­‐6
7
-­‐5]
[2
3
-­‐8]
[2
3
8
6
7
-­‐5]
Sudoku
Feature
Model
Test1
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br
Feature
Model
Test
Base(2)
Generator(8)
States(3)
Generator(8)
Color(6)
Solver(7)
Solver(7)
Undo(8)
Undo(8)
X
X
X
X
X
X
X
X
✓ ✓
-­‐2
2
-­‐3
3
8
-­‐8
8
-­‐8
-­‐6
6
-­‐7
-­‐7
7
7
5
5
-­‐5
-­‐5

20. Par%%on
of
configura%on
space
…
P1
P2
P3
P4
P5
P1
P2
P3
...Pm
P1
P2
P1
P2
P3
…Pn
P4
P1
P2
P3
...Pm
2n
There
is
no
dis%nc%on
amongst
the
configura%ons
from
the
same
par%%on
with
respect
to
the
states
they
elaborate
during
the
test
execu%on.
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

21. Research
Ques%ons
• RQ1:
How
SEF
compares
to
an
ideal
sampler?
• RQ2:
What
is
the
impact
of
further
constraining
the
model
for
test
execu%on?
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

22. Subjects
of
Analysis
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

23. RQ1
How
SEF
compares
to
an
ideal
sampler?
101.1.1
101.2.1
101.3.1
101.4.1
101.5.1
101.6.1
101.7.1
101.8.1
101.9.1
01.10.1
01.11.1
01.12.1
01.13.1
bdb.5.1
jto.1.1
jto.1.2
jto.2.1
jto.3.1
jto.4.1
npd.1.1
npd.2.1
npd.3.1
prv.2.1
prv.3.1
prv.4.1
sdk.1.1
sdk.2.1
sdk.3.1
sdk.4.1
xst.1.1
xst.1.2
xst.1.3
xst.1.4
xst.1.5
xst.1.6
xst.1.7
xst.1.8
xst.1.9
zip.1.1
zip.2.1
time(inseconds)
0.00.20.40.6
running feasible paths
running infeasible paths
solving constraints <
1
second

24. gpl.1.1
gpl.1.2
gpl.1.3
gpl.1.4
gpl.1.5
gpl.1.6
gpl.1.7
gpl.1.8
gpl.1.9
jto.1.3
jto.1.4
jto.1.5
jto.1.6
jto.1.7
jto.1.8
jto.1.9
prv.1.1
time(inseconds)
0102030
running feasible paths
running infeasible paths
solving constraints
RQ1
How
SEF
compares
to
an
ideal
sampler?
Average
overhead
of
SEF
compared
to
an
ideal
sampler
is
of
7%
>
1
second
<
1
minute

25. Observa%ons
• We
considered
9
previously-­‐used
SPLs
with
exis%ng
test
cases
(We
did
not
generate
new)
• For
most
SEF
runs
it
took
less
than
1
second
• BerkeleyDB
took
much
longer
to
run!
– System
tests
covering
most
features
in
each
path
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

26. RQ2
What
is
the
impact
of
further
constraining
the
model
on
SEF?
Can
one
iden%fy
groups
of
related
features
from
changes
and
one
test?

27. Possible
Future
Work
(I
need
your
help
here)
• Bener
understand
scalability
for
SPL
tes%ng
– Evaluate
the
importance
of
scalability
for
tes%ng
on
a
much
larger
set
of
SPLs
• Address
scalability
(if
indeed
a
big
problem)
– Possible
Solu%ons:
• Use
composi%onal
symbolic
execu%on
• Collapse
paths
with
similar
states
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

28. Possible
Future
Work
(I
need
your
help
here)
• Improve
automa%on
– Program
schemata
genera%on
can
be
imprac%cal
– Possible
Solu%on:
• Generate
products
on
demand
and
use
incremental
compila%on
• Use
Concolic
Execu%on
of
Features
(CEF)
• We
can
also
contribute
to
find
type-­‐errors
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br

29. Ques%ons
Revisited
• Is
efficiency
important?
• Is
combinatorial
explosion
(scalability)
rela%vely
less
important
for
tes%ng?
• Is
soundness
important
for
tes%ng?
Our
answer
was
YES
to
all!
What
is
your
opinion?
Dagstuhl
Seminar
13091,
Sabrina
Souto,
sfs@cin.ufpe.br