Scalable Product Line Configuration: A Straw to Break the Camel’s Back Abdel Salam Sayyad Joseph Ingram Tim Menzies Hany Ammar IEEE Automated SE, Palo Alto, CA Nov 2013

1. Scalable
Product
Line
Configura4on:
A
Straw
to
Break
the
Camel’s
Back
Abdel
Salam
Sayyad
Joseph
Ingram
Tim
Menzies
Hany
Ammar

2. Sound
bites
Real
SoGware
Product
Lines…
…
are
big,
complex
…
representa4ve
feature
models
are
now
available
for
research
…
scale
up
or
die
!
What
does
it
take?
Mul4-­‐objec4ve
problem
formula4on
An
op4mizer
that
relies
heavily
on
user
preferences
Respect
domain
constraints,
and...
…
A
Straw
to
Break
the
Camel’s
Back
2

3. Roadmap
① State
of
the
Art
② Experiment
&
Results
③ Conclusion

4. Roadmap
① State
of
the
Art
② Experiment
&
Results
③ Conclusion

5. Feature
Models
• Feature
models
=
a
lightweight
method
for
defining
a
space
of
op4ons
• De
facto
standard
for
modeling
variability
Cross-­‐Tree
Constraints
Size
?
10
Features,
8
Rules
Cross-­‐Tree
Constraints
5

6. Feature
Model
Repository
Size
?
290
Features,
421
Rules
cardinal
=
2.26x1049
[Pohl
‘11]
6

7. Feature
Models
of
Real
SoGware
Projects
Size
?
6888
Features;
344,000
Rules
cardinal?
Forgeddabou4t!
7

8. Size
isn’t
all…
Property
Size
Constraints
Feature
Groups
Leaves
SPLOT
Linux
Significantly
smaller
Significantly
larger
Significantly
less
Significantly
more
High
ra4o
Low
ra4o
Deeper
Shallower
T.
Berger,
S.
She,
R.
Lotufo,
A.
Wasowski,
and
K.
Czarnecki,
"A
Study
of
Variability
Models
and
Languages
in
the
Systems
SoGware
Domain,"
IEEE
Tran
So<
Eng.
2013
8

9. A
Mul4-­‐Objec4ve
Formula4on
Mul4-­‐objec4ve
op4miza4on
=
naviga4ng
compe4ng
concerns
– Success
criteria
=
choose
features
that
achieve
the
user
preferences!
Suppose
each
feature
had
the
following
metrics:
1. Boolean
USED_BEFORE?
2. Integer
DEFECTS
3. Real
COST
Show
me
the
space
of
“best
op4ons”
according
to
the
objec4ves:
1.
2.
3.
4.
5.
That
sa4sfies
most
domain
constraints
(0
≤
#viola4ons
≤
100%)
That
offers
most
features
That
we
have
used
most
before
Using
features
with
least
known
defects
Using
features
with
least
cost
9

10. No
single
“op4mum”
solu4on
The
Pareto
Front
Higher-­‐level
Decision
Making
The
Chosen
Solu4on
10

11. Features
Single-­‐obj
Mul4-­‐obj
Linux
(LVAT)
6888
State
of
the
Art
544
SPLOT
290
9
Objec4ves
11

12. Features
6888
State
of
the
Art
Mul4-­‐obj
Linux
(LVAT)
Single-­‐obj
544
SPLOT
290
Single-­‐obj,
CSP
9
Up
to
25
features
Exponen[al
[me
Benavides
‘05
Objec4ves
12

13. Features
State
of
the
Art
6888
Mul4-­‐obj
Linux
(LVAT)
Single-­‐obj
544
SPLOT
290
9
Pohl
‘11
Benavides
‘05
BDD,
SAT,
CSP,
90
models
“All
products”
not
a`empted
if
cardinal
>
3x106
Objec4ves
13

14. Features
State
of
the
Art
6888
Mul4-­‐obj
Linux
(LVAT)
Single-­‐obj
544
SPLOT
290
9
Pohl
‘11
Benavides
‘05
Lopez-­‐
Herrejon
‘11
Fixing
model
inconsistencies
27
min.
for
94
features
Objec4ves
14

15. Features
State
of
the
Art
6888
Mul4-­‐obj
Linux
(LVAT)
Single-­‐obj
544
SPLOT
290
9
Pohl
‘11
Benavides
‘05
Lopez-­‐
Herrejon
‘11
Sayyad
’13a
Mul[-­‐obj
configura[on,
Z3
>15
days
for
E-­‐Shop
Objec4ves
15

16. State
of
the
Art
Features
6888
Mul4-­‐obj
Linux
(LVAT)
Single-­‐obj
544
White
‘07,
‘08,
09a,
09b,
Shi
‘10,
Guo
‘11
SPLOT
290
9
Pohl
‘11
Benavides
‘05
Lopez-­‐
Herrejon
‘11
Scale
up
with
randomly-­‐
generated
feature
models
Sayyad
’13a
Velazco
‘13
Objec4ves
16

17. State
of
the
Art
Features
Linux
(LVAT)
6888
544
SPLOT
290
9
Single-­‐obj
Mul4-­‐obj
Johansen
‘11
Test
covering
arrays,
[med
out
on
all
ops
for
Linux
6888
features
(and
some
ops
on
smaller
FMs)
Pohl
‘11
Benavides
‘05
Lopez-­‐
Herrejon
‘11
Sayyad
’13a
Velazco
‘13
Objec4ves
17

18. State
of
the
Art
Features
Linux
(LVAT)
6888
Single-­‐obj
Johansen
‘11
Mul4-­‐obj
Henard
‘12
Test
covering
arrays
20
hours
for
Linux
6888
features
6-­‐wise
coverage
544
SPLOT
290
9
Pohl
‘11
Benavides
‘05
Lopez-­‐
Herrejon
‘11
Sayyad
’13a
Velazco
‘13
Objec4ves
18

19. State
of
the
Art
Features
Linux
(LVAT)
6888
Single-­‐obj
Johansen
‘11
Mul4-­‐obj
Henard
‘12
Sayyad
’13b
Mul[-­‐obj
configura[on,
IBEA
30
configs
in
30
minutes
for
Linux
6888
features
544
SPLOT
290
9
Pohl
‘11
Benavides
‘05
Lopez-­‐
Herrejon
‘11
Sayyad
’13a
Velazco
‘13
Objec4ves
19

20. Roadmap
① State
of
the
Art
② Experiment
&
Results
③ Conclusion

21. Two
Algorithms
1) NSGA-­‐II
[Deb
et
al.
‘02]
Non-­‐dominated
Sor4ng
Gene4c
Algorithm
Binary
Dominance
2)
IBEA
[Zitzler
and
Kunzli
‘04]
Indicator-­‐Based
Evolu4onary
Algorithm
Con4nuous
Dominance
21

22. 7
Feature
Models
22

23. Feature
Fixing
• Look
for
mandatory
or
dead
features.
• Fix
those
feature
in
the
evolu4on.
• Skip
rules
with
only
those
features.
23

24. Experiment
Low
Crossover
&
Muta[on
rates
Stopping
Criteria
24

25. HV
=
Hypervolume
of
dominated
region
%Correct
=
%
of
solu4ons
that
are
fully
correct
(out
of
300
solu4ons)
25

26. Comparing
%Correct
i.e.
%
of
solu4ons
that
are
fully
correct
(out
of
300
solu4ons)
NSGA-­‐II
No
FF
100%
NSGA-­‐II
with
FF
80%
IBEA
No
FF
IBEA
with
FF
60%
40%
20%
0%
ToyBox
axTLS
eCos
FreeBSD
Fiasco
uClinux
Linux
X86
26

27. How
to
crack
the
nut?
• What
if
we
told
IBEA
what
good
solu4ons
look
like?
• Good
solu4on?
…
A
correct
solu4on,
with:
• Minimal
skeleton?
• Maximum
features?
• Then
what?
• Seed
the
ini4al
popula4on
with
a
pre-­‐calculated
“good”
solu4on.
27

28. • How
to
generate
good
(correct)
solu4ons?
• Z3
SMT
(Sa4sfiability
Module
Theory)
Solver
• Two-­‐objec4ve
IBEA,
min
viola4ons,
max
features
• Z3:
super
fast,
low
on
selected
features
• 2-­‐obj
IBEA:
slow,
but
feature-­‐rich
28

29. Correct
solu4ons
aGer
30
minutes
for
Linux
Kernel
feature
model
29

30. Correct
solu4ons
aGer
30
minutes
for
Linux
Kernel
feature
model
…
A
Straw
to
Break
the
Camel’s
Back
30

31. More
recent
result
• added
the
“PULL”
trick
(give
more
objec4ve
weight
to
constraint
viola4ons)
31

32. Why
is
this
interes4ng?
• 20
correct
solu4ons
to
choose
from
within
2
minutes
32

33. Do
I
have
the
user’s
aven4on?
• 2
seconds…
• to
stay
focused
• 10
seconds…
• to
stay
on
task
• few
minutes?
• To
provide
4mely
input
at
a
mee4ng
33

34. Roadmap
① State
of
the
Art
② Experiment
&
Results
③ Conclusion

35. Sound
bites
Scalability
Method
innova[on
is
key
Problem
Formula4on
Give
the
best
(comprehensive
yet
concise)
picture
to
the
user.
IBEA
with
Feature
Fixing
Con[nuous
dominance
Respect
domain
constraints
Enter
Z3?
Acknowledgment
This
research
work
was
funded
by
the
Qatar
Na[onal
Research
Fund
(QNRF)
under
the
Na[onal
Priori[es
Research
Program
(NPRP)
Grant
No.:
09-­‐1205-­‐2-­‐470.
One
day
we
will
build
that
bridge!
35