ACBC outperformed CBC in predicting TV preferences in holdout tasks. Researchers tested variations of ACBC and found that performance was robust across variations. Dynamic CBC performed better than standard CBC. Increasing the number of price cutpoints in piecewise estimation significantly improved ACBC performance. However, models overestimated brand preferences and underestimated the effect of price compared to actual choices. Future research aims to address discrepancies between predictions and choices.

1. expect great answers
ACBC Revisited
2013 Sawtooth Software Conference
October 2013 | Marco Hoogerbrugge, Jeroen Hardon, Chris Fotenos

2. ACBC Introduction
2

3. Respondents first complete a build your own (BYO) exercise to
reveal their ideal product
3

4. Next they are shown a screening section where they can accept or
reject several possible purchase options
4

5. Within the screening section, respondents are asked whether certain
levels of attributes are ‘unacceptable’ or ‘must-have’ for them
5

6. Given some insight into the respondents’ preferences, they finally
see a series of choice tasks more aligned with their consideration set
6

7. Reduced design spaces for each respondent
•
In highly complex categories with
many possible concepts ACBC
helps to collect data on the
combinations that are most
relevant to each respondent.
Orthogonal designs are created
with the emphasis on the relevant
sphere for each respondent at the
cost of D-efficiency of the design
at individual level.
Design Space
Respondent 1
Attribute B
•
Design Space
Respondent 2
Design Space
Respondent 3
Attribute A
7

8. Aimed at getting more relevant choice data in big and complex
markets
•
Big markets  say, you have a simulator with 50 products (e.g. 7 brands
x 7 tiers)
• In regular CBC tasks a respondent chooses between concepts of
different brands and different tiers
• In ACBC, respondents choose between brands within a tier, or
between tiers within a brand, or even more refined
•
Complex markets
• Multi-attribute products (not just one SKU attribute)
• Optional: price dependent on all or most underlying product features
8

9. To assess and search for ways to improve ACBC, we
tested several variations of it along with other
methodologies in the US TV market.
9

10. Our thoughts behind also including CBC legs
•
What are possible success factors and potential weaknesses of ACBC?
And can we apply them also within CBC?
• Reduced design space per respondent
• Increased utility balance in every choice task
•
If ACBC is (perhaps) too extreme in applying these properties, CBC +
one of these properties might become more successful.
• Reduce design space  Dynamic CBC (see later)
• Increased utility balance  CBC with between concept prohibitions
10

11. Our thoughts behind improving ACBC
Price
Quality
Actual products in the market simulator
11

12. Modern televisions contain many features which creates a wide price
spectrum and more complex purchase decision for shoppers
Brand
Screen Size
Screen Type
LED, LCD, LED-LCD, Plasma
Resolution
720p, 1080p
Wi-Fi Capability
Yes/No
3D Capability
Yes/No
HDMI Inputs
0 – 3 HDMI Connections
Price
12
22”, 32”, 42”, 52”, 57”, 62”, 67”, 72”
$120 - $3,500

13. To benchmark and improve ACBC, we tested it against 7 other
methodologies, including slight variations of ACBC itself
Methodology
Description
A
CBC
Standard CBC
B
ACBC
Standard ACBC
C
ACBC
ACBC with price included in the unacceptable questions
D
Dynamic
CBC
SKIM-developed version of CBC where the design is adapted based on
previous answers
E
ACBC
ACBC with a more narrow tested price range of 90% to 120%
F
ACBC
ACBC with 4 attributes varied from the BYO choice in the generation of
concepts
G
CBC
Price balanced CBC
H
ACBC
Each leg 300
resps, total
2400 resps
ACBC without the screener section, giving every respondent 12 total
choice tasks
In all ACBC legs, brand was not included in the BYO task.
All legs showed respondents a summed total price for each concept (according to ACBC logic)
13

14. Dynamic CBC shows respondents a full set of CBC-like choice tasks
with an ACBC-like design that is adapted based on their responses
Like standard CBC, Dynamic CBC starts
with a near-D-efficient design space for all
respondents.
•
Respondents see a full set of choice tasks.
•
After 1/3 and 2/3 of the choice tasks, the
design space is narrowed down based on
the respondents’ choices in order to collect
more data on the concepts that are most
relevant to them.
Design Space: Tasks 1-3
Design Space: Tasks 4-6
Attribute B
•
Design Space: Tasks 7-9
Attribute A
14

15. Following the ACBC/CBC exercise, respondents were shown a
series of three holdout choice tasks
•
•
15
The holdout tasks presented
respondents with concepts
that were similar in their
features and total prices.
With a large number of
products in every holdout task,
this is much more like a real
simulator than a usual
holdout task of three or four
concepts!

16. Conjoint models can seldom predict one clear ‘winner’ in these
holdout tasks
•
Example of a share of preference distribution (model prediction) for a respondent:
concept1
1.48%
0.03%
concept12
0.88%
concept3
0.65%
concept13
29.78%
concept4
0.33%
concept14
0.95%
concept5
4.68%
concept15
27.31%
concept6
0.18%
concept16
12.99%
concept7
4.67%
concept17
4.67%
concept8
16
concept11
concept2
•
0.64%
2.40%
concept18
0.17%
concept9
1.75%
concept19
4.28%
This same problem may well occur with actual simulators with many products!
concept10
0.07%
concept20
2.11%

17. Success metric: mean SoP
So we thought it better to look at the means of the
predicted share of preference
of the
concepts that the respondent chose in the three holdout tasks.
17

18. First results: respondents enjoy taking a (well
designed) conjoint experience
18

19. Respondents liked taking the ACBC and CBC exercises with not
many finding the tasks too difficult to complete
“Thank you so much for letting
me know about different types
of TVs. Now I have an idea
what is the best TV I should
buy within 6 months.”
(CBC leg)
Standard CBC
Standard ACBC
ACBC w/ price in
unacceptables
Dynamic CBC
ACBC w/ price range
of 90%-120%
ACBC w/ 4 atts varied
from BYO
Price balanced CBC
ACBC without a
screening section
0%
I liked filling in this questionnaire
19
20%
40%
60%
80%
100%
It was difficult for me to fill in this questionnaire

20. Respondents liked taking the ACBC and CBC exercises with not
many finding the tasks too difficult to complete
“I liked the format used in this
survey. It was not confusing
and questions were based on
answers that I had previously
given rather than random
repetition.” (ACBC leg)
Standard CBC
Standard ACBC
ACBC w/ price in
unacceptables
Dynamic CBC
ACBC w/ price range
of 90%-120%
ACBC w/ 4 atts varied
from BYO
Price balanced CBC
ACBC without a
screening section
0%
I liked filling in this questionnaire
20
20%
40%
60%
80%
100%
It was difficult for me to fill in this questionnaire

21. Main results
21

22. All ACBC legs outperformed all CBC legs though there is not a
significant difference between different ACBC variations
ACBC without a screening
section
ACBC w/ price range of 90%120%
ACBC w/ 4 atts varied from
BYO
ACBC
Standard ACBC
ACBC w/ price in
unacceptables
Dynamic CBC
Price balanced CBC
CBC
Standard CBC
0%
10%
20%
Mean SoP of Selected Holdout Task Concept
22
30%

23. Intermezzo
All legs showed respondents a summed total price for each concept
(according to ACBC logic)
•
•
23
Summed price = (summed value of attribute levels) * random number
•  Price becomes a continuous variable
Consequently we applied piecewise utility estimation for price in all legs
• Number of cutpoints to be defined before running HB. ACBC allows up
to 12 cutpoints.

24. Intermezzo
Point estimation
Utility
3
2
1
0
-1 $0
-2
-3
-4
-5
24
$50
$100
$150
$200

25. Intermezzo
Piecewise estimation
Utility
3
2
1
0
-1 $0
-2
-3
-4
-5
25
$50
$100
$150
$200

26. Adding more cut points in the piecewise estimation (more granularity
in price utilities) significantly increases the performance, especially
of the ACBC legs!
ACBC without a
screening section
ACBC w/ price range of
90%-120%
ACBC w/ 4 atts varied
from BYO
ACBC
Standard ACBC
ACBC w/ price in
unacceptables
Dynamic CBC
Price balanced CBC
CBC
Standard CBC
0%
10%
20%
Mean SoP of Selected Holdout Task Concept
Original Analysis with 12 cut points
Analysis with 25 cutpoints
26
30%

27. We were thrilled about some of these findings but disappointed
about others
• +
• Omitting the screening section in ACBC isn’t bad
• A narrower price range in ACBC isn’t bad
• More cutpoints in piecewise estimation improves ACBC
• All models beat random choice by a factor 3-4
• +/• Dynamic CBC performs significantly better than standard CBC
27

28. We were disappointed about some of these findings but thrilled
about others
• • None of the alternative ACBC legs outperformed current ACBC
» The flipside of the coin is that ACBC is actually very robust!
• Price balanced CBC didn’t outperform standard CBC
• The absolute value of the mean SoP is actually quite low (~20%)
» This may well be the case in actual simulators with many products too!
28

29. Mean Squared Error in a holdout task
MSE =
∑
(Aggregate actual share – Aggregate mean predicted share)
2
concepts
Or
√
MSE =
√∑
(
(Aggregate actual share – Aggregate mean predicted share) 2 )
concepts
29

30. MSE as a success metric confirms previous conclusions
ACBC without a screening
section
ACBC w/ price range of 90%120%
ACBC w/ 4 atts varied from
BYO
ACBC
Standard ACBC
ACBC w/ price in
unacceptables
Dynamic CBC
Price balanced CBC
CBC
Standard CBC
0%
1%
2%
3%
4%
5%
6%
√(MSE), average of three holdout tasks
30

31. One concept in one holdout task causes a large amount of the high
MSE scores
•
Actual holdout share (aggregate level): 43%
•
Predicted holdout share:
• Standard CBC leg is worst (predicts 8%)
• CBC with price balance predicts 17%
• ACBC modules are better don’t come very close (avg prediction 27%)
31

32. Results with and without this one concept
ACBC without a screening
section
ACBC w/ price range of
90%-120%
ACBC w/ 4 atts varied from
BYO
ACBC
Standard ACBC
ACBC w/ price in
unacceptables
Dynamic CBC
CBC
Price balanced CBC
Standard CBC
0%
1%
2%
Excluding one concept
32
3%
4%
5%
6%
√(MSE), average of three holdout tasks
Original Analysis

33. A step aside
Cluster analysis on SoP predictions leads to a
conclusion but not to a solution
33

34. Cluster analysis across the legs leads to two layers in respondent
grouping
All respondents
Respondents who are
predicted to buy a ‘low end’ TV
(1/3 of sample)
Respondents who are
predicted to buy a ‘mid/high
end’ TV (2/3 of sample)
Different respondent groups for whom Different respondent groups for whom
a particular brand is predicted
a particular brand is predicted
34

35. Example: the high end Vizio cluster
Mean predicted SoP for holdout
task1
52.7%
Mean predicted SoP for holdout
task2
51.9%
Mean predicted SoP for holdout
task3
50.6%
LG
Panasonic
Samsung
Sony
Vizio
Actuals of holdout task1
Actuals of holdout task2
Actuals of holdout task3
0%
35
20%
40%
60%
80%
100%

36. The actual shares in the holdout tasks are much lower for Vizio
Mean predicted SoP for holdout
task1
52.7%
Mean predicted SoP for holdout
task2
51.9%
Mean predicted SoP for holdout
task3
50.6%
30.1%
Actuals of holdout task1
35.2%
Actuals of holdout task2
25.0%
Actuals of holdout task3
0%
36
20%
40%
60%
80%
100%
LG
Panasonic
Samsung
Sony
Vizio

37. Price plays a much bigger role in the actual choices than predicted
Mean predicted SoP for holdout
task1
52.7%
Mean predicted SoP for holdout
task2
51.9%
Mean predicted SoP for holdout
task3
50.6%
30.1%
Actuals of holdout task1
35.2%
Actuals of holdout task2
LG
Panasonic
Samsung
Sony
Vizio
25.0%
Actuals of holdout task3
0%
20%
40%
60%
80%
100%
Panasonic has cheapest product within the highest tier of holdout 1, Samsung 2nd cheapest.
Samsung has cheapest product within the lowest tier of holdout 3.
37

38. This same phenomenon applies to all clusters
•
•
38
Brand is apparently estimated as too important, at the cost of price.
This is perfectly in line with the most extreme mismatch in a concept that
we saw before in the MSE calculation: the share of the cheapest product
was greatly underestimated in this concept.

39. Another analytical approach (regression) leads to the same
conclusions
Dependents: actual holdout answers
Aggregate
multinominal
logistic
regression
Explanatory: SoP predictions for all holdout tasks
This aggregate regression leads to:
- Positive coefficients for ‘own’ SoP of a concept, as expected
- Also positive coefficients for SoPs of concepts with the same TV
of competing brands  in order to partially undo the brand
emphasis in the own SoPs
39

40. Conclusions
40

41. Conclusions
•
41
ACBC again outperforms CBC (in a complex market).

42. Conclusions
•
ACBC again outperforms CBC (in a complex market).
•
We’ve seen no significant differences within the tested ACBC variations
• This indicates that the performance of ACBC is robust.
• It seems like ACBC can be made more efficient (omitting screener) with the
same performance.
42

43. Conclusions
•
•
We’ve seen no significant differences within the tested ACBC variations
•
43
ACBC again outperforms CBC (in a complex market).
Dynamic CBC performs much better than standard CBC; further
refinement may lead to results close to or better than ACBC.

44. Conclusions
•
•
We’ve seen no significant differences within the tested ACBC variations
•
Dynamic CBC performs much better than standard CBC; further
refinement may lead to results close to or better than ACBC.
•
44
ACBC again outperforms CBC (in a complex market).
Performance of ACBC improves quite a lot by increasing the number of
price slopes >11.

45. Conclusions
•
•
We’ve seen no significant differences within the tested ACBC variations
•
Dynamic CBC performs much better than standard CBC; further
refinement may lead to results close to or better than ACBC.
•
Performance of ACBC improves quite a bit by increasing the cutpoints
for piecewise uitility estimation >11.
•
45
ACBC again outperforms CBC (in a complex market).
Respondents like a (well designed) conjoint survey better than we had
thought, including standard CBC.

46. Directions for future research
•
Finding solutions for the discrepancies that we found between actual
holdout answers and conjoint predictions
•
Developing more variants of Dynamic CBC
•
Double checking on existing data with holdout tasks
• Increase number of cutpoints in piecewise estimation
• Cluster analysis approach with predicted SoP
46

47. contact us or follow us online!
Marco Hoogerbrugge
Research Director
m.hoogerbrugge@skimgroup.com
Jeroen Hardon
Senior Research Consultant
j.hardon@skimgroup.com
Chris Fotenos
Project Manager
c.fotenos@skimgroup.com