This document presents research comparing different conjoint methods for use in large product simulators. Preference-Based Conjoint (PBC) performed better than standard Adaptive Choice-Based Conjoint (ACBC) and Choice-Based Conjoint (CBC) in hit rate and accuracy. Further refinements to PBC, like adding past response data and brand covariates, improved performance up to a 40% hit rate. While improved, more research is still needed to validate PBC for very large simulators with 50 or more products.

1. Big simulators with
dozens of products:
Which conjoint method is most suitable?
Jeroen Hardon | Marco Hoogerbrugge

2. 2
Required prior knowledge: CBC vs ACBC
Choice-Based Conjoint: (nearly)
balanced level frequencies for every
respondent
Adaptive Choice-Based Conjoint: level
frequencies are skewed and different for every
respondent depending on his/her earlier
answers in BYO task
0
2
4
6
8
10
12
14
16
Frequency
30 GB was the
respondent’s
preferred level
in the earlier
BYO task
0
2
4
6
8
10
12
14
16
Frequency

3. 3
Introduction

4. 4
What we test What we simulate
What is our problem?

5. 5
Holdout tasks with
50-100 products are
impossible(?)
Hit rate in CBC 20.5%
> Much better than random (5%)
> But very poor in absolute sense
> And it will get worse in a real simulator
MAE 1.81%
> Nearly random (1.85%)
We have a wrong prediction for
80% of our respondents
In this test study we
used a holdout task with
20 concepts.

6. 6
Performance of standard ACBC* is very similar
Hit rate 19.8%
> Although slightly lower, the hit-rates are very comparable to CBC
> And again, it will get worse in a real simulator
MAE 1.83%
> Nearly random (1.85%)
* With 3 concepts per screen, no screening section, BYO tasks included in estimation

7. 7
Preference-Based Conjoint (PBC)
o level 1 (fixed)
o level 2 (fixed)
o level 3 (fixed)
o level 4 (fixed)
o level 5 (fixed)
o level 6 (fixed)
o respondent's choice in BYO task (flexible)
o respondent's choice in BYO task (flexible)

8. 8
Preference-Based Conjoint (PBC)
o level 1 (fixed)
o level 2 (fixed)
o level 3 (fixed)
o level 4 (fixed)
o level 5 (fixed)
o level 6 (fixed)
o respondent's choice in BYO task (flexible)
o respondent's choice in BYO task (flexible)
o respondent's choice in BYO + 1 level (flexible)
o respondent's choice in BYO - 1 level (flexible)

9. 9
Preference-Based Conjoint (PBC) as a midway
between CBC and ACBC
0
2
4
6
8
10
12
14
16
Frequency
0
2
4
6
8
10
12
14
16
Frequency
0
2
4
6
8
10
12
14
16
Frequency
ACBC PBC
(for example)
CBC

10. 10
Test study

11. Different “legs”, each 250 respondents
11
1. CBC
2. ACBC*
3. PBC*
4. PBC2*
5. PBC2, frequency of all attributes dependent on initial preference
6. PBC*, half with 4 concepts/task, half with partial profile 10 concepts/task
* Frequencies of three attributes dependent on initial preference (e.g. minutes and data, but not brand)

12. 12
Note on research leg with 10 concepts/screen
Task 1-6
Task 7-9
Task 10-12
Holdout
task

13. 13
Results

14. 0.01
0.015
0.02
0.15 0.2 0.25 0.3
14
Scatterplot of hit rate and MAE
Good direction
Wrong direction
Hit rate (FC-based)
MAE (SoP-based)

15. 15
PBC clearly performs better than currently
available methods
0.01
0.015
0.02
0.15 0.2 0.25 0.3
Hit rate (FC-based)
MAE (SoP-based)
PBC
ACBC CBC

16. 16
PBC2 worse than PBC
0.01
0.015
0.02
0.15 0.2 0.25 0.3
Hit rate (FC-based)
MAE (SoP-based)
PBC
ACBC CBC
PBC2

17. 17
PBC2 worse than PBC
0.01
0.015
0.02
0.15 0.2 0.25 0.3
Hit rate (FC-based)
MAE (SoP-based)
PBC
ACBC CBC
PBC2
PBC (all atts)
PBC2 (all atts)

18. 18
PBC2 worse than PBC
0.01
0.015
0.02
0.15 0.2 0.25 0.3
Hit rate (FC-based)
MAE (SoP-based)
PBC
ACBC CBC
PBC2
PBC2 (all atts)
PBC (partial concepts)

19. Adding BYO data improves hit rates even further
0.8%
1.2%
1.6%
2.0%
15.0% 20.0% 25.0% 30.0% 35.0% 40.0%
Hit rate (FC-based)
PBC PBC2 (all atts)
ACBC CBC
PBC2
PBC (partial concepts)
Note: for ACBC this is the default setting
19
MAE (SoP-based)

20. Adding brand covariate improves hit rates further
0.8%
1.2%
1.6%
2.0%
15.0% 20.0% 25.0% 30.0% 35.0% 40.0%
Hit rate (FC-based)
MAE (SoP-based)
PBC
PBC2 (all atts)
ACBC
CBC
PBC2
PBC (partial concepts)
20

21. 21
By combining multiple approaches
we successfully improved the
hit-rate from ~20% to ~40%
> New method – PBC
> Adding tasks with 10 partial profile concepts, sorted by brand
> Adding BYO data to estimation
> Adding current brand as a “smart” covariate

22. 22
Further research needed
> Applying PBC to all attributes seems promising
> Although we doubled the hit-rate from 20% to 40%, there are opportunities to improve further
> Can we have a holdout tasks with 50 products?
> More test studies needed to validate findings

23. SKIMgroup.com
Thank you
Jeroen Hardon
Director Methodology & Innovation
j.hardon@skimgroup.com
Marco Hoogerbrugge
Research Director
m.hoogerbrugge@skimgroup.com