1. BAD BOY MATRIX QUESTION
Whatcha gonna do when they come for you?
Florian Tress
ODC Services GmbH, Germany
GENERAL ONLINE RESEARCH 2012
5 – 7 March 2012 at the DHBW Mannheim

2. 2
THE VENN DIAGRAM OF FIELDWORK
Fun & entertainment Solve problems
Incentives
Collect (a lot of) valid data
Disport
Gain a lot of knowledge
Curiosity
Implement special
(and possibly boring)
Express an opinion methods
PEOPLE RESEARCHERS

3. 3
BAD BOY MATRIX QUESTION
RESEARCHERS
 Standardized data format, comparability of responses (with common orientation)
 Maximize amount of data, minimize length of interview
 Facilitate statistical procedures, e.g. factorial analysis, indices, etc.
 Availability of validated test instruments , e.g. Big Five
PEOPLE
 Monotonous, boring
 Overwhelming variety of options: “decision paralysis”
 Nondifferentiation, satisficing behavior
 Another question format would be more appropriate: Inferior data

4. 4
STANDARD ALTERNATIVES
MULTIPLE CHOICE
 Matrix with a two point scale
 Additional option “None of these” (mandatory question)
RANKING
 Bring statements in an order
 Perfectly differentiated data

5. 5
SPECIAL ALTERNATIVE: THE CAROUSEL
 Only one statement presented at once
 New statements slide in from the left
 Response options remain in the same place

6. 6
SPECIAL ALTERNATIVE: DRAG ME
 All statements presented at once
 Center stands for respondent
 Measures distance from center
 Arrangement of statements unimportant

7. 7
COMPARISON OF THESE ALTERNATIVES
Monadic Questionnaire : Random assignment to one of these alternatives
n = 1080; 216 interviews per alternative; good spread over age and education
Indicators: Comparability, Trustworthiness, Data Quality, Satisfaction, Technical Requirements

8. 8
COMPARABILITY: BRAND LIKEABILITY
Multiple Choice Carousel
Amazon 73 Amazon 4,1
3 Factors (Cum.%: 56) | Cronbach„s α: 0,77.
3 Factors (Cum.%: 58) | Cronbach„s α: 0,78.
Nivea 66 Nivea 4,1
Google 62 Google 3,9
IKEA 54 Volkswagen 3,7
Volkswagen 53 Ikea 3,5
Matrix
Nutella 52 Siemens 3,5
Coca Cola 47 Amazon 4,0 BMW 3,5
BMW 43 Nivea 3,9 Coca Cola 3,4
Siemens 39 Google 3,9 Nutella 3,4
McDonalds 39 Volkswagen 3,7 McDonalds 3,2
Nutella 3,6
Ranking Coca Cola 3,5 Drag Me
Siemens 3,5
Amazon 6,6 BMW 3,5 Amazon 68
3 Factors (Cum.%: 57) | Cronbach„s α: 0,73.
3 Factors (Cum.%: 49) | Cronbach„s α: n.a.
Nivea 5,8 IKEA 3,5 Google 66
Google 5,0 McDonalds 3,1 Nivea 61
Volkswagen 4,6 3 Factors (Cum.%: 60) | Cronbach„s α: 0,80. Coca Cola 57
BMW 4,2 Ikea 52
Coca Cola 4,2 Siemens 52
Nutella 3,9 Volkswagen 51
IKEA 3,8 Nutella 49
Siemens 3,7 BMW 48
McDonalds 3,2 McDonalds 44

9. 9
COMPARABILITY: ATTITUDES
Multiple Choice Carousel
A 62 A 3,9
3 Factors (Cum.%: 50) | Cronbach„s α: 0,56.
3 Factors (Cum.%: 61) | Cronbach„s α: 0,80.
C 44 C 3,7
E 38 B 3,5
F 30 D 3,5
D 28 E 3,4
Matrix
B 28 F 3,4
G 24 A 3,8 H 3,1
J 18 B 3,6 G 3,1
H 14 C 3,5 J 3,0
I 11 D 3,5 I 2,9
E 3,2
Ranking F 3,2 Drag Me
G 3,1
C 6,4 H 3,0 A 65
3 Factors (Cum.%: 63) | Cronbach„s α: 0,82.
3 Factors (Cum.%: 51) | Cronbach„s α: n.a.
A 6,4 I 2,8 C 63
D 4,9 J 2,8 E 55
E 4,9 3 Factors (Cum.%: 63) | Cronbach„s α: 0,82. F 52
F 4,6 G 50
B 4,0 D 49
J 3,8 B 44
I 3,6 J 41
H 3,4 I 39
G 3,0 H 32
(A) Wenn ich gute Erfahrungen mit einer Marke mache, empfehle ich sie aktiv weiter. (B) Werbung sollte mich stärker über Marken informieren, die ich noch nicht kenne. (C) Ich
habe feste Marken, die ich bevorzugt einkaufe. (D) Neuartige und innovative Produkte passen gut zu meinem Lebensstil. (E) Ich probiere häufig neue Marken aus, die ich noch
nicht kenne. (F) Ich bevorzuge Marken, die auf eine lange Tradition zurückblicken. (G) Ich bin bereit, für Markenprodukte mehr Geld auszugeben. (H) Werbung sollte mich stärker
über Marken informieren, die ich bereits gut kenne. (I) Je bekannter eine Marke ist, desto leichter kann man ihr vertrauen. (J) Die Bekanntheit einer Marke sagt etwas ihre Qualität aus.

10. 10
RESULT: COMPARABILITY
Results correspond (roughly) for the most / least likeable brands (agreeable statements)
But: Drag Me seems to measure something different / to be biased by third variables
WINNER LOSER

11. 11
TRUSTWORTHINESS: BRAND LIKEABILITY
Follow-Up-Exploration: Why do you think, this is the most / least likeable brand?
MULTIPLE CHOICE MATRIX CAROUSEL
Ø Words pos.: 5,5 Ø Words pos.: 6,8 Ø Words pos.: 7,7
Ø Words neg.: 5,4 Ø Words neg.: 7,3 Ø Words neg.: 7,0
Non-Response: 3% Non-Response: 2% Non-Response: 3%
RANKING DRAG ME
Ø Words pos.: 6,3 Ø Words pos.: 7,2
Ø Words neg.: 6,7 Ø Words neg.: 7,2
Non-Response: 3% Non-Response: 2%

12. 12
TRUSTWORTHINESS: ATTITUDES
Follow-Up-Exploration: Why did you agree / disagree with this statement?
MULTIPLE CHOICE MATRIX CAROUSEL
Ø Words pos.: 7,4 Ø Words pos.: 7,9 Ø Words pos.: 9,3
Ø Words neg.: 8,6 Ø Words neg.: 9,6 Ø Words neg.: 10,0
Non-Response: 11% Non-Response: 9% Non-Response: 8%
RANKING DRAG ME
Ø Words pos.: 8,3 Ø Words pos.: 8,4
Ø Words neg.: 8,1 Ø Words neg.: 8,6
Non-Response: 7% Non-Response: 7%

13. 13
RESULT: TRUSTWORTHINESS
WINNER
LOSER
nonspecific / general specific / rich in detail
„because I like it“ „good quality, fair prices“

14. 14
DATA QUALITY: NONDIFFERENTIATION
13%
8%
8%
6%
5%
4%
0% 0% 0% 0%
Multiple Choice Ranking Matrix Carousel Drag Me
Brand Likeability Attitudes
Low sample size!

15. 15
DATA QUALITY: NONDIFFERENTIATION
younger  40  older
7%
Age
3%
5% 2% 1%
6% 2%
5%
2% 3% 3%
2%
Brand Likeability Attitudes Brand Likeability Attitudes Brand Likeability Attitudes
lower A-L. higher
7%
Education
3%
4%
3% 2%
6% 2%
5%
3% 2% 2% 2%
Brand Likeability Attitudes Brand Likeability Attitudes Brand Likeability Attitudes
MULTIPLE CHOICE MATRIX CAROUSEL
Low sample size!

16. 16
RESULT: DATA QUALITY
The data is perfectly differentiated with the Ranking and Drag Me Question.
Among the other three alternatives, the Carousel performs best.
WINNER
LOSER

17. 17
SATISFACTION
Layout Usability Comprehensibility
Carousel 71 20 5 Carousel 75 19 3 Carousel 74 15 2
Ranking 67 22 11 Ranking 73 20 6 Ranking 71 15 4
DragMe 66 23 10 DragMe 71 18 8 Matrix 70 16 4
Matrix 65 25 9 Matrix 70 23 6 DragMe 70 16 5
Multiple Choice 63 31 4 Multiple Choice 66 28 3 Multiple Choice 67 17 4
Topic Length Fun
Carousel 64 15 8 Carousel 69 15 13 Carousel 62 14 10
Ranking 57 14 11 Ranking 66 18 11 Ranking 57 13 10
DragMe 55 16 11 DragMe 64 19 11 Matrix 54 13 12
Multiple Choice 54 15 11 Multiple Choice 61 21 15 DragMe 54 16 11
Matrix 54 12 14 Matrix 60 22 15 Multiple Choice 53 14 12

18. 18
SATISFACTION
Overall
Carousel 59 24 14
Ranking 58 25 16
DragMe 57 25 15
Matrix 46 34 15
Multiple Choice 45 35 17
WINNER LOSER

19. 19
SUMMARY
MULTIPLE
RANKING MATRIX CAROUSEL DRAG ME
CHOICE
Comparability
Trustworthiness
Data Quality
Satisfaction
Technical
none JScript none JScript, Flash JScript, Flash
Requirements

20. 20
RECOMMENDATIONS
CHECK, IF YOUR STUDY PERMITS THE USAGE OF JSCRIPT AND FLASH!
(in most cases, it will)
SELECT THE QUESTIONTYPES CAREFULLY!
(there might be better alternatives)
LAYOUT AND USABILITY MATTER!
(the longer the interview, the more they matter)
IF YOU HAVE DOUBTS ASK YOUR FIELDWORK PROVIDER!
(they should have enough experience)

21. FLORIAN TRESS
THANK YOU
VERY MUCH!
www.odc-services.com
f.tress@odc-services.com
@FTress