1. Does the Sample Source Matter?
Wave 2 of a comparison of leading sample sources
Sept 07, 2012
2. Sample is the soil
and water of
research.
The quality of it
determines the
value of what you
produce.
3. We compared other sample sources that VC
regularly uses
We conducted a five minute survey April 12, 2012 with
a sample of about 500 people each, using sample from
Vision Critical’s Springboard America, and four of the
leading sample providers that we have anonymised as
“Panel B”, “River C”, “Panel D” and “River E”.
The questionnaire included items from the US 2010
General Social Survey, as well as a questions about
snack consumption and a concept test for a snack food.
We counted the number of statistically significant
differences between the panels (95% CI)
4. Methodology cont.
The questionnaire included items from the US 2010
General Social Survey, as well as a questions about
snack consumption and a concept test for a snack food.
We counted the number of statistically significant
differences between the panels (95% CI), and looked at
drop-out rates, straight lining, and the use of “don’t
know”.
5. Sample Quality: Dropout rate during the survey
Drop-outs as a percent of all who started the survey
Nov-11 Apr-12
20% 19%
18% 16%
16%
14%
12% 10%
10% 8% 9%
8% 7%
6% 5% 5% 5%
4% 4%
4% 2%
2% 0 0
0%
SBA - SBA - Panel B
Lightspeed Panel C
SSI Research
Panel D River United
Panel E
Targeted Routed Now Sample
6. Data quality: straight-lining
10%
9%
7.9% 7.8%
8%
7% 6.5%
6% 5.5% 5.6%
5% 4.8%
3.9%
4%
3.1%
3% 2.7%
2.1%
2%
NA
1% 0.7%
0.5%
0%
SBA Panel B
SBA Routed Lightspeed Panel C
SSI Panel D
Research SB River Panel E
uSamp
Now
7. Data quality: use of “don’t know”
Nov-11 Apr-12
10%
9%
8%
7%
6% 6% 6%
6%
5% 5%
4% 4% 4%
4% 4% 4%
4% 4%
3% 3%
2%
1%
0 0
0%
SBA - SBA - Panel B
Lightspeed Panel C
SSI Research SB River Panel E
Panel D uSamp
Targeted Routed Now
Average percent who choose the option "don’t know", "can't choose", "not
sure", "prefer not to say".
8. Differences between SBA and other panels in
how people answered all questions
10%
9%
8%
7% 7%
6%
Expected
5% 4% by chance
4%
4% 3%
3% 2% 2%
2%
1%
0%
Panel B
SBA Routed Lightspeed Panel C
SSI Panel D
Research Panel E
uSamp River
Now
Count of the number of statistically significant differences (95% CI) as a percent
of the total number of questions
9. Differences between SBA and other panels in
how people answered all questions
While, for the most part, these differences appear
within the realm of random, there were some
differences that seem to be very real.
For example, in both April and November Panel B
respondents were less likely to purchase snack
products and had lower purchase intent scores for
a snack concept.
10. Differences between panels in how people
answered the purchase intent question
Nov-11 Apr-12
50% 46%
44% 44%
45% 41% 40%
39%
40%
34% 33%
35% 31% 30%
30% 29%
25% 20%
20%
15%
10%
5% 0 0
0%
SBA - SBA - Lightspeed Panel C Research
Panel B SSI Panel D River Panel E
United
Targeted Routed Now Sample
Which statement below best describes how likely you would be purchase this
snack if it were available today?—top 2 box: definitely and probably
11. Conclusions
Sample matters. A lot.
Springboard America has fewer drop-outs, notably
better data quality, and is “right down the middle”
when it comes to comparison to the other panels on
the findings, unlike some panels which show real
variation—particularly on the concept test.
SBA also shows consistency in how people
answered, as illustrated by reaction to the concept.
12. Conclusions
The same cannot be said for the other panels,
however. They showed considerable variation
between time 1 and time 2 in how they reacted to
the snack concept, which was a non-trendy concept.
This may be a concern for tracking. I will dig deeper
into the comparison between waves in another
report.