SlideShare una empresa de Scribd logo
1 de 12
Descargar para leer sin conexión
Does the Sample Source Matter?
Wave 2 of a comparison of leading sample sources
Sept 07, 2012
Sample is the soil
and water of
research.

The quality of it
determines the
value of what you
produce.
We compared other sample sources that VC
regularly uses
We conducted a five minute survey April 12, 2012 with
a sample of about 500 people each, using sample from
Vision Critical’s Springboard America, and four of the
leading sample providers that we have anonymised as
“Panel B”, “River C”, “Panel D” and “River E”.

The questionnaire included items from the US 2010
General Social Survey, as well as a questions about
snack consumption and a concept test for a snack food.

We counted the number of statistically significant
differences between the panels (95% CI)
Methodology cont.


The questionnaire included items from the US 2010
General Social Survey, as well as a questions about
snack consumption and a concept test for a snack food.

We counted the number of statistically significant
differences between the panels (95% CI), and looked at
drop-out rates, straight lining, and the use of “don’t
know”.
Sample Quality: Dropout rate during the survey
                 Drop-outs as a percent of all who started the survey


                                     Nov-11     Apr-12
20%                                                                  19%
18%                                                                16%
16%
14%
12%                                       10%
10%                             8%                            9%
 8%                                                      7%
 6%         5%            5%                  5%
       4%                            4%
 4%                                                                              2%
 2%                   0                                                      0
 0%
        SBA -        SBA - Panel B
                           Lightspeed Panel C
                                        SSI          Research
                                                     Panel D       River    United
                                                                           Panel E
      Targeted      Routed                             Now                  Sample
Data quality: straight-lining

10%
 9%
                                        7.9%                 7.8%
 8%
 7%                          6.5%
 6%                   5.5%     5.6%
 5%                                       4.8%
                                                  3.9%
 4%
                                                                          3.1%
 3%                                                  2.7%
                                                                2.1%
 2%
                 NA
 1%      0.7%
      0.5%
 0%
       SBA                  Panel B
                SBA Routed Lightspeed   Panel C
                                          SSI     Panel D
                                                  Research   SB River   Panel E
                                                                        uSamp
                                                    Now
Data quality: use of “don’t know”
                                 Nov-11        Apr-12
10%
 9%
 8%
 7%
                            6%            6%                      6%
 6%
 5%                              5%
       4% 4%          4%
                                               4%       4%             4%
 4%                                                          4%
 3%                                                                             3%
 2%
 1%
                  0                                                         0
 0%
        SBA -     SBA - Panel B
                         Lightspeed Panel C
                                      SSI           Research SB River Panel E
                                                     Panel D           uSamp
      Targeted   Routed                               Now


  Average percent who choose the option "don’t know", "can't choose", "not
  sure", "prefer not to say".
Differences between SBA and other panels in
how people answered all questions

10%
 9%
 8%
 7%                                                7%
 6%
                                                                                    Expected
 5%                                                             4%                  by chance
            4%
 4%                                   3%
 3%                      2%                                                  2%
 2%
 1%
 0%
                    Panel B
        SBA Routed Lightspeed        Panel C
                                      SSI        Panel D
                                                Research      Panel E
                                                              uSamp         River
                                                  Now

      Count of the number of statistically significant differences (95% CI) as a percent
      of the total number of questions
Differences between SBA and other panels in
how people answered all questions


While, for the most part, these differences appear
within the realm of random, there were some
differences that seem to be very real.

For example, in both April and November Panel B
respondents were less likely to purchase snack
products and had lower purchase intent scores for
a snack concept.
Differences between panels in how people
answered the purchase intent question
                               Nov-11       Apr-12
50%                                                46%
                                         44%                               44%
45%                                              41%          40%
                      39%
40%
      34%                             33%
35%     31%                   30%
30%                                                        29%
25%                         20%
20%
15%
10%
 5%               0                                                    0
 0%
        SBA -     SBA - Lightspeed Panel C Research
                         Panel B    SSI     Panel D         River   Panel E
                                                                     United
      Targeted   Routed                      Now                     Sample


  Which statement below best describes how likely you would be purchase this
  snack if it were available today?—top 2 box: definitely and probably
Conclusions


Sample matters. A lot.

Springboard America has fewer drop-outs, notably
better data quality, and is “right down the middle”
when it comes to comparison to the other panels on
the findings, unlike some panels which show real
variation—particularly on the concept test.

SBA also shows consistency in how people
answered, as illustrated by reaction to the concept.
Conclusions


The same cannot be said for the other panels,
however. They showed considerable variation
between time 1 and time 2 in how they reacted to
the snack concept, which was a non-trendy concept.

This may be a concern for tracking. I will dig deeper
into the comparison between waves in another
report.

Más contenido relacionado

Similar a Does Sample Source Matter? [Part 2]

Engaging interaction part two
Engaging interaction part twoEngaging interaction part two
Engaging interaction part twocarolinestallings
 
Group 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hdGroup 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hdRobin Agarwal
 
2012 october 3 survey of tunisian public opinion, july 26 august 8, 2012
2012 october 3 survey of tunisian public opinion, july 26 august 8, 20122012 october 3 survey of tunisian public opinion, july 26 august 8, 2012
2012 october 3 survey of tunisian public opinion, july 26 august 8, 2012Moez
 
Global Opportunities for Social & Mobiles Games - Rex Ng, 6waves
Global Opportunities for Social & Mobiles Games -  Rex Ng, 6waves  Global Opportunities for Social & Mobiles Games -  Rex Ng, 6waves
Global Opportunities for Social & Mobiles Games - Rex Ng, 6waves LondonGamesConference
 
Hispanic Labor in the Swine Industry… a Thing of the Past?
Hispanic Labor in the Swine Industry…  a Thing of the Past?Hispanic Labor in the Swine Industry…  a Thing of the Past?
Hispanic Labor in the Swine Industry… a Thing of the Past?John Blue
 
Delvinia Digital Diseases Presentation Smei
Delvinia Digital Diseases Presentation SmeiDelvinia Digital Diseases Presentation Smei
Delvinia Digital Diseases Presentation SmeiDelvinia
 
Brave New World: The End of Survey Research
Brave New World: The End of Survey ResearchBrave New World: The End of Survey Research
Brave New World: The End of Survey ResearchMichael Bystry
 
IMCA 2009 Practice Management Survey Results
IMCA 2009 Practice Management Survey ResultsIMCA 2009 Practice Management Survey Results
IMCA 2009 Practice Management Survey ResultsExplore Atlanta
 
From suppressed urbanization to inter-city mobility in the Israeli Arab sector
From suppressed urbanization to inter-city mobility in the Israeli Arab sectorFrom suppressed urbanization to inter-city mobility in the Israeli Arab sector
From suppressed urbanization to inter-city mobility in the Israeli Arab sectorNachman Shelef
 
What Are Our Prospects Are Thinking Right Now Michael Starke, Vp Of Research
What Are Our Prospects Are Thinking Right Now   Michael Starke, Vp Of ResearchWhat Are Our Prospects Are Thinking Right Now   Michael Starke, Vp Of Research
What Are Our Prospects Are Thinking Right Now Michael Starke, Vp Of Researchmichaelstarke
 
Group 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hdGroup 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hdRobin Agarwal
 
Comparativo estudios de_opinion
Comparativo estudios de_opinionComparativo estudios de_opinion
Comparativo estudios de_opinionMartin Triana
 
Univision Poll Results
Univision Poll Results Univision Poll Results
Univision Poll Results univisionnews
 
Genome Olympics viewing habits
Genome Olympics viewing habitsGenome Olympics viewing habits
Genome Olympics viewing habitsMaria Pousa
 
123people online privacy survey results
123people online privacy survey results123people online privacy survey results
123people online privacy survey results123people
 

Similar a Does Sample Source Matter? [Part 2] (20)

Engaging interaction part two
Engaging interaction part twoEngaging interaction part two
Engaging interaction part two
 
PBS LM intro_052012
PBS LM intro_052012PBS LM intro_052012
PBS LM intro_052012
 
Group 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hdGroup 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hd
 
Moral Panics over the Internet
Moral Panics over the InternetMoral Panics over the Internet
Moral Panics over the Internet
 
2012 october 3 survey of tunisian public opinion, july 26 august 8, 2012
2012 october 3 survey of tunisian public opinion, july 26 august 8, 20122012 october 3 survey of tunisian public opinion, july 26 august 8, 2012
2012 october 3 survey of tunisian public opinion, july 26 august 8, 2012
 
Global Opportunities for Social & Mobiles Games - Rex Ng, 6waves
Global Opportunities for Social & Mobiles Games -  Rex Ng, 6waves  Global Opportunities for Social & Mobiles Games -  Rex Ng, 6waves
Global Opportunities for Social & Mobiles Games - Rex Ng, 6waves
 
Hispanic Labor in the Swine Industry… a Thing of the Past?
Hispanic Labor in the Swine Industry…  a Thing of the Past?Hispanic Labor in the Swine Industry…  a Thing of the Past?
Hispanic Labor in the Swine Industry… a Thing of the Past?
 
UBS Ad:Tech 2009
UBS Ad:Tech 2009UBS Ad:Tech 2009
UBS Ad:Tech 2009
 
Delvinia Digital Diseases Presentation Smei
Delvinia Digital Diseases Presentation SmeiDelvinia Digital Diseases Presentation Smei
Delvinia Digital Diseases Presentation Smei
 
Brave New World: The End of Survey Research
Brave New World: The End of Survey ResearchBrave New World: The End of Survey Research
Brave New World: The End of Survey Research
 
IMCA 2009 Practice Management Survey Results
IMCA 2009 Practice Management Survey ResultsIMCA 2009 Practice Management Survey Results
IMCA 2009 Practice Management Survey Results
 
From suppressed urbanization to inter-city mobility in the Israeli Arab sector
From suppressed urbanization to inter-city mobility in the Israeli Arab sectorFrom suppressed urbanization to inter-city mobility in the Israeli Arab sector
From suppressed urbanization to inter-city mobility in the Israeli Arab sector
 
What Are Our Prospects Are Thinking Right Now Michael Starke, Vp Of Research
What Are Our Prospects Are Thinking Right Now   Michael Starke, Vp Of ResearchWhat Are Our Prospects Are Thinking Right Now   Michael Starke, Vp Of Research
What Are Our Prospects Are Thinking Right Now Michael Starke, Vp Of Research
 
Thomas net project review
Thomas net project reviewThomas net project review
Thomas net project review
 
Group 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hdGroup 3 arp presentation_final_bj_hd
Group 3 arp presentation_final_bj_hd
 
Raddon Chart of the Day May 17, 2012
Raddon Chart of the Day May 17, 2012Raddon Chart of the Day May 17, 2012
Raddon Chart of the Day May 17, 2012
 
Comparativo estudios de_opinion
Comparativo estudios de_opinionComparativo estudios de_opinion
Comparativo estudios de_opinion
 
Univision Poll Results
Univision Poll Results Univision Poll Results
Univision Poll Results
 
Genome Olympics viewing habits
Genome Olympics viewing habitsGenome Olympics viewing habits
Genome Olympics viewing habits
 
123people online privacy survey results
123people online privacy survey results123people online privacy survey results
123people online privacy survey results
 

Más de vcuniversity

What Pandora Means for Radio
What Pandora Means for RadioWhat Pandora Means for Radio
What Pandora Means for Radiovcuniversity
 
Blending New Research Techniques & Technologies
Blending New Research Techniques & TechnologiesBlending New Research Techniques & Technologies
Blending New Research Techniques & Technologiesvcuniversity
 
The Vox Populi on Canada’s Municipal Brands
The Vox Populi on Canada’s Municipal BrandsThe Vox Populi on Canada’s Municipal Brands
The Vox Populi on Canada’s Municipal Brandsvcuniversity
 
Stronger Research Reporting Using Visuals
Stronger Research Reporting Using VisualsStronger Research Reporting Using Visuals
Stronger Research Reporting Using Visualsvcuniversity
 
Virtual Shopping Mythbusters - Part 1
Virtual Shopping Mythbusters - Part 1Virtual Shopping Mythbusters - Part 1
Virtual Shopping Mythbusters - Part 1vcuniversity
 
Virtual Shopping Mythbusters - Part Two
Virtual Shopping Mythbusters - Part Two  Virtual Shopping Mythbusters - Part Two
Virtual Shopping Mythbusters - Part Two vcuniversity
 
Respondent Engagement: Why do they respond anyway?
Respondent Engagement: Why do they respond anyway?Respondent Engagement: Why do they respond anyway?
Respondent Engagement: Why do they respond anyway?vcuniversity
 
Test, re-test and test again – New techniques and technologies for testing id...
Test, re-test and test again – New techniques and technologies for testing id...Test, re-test and test again – New techniques and technologies for testing id...
Test, re-test and test again – New techniques and technologies for testing id...vcuniversity
 
Mobile Research (Stockholm, June 2012)
Mobile Research (Stockholm, June 2012)Mobile Research (Stockholm, June 2012)
Mobile Research (Stockholm, June 2012)vcuniversity
 
Key Trends in Market Research (Stockholm, June 2012)
Key Trends in Market Research (Stockholm, June 2012)Key Trends in Market Research (Stockholm, June 2012)
Key Trends in Market Research (Stockholm, June 2012)vcuniversity
 
Driving Your Brand with Research Communities (Stockholm June 2012)
Driving Your Brand with Research Communities (Stockholm June 2012)Driving Your Brand with Research Communities (Stockholm June 2012)
Driving Your Brand with Research Communities (Stockholm June 2012)vcuniversity
 
What Clients & Customers Want
What Clients & Customers WantWhat Clients & Customers Want
What Clients & Customers Wantvcuniversity
 
Live Forum Results
Live Forum ResultsLive Forum Results
Live Forum Resultsvcuniversity
 
Keeping Customer Conversations Real
Keeping Customer Conversations RealKeeping Customer Conversations Real
Keeping Customer Conversations Realvcuniversity
 
Customer experience management
Customer experience managementCustomer experience management
Customer experience managementvcuniversity
 
Selling Panels Internally - Australia Post
Selling Panels Internally - Australia PostSelling Panels Internally - Australia Post
Selling Panels Internally - Australia Postvcuniversity
 
What Can Marketing Learn from Vanilla Ice
What Can Marketing Learn from Vanilla IceWhat Can Marketing Learn from Vanilla Ice
What Can Marketing Learn from Vanilla Icevcuniversity
 
The Future of Research
The Future of Research The Future of Research
The Future of Research vcuniversity
 

Más de vcuniversity (20)

What Pandora Means for Radio
What Pandora Means for RadioWhat Pandora Means for Radio
What Pandora Means for Radio
 
Blending New Research Techniques & Technologies
Blending New Research Techniques & TechnologiesBlending New Research Techniques & Technologies
Blending New Research Techniques & Technologies
 
The Vox Populi on Canada’s Municipal Brands
The Vox Populi on Canada’s Municipal BrandsThe Vox Populi on Canada’s Municipal Brands
The Vox Populi on Canada’s Municipal Brands
 
Stronger Research Reporting Using Visuals
Stronger Research Reporting Using VisualsStronger Research Reporting Using Visuals
Stronger Research Reporting Using Visuals
 
Virtual Shopping Mythbusters - Part 1
Virtual Shopping Mythbusters - Part 1Virtual Shopping Mythbusters - Part 1
Virtual Shopping Mythbusters - Part 1
 
Virtual Shopping Mythbusters - Part Two
Virtual Shopping Mythbusters - Part Two  Virtual Shopping Mythbusters - Part Two
Virtual Shopping Mythbusters - Part Two
 
Respondent Engagement: Why do they respond anyway?
Respondent Engagement: Why do they respond anyway?Respondent Engagement: Why do they respond anyway?
Respondent Engagement: Why do they respond anyway?
 
Test, re-test and test again – New techniques and technologies for testing id...
Test, re-test and test again – New techniques and technologies for testing id...Test, re-test and test again – New techniques and technologies for testing id...
Test, re-test and test again – New techniques and technologies for testing id...
 
Mobile Research (Stockholm, June 2012)
Mobile Research (Stockholm, June 2012)Mobile Research (Stockholm, June 2012)
Mobile Research (Stockholm, June 2012)
 
Key Trends in Market Research (Stockholm, June 2012)
Key Trends in Market Research (Stockholm, June 2012)Key Trends in Market Research (Stockholm, June 2012)
Key Trends in Market Research (Stockholm, June 2012)
 
Driving Your Brand with Research Communities (Stockholm June 2012)
Driving Your Brand with Research Communities (Stockholm June 2012)Driving Your Brand with Research Communities (Stockholm June 2012)
Driving Your Brand with Research Communities (Stockholm June 2012)
 
What Clients & Customers Want
What Clients & Customers WantWhat Clients & Customers Want
What Clients & Customers Want
 
Live Forum Results
Live Forum ResultsLive Forum Results
Live Forum Results
 
Keeping Customer Conversations Real
Keeping Customer Conversations RealKeeping Customer Conversations Real
Keeping Customer Conversations Real
 
Customer experience management
Customer experience managementCustomer experience management
Customer experience management
 
Selling Panels Internally - Australia Post
Selling Panels Internally - Australia PostSelling Panels Internally - Australia Post
Selling Panels Internally - Australia Post
 
Young Australians
Young AustraliansYoung Australians
Young Australians
 
Why Listen ESOMAR
Why Listen ESOMARWhy Listen ESOMAR
Why Listen ESOMAR
 
What Can Marketing Learn from Vanilla Ice
What Can Marketing Learn from Vanilla IceWhat Can Marketing Learn from Vanilla Ice
What Can Marketing Learn from Vanilla Ice
 
The Future of Research
The Future of Research The Future of Research
The Future of Research
 

Does Sample Source Matter? [Part 2]

  • 1. Does the Sample Source Matter? Wave 2 of a comparison of leading sample sources Sept 07, 2012
  • 2. Sample is the soil and water of research. The quality of it determines the value of what you produce.
  • 3. We compared other sample sources that VC regularly uses We conducted a five minute survey April 12, 2012 with a sample of about 500 people each, using sample from Vision Critical’s Springboard America, and four of the leading sample providers that we have anonymised as “Panel B”, “River C”, “Panel D” and “River E”. The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI)
  • 4. Methodology cont. The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI), and looked at drop-out rates, straight lining, and the use of “don’t know”.
  • 5. Sample Quality: Dropout rate during the survey Drop-outs as a percent of all who started the survey Nov-11 Apr-12 20% 19% 18% 16% 16% 14% 12% 10% 10% 8% 9% 8% 7% 6% 5% 5% 5% 4% 4% 4% 2% 2% 0 0 0% SBA - SBA - Panel B Lightspeed Panel C SSI Research Panel D River United Panel E Targeted Routed Now Sample
  • 6. Data quality: straight-lining 10% 9% 7.9% 7.8% 8% 7% 6.5% 6% 5.5% 5.6% 5% 4.8% 3.9% 4% 3.1% 3% 2.7% 2.1% 2% NA 1% 0.7% 0.5% 0% SBA Panel B SBA Routed Lightspeed Panel C SSI Panel D Research SB River Panel E uSamp Now
  • 7. Data quality: use of “don’t know” Nov-11 Apr-12 10% 9% 8% 7% 6% 6% 6% 6% 5% 5% 4% 4% 4% 4% 4% 4% 4% 4% 3% 3% 2% 1% 0 0 0% SBA - SBA - Panel B Lightspeed Panel C SSI Research SB River Panel E Panel D uSamp Targeted Routed Now Average percent who choose the option "don’t know", "can't choose", "not sure", "prefer not to say".
  • 8. Differences between SBA and other panels in how people answered all questions 10% 9% 8% 7% 7% 6% Expected 5% 4% by chance 4% 4% 3% 3% 2% 2% 2% 1% 0% Panel B SBA Routed Lightspeed Panel C SSI Panel D Research Panel E uSamp River Now Count of the number of statistically significant differences (95% CI) as a percent of the total number of questions
  • 9. Differences between SBA and other panels in how people answered all questions While, for the most part, these differences appear within the realm of random, there were some differences that seem to be very real. For example, in both April and November Panel B respondents were less likely to purchase snack products and had lower purchase intent scores for a snack concept.
  • 10. Differences between panels in how people answered the purchase intent question Nov-11 Apr-12 50% 46% 44% 44% 45% 41% 40% 39% 40% 34% 33% 35% 31% 30% 30% 29% 25% 20% 20% 15% 10% 5% 0 0 0% SBA - SBA - Lightspeed Panel C Research Panel B SSI Panel D River Panel E United Targeted Routed Now Sample Which statement below best describes how likely you would be purchase this snack if it were available today?—top 2 box: definitely and probably
  • 11. Conclusions Sample matters. A lot. Springboard America has fewer drop-outs, notably better data quality, and is “right down the middle” when it comes to comparison to the other panels on the findings, unlike some panels which show real variation—particularly on the concept test. SBA also shows consistency in how people answered, as illustrated by reaction to the concept.
  • 12. Conclusions The same cannot be said for the other panels, however. They showed considerable variation between time 1 and time 2 in how they reacted to the snack concept, which was a non-trendy concept. This may be a concern for tracking. I will dig deeper into the comparison between waves in another report.