1. Better Surveys through Social Media
May 20, 2011

2. Gartner’s Hype Cycle
2

3. MR Firms’ Attitudes towards SMR
63% of MR firms are experimenting with
SMR or considering it for the future
Only 17% have conducted any social media
research to date
For 65% of those who are conducting such
research, it’s less than 5% of their revenues
66% consider SMR to be neither qual nor
quant but something new
Source: meaning ltd. 2010 survey
3

4. Social Media Research at the Peak of Hype
Social
media
research
Text analytics
MROCs Phone Focus Mail
surveys groups surveys
Sentiment
Mobile analysis
research Online access panels
Online surveys
Research
games
4

5. Social Media Exemplified
“I need to pee.”
“I peed.”
“This is where I pee.”
“Why am I peeing?”
“Look at this pee!”
“I’m good at peeing.”
Concept courtesy of epicponyz.com
5

6. Social Links Weak on Twitter; Like Subscriptions
Facebook Twitter
Researcher Researcher
Member with
Member Member Member Protected
Tweets
Updates Updates
Member Member
often typically
inaccessible accessible
6

7. Survey Research Steps, with Social Media
Study Design
Use social media to frame
hypothesis
7

8. Market Research in the Groundswell
Groundswell: “A social trend in which people
use technologies to get the things they need
from each other, rather than from traditional
institutions like corporations.”
Josh Bernoff’s First Rule of Social-Media Market
Research
“Social applications are typically better at generating
hypotheses than at testing them.”
“Projecting to the total population is a perilous thing but this
is an excellent place to hear what is going on.”
8

9. Framing Hypothesis
Case Study: Why are Some Dieters Uninterested
in the Atkins Diet?
Love of pasta and bread
9

10. Framing Hypothesis
Case Study: Why are Some Dieters Uninterested
in the Atkins Diet?
Love of pasta and bread
Distaste for red meat
Concerns about heart disease
Belief it’s an unproven fad
10

11. Twitter Search Tips
http://search.twitter.com/advanced
Limited to past 30 days
Can constrain geographically
Can search on basic sentiments
(emoticons)
No search counts provided;
compare age of oldest tweet for
relative measure
11

12. Tortoise or the Hare: Social Media Sampling
Twitter searches are easiest, but only
represent 7-8% of online users
Keyword searches vs. brand searches
Errors of inclusion: Target search returns “likely
target”, “target market”, “above target”, etc.
Errors of exclusion: McDonalds search excludes
“McD”, “Mickey D”, “MickeyDee”, “Golden Arches”
Keyword searches vs. constructs
“brown” vs. “Charlie Brown”, “Chris Brown”, etc.
Source: Conversition
12

13. Query Bias in Social Media Research
Sentiment Varies by Search Term
1.00
0.67
0.33
0.33 0.27 0.24
0.14 0.09
0.00
Civ 5 #Civ5 Civilization 5 Civilization V Civ V #CivV
-0.33 -0.27
-0.67
-1.00
13

14. Formality of Search Terms Affects Type of Results
25%
#Civ5
20%
Conversational
Civ 5
#CivV
15%
Civ V
10%
Civilization 5
5%
Civilization V
0%
0% 20% 40% 60% 80% 100%
Newsy
14

15. Survey Research Steps, with Social Media
Sample Selection
Identify potential respondents
15

16. Twitter Sample Sourcing
Twitter reaches that part of
your audience on Twitter
Will differ in key ways from
non-Twitter users Qualitative Research
More likely to be early adopter of
technology Representative
More likely to be “Creator” persona
of Web usage
In fact, probably not
representative of your users on
Twitter either!
16

17. Demographics of Twitter Users
Source: Quantcast, May 2010
17

18. Where To Find Them
Facebook Fan Pages
Twitter Hashtags
#MRX
#NewMR
LinkedIn Discussion Groups
18

19. Primary Reasons for Returning iPad, for 50 Twitter Users
Returned Considering
Upgrade to 3G
Poor Value
Incompatibilities
Other
No Reason Given
19

20. Going Wrong Listening to Customers through Qual
“We didn't get the passion this loyal group of consumers have.
That wasn't something that came out in the research.” –
Tropicana
“We have decided to return to our previous terms of use while
we resolve the issues that people have raised.” – Facebook
“The testing we've done has been incredibly positive.” – Syfy
“We have heard your concerns about the ad that was featured on
our website.” – Motrin
http://bit.ly/95ZfxW
20

21. Survey Research Steps, with Social Media
Questionnaire Design
1. Learn language of respondents
2. Develop choice lists
21

22. Need to Learn Language of Respondents
Typical Questionnaires
Written from sponsoring
organization’s perspective
Uses industry jargon
Makes subtle distinctions
Result for Respondent
Less accurate answers
Fewer questions answered
Higher abandonment rate
22

23. MP3 Player Size
Sponsor’s Language Customer’s Language
What size of storage do you How many songs do you want to
want in an MP3 player? keep on your music player?
o 1GB o 500 songs
o 2GB
o 1,000 songs
o 4GB
o 2,000 songs
o 8GB
o 4,000 songs
o 16GB
o 8,000 songs
23

24. Closed-End Questions Great to Extrapolate From…
Top 3 Favorite Colors As Determined by Questions
with Different Numbers of Choices
15
Except when you get the
choice list wrong!
10
5
0
A B C D E F Correct Ranking
G H
24

25. Develop Choice Lists for Closed-Ended Questions
In the past, why have you thought the Atkins Diet
was not right for you?
(Choose all that apply.)
 Love pasta too much to go without
 Love bread too much to go without
 Love sweets too much to go without
 Try to avoid eating red meat
 Worried about health problems that Atkins
might cause
 Other
_________________________________
25

26. Survey Research Steps, with Social Media
Fielding the Survey
Inviting respondents by SM
26

27. Invite Twitter Users to Your Survey
27

28. But Recruiting Can be Representative of Behavior…
Source: Burke, Inc.
28

29. Survey Research Steps, with Social Media
Analysis
Add tweets for color
commentary
29

30. Quantitative Questionnaire Results Lack Detail
Representative of U.S. adults who watched the 9-9-09 Obama
healthcare address
No qualitative insights into why viewers reported this
30

31. Comments Provide Narrative to the Numbers
Positive Negative
“President Obama Rocked My World Last “Had to laugh at Obamas speech last
Night Greatest Speech Ever by any night! Bi Partiscian negotiations on the
President ever in History of USA!” healthcare bill? NOT!!!!!!”
“hi good evening! Have u watched “Watching Obama give a speech is like
obama's speech on health care? It was a watching someone watch a tennis match
good one, as expected. Obama is one as his head just bobs back and from
eloquent man.” teleprompter to teleprompter.”
“I thought President's Obama's speech “Dennis Miller had it right - if POTUS
on healthcare was much needed.” cannot start a speech on time, do you
trust the govt to deliver healthcare on
time & under budget?”
31

32. “Word Cloud” Highlights Unexpected Theme:
Created with Wordle
32

33. Tweets Must Be Extensively Filtered
False
News hits
Links Shills
RTs 25.4% of Spam
tweets are
replies - HP
Conversation Quality Content
33

34. Tweets as Verbatim Responses
Different types of tweets Practical barriers
Shared links Some brands have too little
PR feeds public discussion, esp. B2B
RTs (Retweets) Some brands are too generic
Replies (e.g., Sedona)
Small talk Many tweets about public
brands are news links
34

35. Social Media Users as Exceptions
They take a public position
That position may often be notable as an outlier
Position may or may not be true
May neither be credible nor reliable
Source: Bill Neal, Market Researcher
35

36. Survey Research Steps, with Social Media
Study Design: Use social media to frame hypothesis
Sample Selection: Identify potential respondents
Questionnaire Design: Learn language of respondents;
develop choice lists
Fielding the Survey: Invite respondents by Twitter
Analysis: Add tweets for color commentary
36

37. So Long!
Thank you for 17.8 great years!
Jeffrey Henning
@jhenning
jeffrey@henning.com
+1 617-694-4012
37

38. Appendix

39. Social Media Defined
social media n. information formats produced from activities that
integrate technology, social interaction, and the construction of
words, pictures, videos, and audio.
39

40. Research 1.0 vs. Research 2.0
Social
Media
Research
Research 2.0
Focus
v MROC
Group
Research 1.0
Survey Interview Ethnography
Directed Moderated Undirected
40

41. Framing Hypothesis
Case Study: Why are Some Dieters Uninterested
in the Atkins Diet?
Love of pasta and bread
Distaste for red meat
41

42. Framing Hypothesis
Case Study: Why are Some Dieters Uninterested
in the Atkins Diet?
Love of pasta and bread
Distaste for red meat
Concerns about heart disease
42

43. Social Media is a Rich Source of Qualitative Insights
Qualitative Research
Twittered- Random-
Sampled Sample
Surveys Representative Research Surveys
43

44. Most Twitter Users Don’t Participate Much
85 followers, 80 friends
1 tweet a day (255 tweets over 7 months)
68% active in last 30 days
Qualitative Research
- HP Lab statistics
Most sampled tweets therefore literally Representative
represent the “vocal minority”
44

45. Twitter Demographics
Source: Dan Zambonini, http://boxuk.com/
45

46. To Maximize Chance of Being ReTweeted…
46

47. And Say “Please”!
47

48. Use Tweets to Trigger Surveys
140 characters of
feedback doesn’t
provide context
Use tweets to trigger
survey invitations
Best handled by a
human rather than
automation
5 McDonalds
complaints out of 200
tweets
48

49. Post Questions to LinkedIn
49

50. Twitter & Social Media One Source Among Many
Customer Customer Past
Tweets
Emails Searches Surveys
50

51. Copyright Status of Tweets
Most Tweets too short and too lacking in
“creativity” to qualify for copyright ownership,
according to EFF
“Short phrases … are not copyrightable, even if
such expressions are novel, distinctive, or lend
themselves to a play on words.” - Compendium of
Copyright Office Practices
“When you're talking about 140 characters, you're
going into a range where there are probably a lot
of things that are copyrightable.” - Jeffrey Johnson,
©
Attorney, Pryor Cashmen
51

52. Social Media Research is Not Ethnography
Ethnography can involve interviews
with singular participants or groups
Ethnography can be participatory like
mystery shopping
Ethnography can be purely
observational using video and audio
recording
Ethnography is about experiencing a
social group and their behaviors and
attitudes from the inside
Source: Chris Bailey
52

53. Value of Social Media in Survey Research Process
Study Design
Sample Selection
Questionnaire Design
Fielding the Survey
Analysis
Not at all Very
Sample size: 1 valuable valuable
53

54. LIRM: Listen, Interpret, React, Monitor
Listen. Capture everything.
Interpret. Put comments
into context and look for
Monitor Listen patterns.
React. Respond, change and
adapt.
React Interpret
Monitor. Track performance.
Source: Bruce Temkin,
Experience Matters blog
54

55. LIRM: Listen, Interpret, React, Monitor
“All components of a VoC
program need clear
mechanisms for capturing
everything from customers’
perspective on specific
interactions to their
satisfaction with the
company.” – Bruce Temkin
55

56. Social Media is Free to Access, Costly to Analyze
Yes, you can download recent comments for
free
However, expect to spend significant
amounts of time on data cleaning
$
56

57. Social Media Research Concerns & Caveats
Little content for brands with low market penetration
“A good insight comes from triangulation of qualitative and
quantitative data to explain the contradictions and
inconsistencies between them.” – Ben Foster
Artifice vs. Ethnography
Social media researcher: “This is a non-interventional way to listen to
the authentic Voice of the Customer. Completely unartificial!”
Survey researcher: “No artifice is taking a public
stance, deciding what to tweet when?! Not natural discourse
at all!”
57

58. Social Media Strike Out
Not every brand’s customers are talking about
them online
Not everything about a brand is talked about,
even for those brands people do talk about
Not everything talked about has enough context
to be useful
Not everything talked about is representative of
customers
58

59. J.D. Power’s Blog Research
1. Brand monitoring, esp. of
competitors
2. Trend analysis
3. Customer information – cross-post
analysis
4. Unmet needs – frustrations, wishes,
ideas
Source: Janet Eden-Harris, VP of Web Intelligence,
interviewed by Adam Sultan, Marketing Sherpa
59

60. LIRM: Listen, Interpret, React, Monitor
“Customer feedback needs
to be examined by asking
questions like ‘Is the issue
we’ve uncovered isolated
or systemic?’ And ‘Where
in our organization can we
best deal with this
situation?’” – Bruce Temkin
60

61. Blog & Social Network Content Analysis
Gartner sees marginal to no ROI from its
clients’ investments in blog content analysis
“It’s eye candy: management likes to see it.”
Gartner sees no strategic value in connecting
customers to their social networks
In terms of concrete ROI, instead learn the
language of the Voice of the Customer:
Ask more open-ended questions in surveys and
use text mining on them
Use text mining on the keywords customers use
in your web site’s search box
Source: Gartner, “Analytics to Action: Key
Analyses for Customer-Centric Decisions”
61

62. Building Models with Predictive Validity
Weekend box-office receipts – Predicting the Future with Social
Media predicted opening box office receipts of U.S. movies with
97.3% accuracy
Consumer confidence – From Tweets to Polls found a 73.1%
correlation between “job” tweets & Gallup consumer confidence
Presidential approval – Found a 72.5% correlation between
tweets about Obama and presidential job approval polls
Congressional elections – The Daily Beast and WiseWindow built
a social media model that predicted 97% of Senate races and 87%
of the House races that it tracked
62

63. Word of Mouth Models
Do gains in share of word of mouth – or “word of mouse” –
precede gains in actual market share?
Build a Twitter tracker of a brand and its major competitors
Good social media monitoring tools let you build chronological
searches into historic social media data
Does your model predict actual gains?
Bonus
Slide
63

64. LIRM: Listen, Interpret, React, Monitor
“For each component of a
VoC program, firms need
explicit processes for
making changes
throughout the
organization – based on
what is learned from
customer insights.”
– Bruce Temkin
64

65. Social Media Monitoring Tools vary by Departmental Need
Social Media Brand Service
Research Monitoring Monitoring
Department Market Marketing Customer
Research Service
Mission Understand Influence Serve
Reports Ad Hoc Trends & Priorities
Campaigns
Segmentation Demographics Influencers Individuals
No interaction Extensive interaction
65

66. LIRM: Listen, Interpret, React, Monitor
“As with any well-run
corporate program, each
component of a VoC
program needs automatic
feedback loops that track
work plans and results.”
– Bruce Temkin
66

67. Monitoring Tools Still Immature
Many false hits for keywords, especially generic terms
Spam often returned
Sentiment analyses only 70% accurate
Difficult to screen for content-specific content
Influence analyses often use old data and “bogus” algorithms
Time consuming
Source: Asi Sharabi
67

68. Types of Online Research Communities
Closed
OLFG BBFG Insight Community
Community Panel
Idea Jam
Open
Idea Voting
Temporary Permanent
68

69. Social Media Research
“[Social media] is neither a be-all and
end-all nor a fad.” – Ben Smithee,
founder, Spych Market Analytics
Apply it where appropriate in your
current projects
Experiment and play!
Share the results where possible
69

70. What are the Rules?
Respect/ignore expectations of privacy.
Respect privacy: “In a public social media site, they don't want us
there. People don't go to social media sites to talk to companies.
They go to talk to their friends.” – Jan Trent, Wendy’s
It’s not private: “I would say that you should quote anything taken
from a blog verbatim and attribute it. Reading content of blogs is
desk research amongst publicly published material, not primary
research.” - Audrey Anand, Listengroup
70

71. What are the Rules?
Respect/ignore expectations of privacy.
Seek/don't seek permission to share consumer
comments in your research.
“I always ask for the permit from the blogger. Only once or twice I
was asked not to use data.” - Mirjana Necak, PR manager
“Would it ever be possible to really get informed consent without
affecting the quality of the learning, especially if you're asking over
the net?” - Letesia Gibson, State of Play
71

72. What are the Rules?
Respect/ignore expectations of privacy.
Seek/don't seek permission to share consumer
comments in your research.
Cite/obscure identities of commenters.
“I would take all the comments I wanted (maybe change word order
a bit) and then apply usual criteria to hide contact info. I would offer
categories of participants.” - Kathy Flament, Flament Associates
“To mask its source and treat it as if it was from a ‘live respondent’
would be a kind of plagiarism of the original author and to some
degree dishonest to any client reading it.” - Audrey Anand
72

73. What are the Rules?
…Cite/obscure identities of commenters.
Engage/don't engage with commenters.
Bloggers merit a reciprocal relationship. Engage & connect in order
to understand context of what is being shared, meeting bloggers half
way. - paraphrase of Josephine Hansom, with GfK NOP
“In the course of carrying out social media research, someone replies
to someone whose data just happens to appears in the research data
set. The person didn't ask to participate and they didn't respond to a
question. For me, this is in direct violation of the Prime Directive.” -
Annie Pettit, Conversition
73

74. What are the Rules?
Respect/ignore expectations of privacy.
Seek/don't seek permission to share consumer
comments in your research.
Cite/obscure identities of commenters.
Engage/don't engage with commenters.
74

75. What do Researchers Think?
Respect (60%)/ignore (26%) expectations of privacy
Cite (26%)/obscure (53%) identities of commenters
Seek (50%)/don't seek (33%) permission
Engage (59%)/don't engage (24%) with commenters
Source: 226 attendees to AMA webinar,
"Excuse Me! We're Having a Conversation Here!"
75

76. Look at Ourselves in the Mirror
76

77. What do Consumers Think?
77

78. Survey
Methodology Limitations
Online survey of 426 U.S. Convenience sample
panel members of Western Instrument bias
Wats People willing to answer
surveys may have a more
May, 2010 positive view of MR
No weighting Facebook privacy concerns
may have heightened
respondent concerns
78

79. AAPOR Standard Disclosure Form
BASIC DISCLOSURE ELEMENTS DETAILS
Survey sponsor Vovici
Data collection supplier Western Wats Opinion Outpost
Population represented Online adults in the United States
Sample size 426
Mode of data collection Web survey
Type of sample Non-probability
Start/end dates of data collection May 7, 2010 to May 10, 2010
Margin of sampling error Not applicable
Are the data weighted? No
Response rate (and how calculated) 8.5% (AAPOR RR2)
Contact for question text and more Jeffrey Henning
information jhenning@vovici.com
79

80. What Do Customers Like About Social Media Research?
“It provides marketing researchers with an understanding of the
real life reaction to a given product. That, in turn, leads to a
better product for consumers.”
“They can provide a better product/service based on what the
actual customers are saying, rather than the focus group they put
together.”
“It makes sense. It would be the most raw and most likely
emotionally real response that could be given.”
“I like that when studying comments, those comments are
actually put into action to please their customers.”
80

81. What Do Consumers Dislike?
“A little too close to ‘Big Brother is watching’. But you should
know enough to never post if you don't want others to read it.
81

82. What Do Consumers Dislike?
“A little too close to ‘Big Brother is watching’. But you should
know enough to never post if you don't want others to read it.”
“I don't like that they invade our privacy in the first place. If we
wanted them to know, we would contact them.”
“I like nothing about this. If it is social media, it should be social
media and not for research.”
“It is skeezy and lecherous stealing data that they would
otherwise have to pay for.”
“I feel that their opinion is a waste of time and that this type of
job should be eliminated.”
82

83. Concern for Privacy
How concerned are you about your privacy on the
Internet?
100%
90%
80%
70%
60%
50%
40% 33%
30% 23% 24%
20% 16%
10% 5%
0%
Not at all Slightly Moderately Very Extremely
concerned concerned concerned concerned concerned
n = 426
83

84. Concerned for Privacy, Really?
35% 23% 33% 24% 16%
5%
-15%
Not at all concerned Slightly concerned Moderately concerned Very concerned Extremely concerned
People concerned about Internet privacy don't engage in fewer online
activities.
Concerned users don't vary in which social networks they use or how many
social networks they use.
Concerned users don't comment less often on web sites than unconcerned
users do.
Concerned users aren't less likely to use photos of themselves or their real
name or email address when posting comments.
n = 426
84

85. “Um, We Didn’t Know You Were Listening”
As far as you know, which types of organizations monitor
and analyze public Internet discussions?
"Some organizations" 69%
Law enforcement agencies 52%
Market researchers 45%
Social networks 42%
Search engines 37%
Marketing departments 35%
Online stores 25%
Service departments 15%
0% 20% 40% 60% 80% 100%
n = 426
85

86. Framing the Discussion
“Market researchers, service departments and marketing
departments often analyze comments made on the Internet to
understand consumer attitudes and satisfaction with products and
services. Researchers want to understand what people think; service
departments want to follow up to improve satisfaction; marketing
departments may want to make you aware of competing products.
Assume you make a comment about a product or service on the
Internet…”
Social Media
Service Brand
Market
Monitoring Monitoring
Research
86

87. Privacy
“Can violate privacy if we didn't know it could happen.”
“It interferes with my understanding of privacy.”
“When they can identify me, follow my activities about
everything I do on the internet and target me specifically, then
it's gets a little scary and I worry about my privacy.”
“Make sure to keep privacy guidelines to those who don't want
you to use their words.”
“It often times invades the privacy of the individuals they are
tracking.”
87

88. Perceptions of Privacy: Depends on your Analogy
Market Researchers
Social media conversations
are secondary research
Matter of public record
88

89. Perceptions of Privacy: Depends on your Analogy
Social media conversations are conversations…stop eavesdropping!
Conversations in public places aren’t public
Researchers aren’t the intended audience
89

90. Sharing Consumer Comments
Would you prefer researchers…?
Ask for permission before reporting your comments 85%
“They sometimes take things out of
context and should always contact
the person before using their
Use your comments without contacting you 4% comments.”
“The idea that comments that I
make could be taken out of context
to support a product or position
Don’t care 11% that I don't personally support.”
0% 20% 40% 60% 80% 100%
n = 419
90

91. Identity when Commenting
“Screen name” or other invented name 70%
Email address 45%
Real name 40%
Picture 23% How they
Anonymously identify
22%
themselves
Photo portrait 15%
Geolocation 5%
None of the above 1%
Identify you by your real name (if given) 7%
Identify you by your screen name (if… 20% How they want
Describe you by your demographics 24% researchers to
Not identify you at all 43% identify them
Don’t care 7%
0% 20% 40% 60% 80% 100%
n = 419
91

92. Contact is Conditional…On Who Comments Where
Which Groups Is It Acceptable to Contact You
The organization you comment on
59%
Company representatives who want to address your comment
40%
Market researchers with the organization you comment on
30%
Other members of the community or site you posted to
25%
Marketers who want to make you aware of a competing product
18%
Competitors to the organization you comment on
17%
Independent market researchers
15%
None of the above
23%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
n = 422
92

93. Contact is Conditional…On Who Comments Where
Which Sites May Groups Contact You Through
(Base = Users of Such Sites)
An online store
33%
Your personal Twitter page
32%
Your online journal or blog
32%
Your personal MySpace page
22%
Your personal LinkedIn page
21%
An online news group, website, blog or photo site
21%
A social networking site
21%
Your personal Facebook page
19%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
n = 422 , 66 , 63 , 89 , 42 , 422 , 284 , 277
93

94. Consumer-Driven Rules of Social Media Market Research
Respect expectations of privacy.
Seek permission to share consumer comments in your
research.
Obscure identities of commenters.
Don't engage with commenters.
94

95. Vovici Customers
Financial Services
“We have 14,000 surveys going out at all times, and the
ability to personalize them has been huge for our response
rates which reach up to 80% completion rate.”
IT / Telecommunications Healthcare
Robert Kimmel, Leadership Development Center
Transportation Manufacturing
“Vovici is the solution Oracle counts on to build a world-
class VOC program. Since using Vovici, we improved our
response rates by 25% resulting in over 200,000 responses
this year alone.”
Jeremy Whyte, Dir. Customer Feedback & Reporting, Oracle
Educational / Government
Government of
Canada
“Vovici’s real-time alerts and email triggers allow us to run
our business around customer responsiveness. Now it
Consumer / Retail
takes an employee a half a day to capture the insights it
would have taken us 6 months to get.”
Marsha Jones, Manager, Lead Acquisition and Solutions
95