1) The document discusses the development of a new social media metric called the Semantic Engagement Index (SEI) which analyzes conversations on social media to understand brand sentiment. 2) Case studies show the SEI has strong correlations to sales for hospitality and food & beverage clients, and is a better predictor than traditional sentiment analysis. 3) Incorporating the SEI into marketing response models provides insights on how to effectively monitor and manage social media conversations to improve business performance.

1. 2011 Market Research in a Mobile World
July 2011
Michael Wolfe
1

2. THE HOLY GRAIL?
• Don’t believe it for a minute, it doesn’t exist!
• But some metrics are better than others.
• There are many social media metrics available today.
– Many simply count words with rudimentary sentiment differentiators
– Others based on Influence, klout scores, number of followers, etc
– Still others based on language and Linguistics science
• We have developed a special metric called the Semantic
Engagement Indexsm(or SEIsm).
– It is based on the principle that conversations on social media are
more than just words or what is said, but how it is said and the context
of the conversation. Semantics matter.
– We directly compare our SEI with other social media sentiment
metrics from a leading social media aggregator company.

3. SEI AND SOCIAL SENTIMENT METRICS
• The comparison is with our Semantic Engagement Indexsm and a Social Media
Sentiment indicator from a leading social media data company. As you can see,
the Sentiment Indicator has poor and somewhat contradictory correlations with
our selected company’s sales. The SEIsm metric, however, shows strong and robust
correlation and logically consistent to the positive and negative forms.
Correlations to Sales

4. SEMANTIC ENGAGEMENT INDEX:
NUTS & BOLTS
• This metric is based on an algorithm devised with the assistance of
Boyd Davis, Professor of Applied Linguistics at the University of
North Carolina and a vendor partner named Linguistics Insights.
The process we go through here involves four steps and it leverages
linguistics science heavily.
1. First we secure large quantities of social media conversations
filtered for topics on the specific brand of interest. In the case of
the hospitality client, this was focused on online review sites
pertaining to hotels, resorts and cruise lines.
2. Next we parse these conversations data into positive and negative
toned conversations.
3. Next we apply our proprietary algorithm which quantitatively
“scores” each of the positive and negative groups along the two
dimensions of “emotional affect” and “personalization”. This
scoring algorithm applies the science & rules of Linguistics.
4. Finally, we time-code each conversation into topics and aggregate
into a time series metric.

5. CASE STUDIES
• To fully leverage the SEIsm for our clients, the task is to understand
its impact on their business.
• To do this, we do exploratory analysis to see how relevant the
metric is to the customer demand of a number of clients.
• Then we utilize the SEI within a full marketing response (aka, mix)
model in order to not only understand its impact on the business,
but also how it interacts with and is affected by direct marketing.
• First, however, we compare the customer SEIsm metric to 2 clients’
customer demand over time.
– One client is a food & beverage retailer, while the other is in the
hospitality industry
– The specific SEIsm metric we use here, and in our model, is the SEIsm
ratio of positive to negative tonality conversations.

6. TOTAL HOSPITALITY BOOKINGS
• For our hospitality client, the key focus of our SEIsm metric was “online” review
sites for hotels, resorts and cruise lines. This makes the metric a proxy for
customer satisfaction. As shown, the correlation is a very robust +76%!
140 160
120 140
120
SEI Positive/Negative Index
100
100
Bookings Index
80
80
60
60
40
40
20 20
- -
1/7/2008 1/7/2009 1/7/2010
Bookings.Index SEI Positive/Negative Ratio

7. TOTAL FOOD & BEVERAGE SALES
130 400
125
350
120
300
Retail Sales Index
115
SEI Positive/Negative Index
250
110
105 200
Correlation of Sales to
100 SEI Metric is 87%
150
(Total 156 Weeks)
95
100
90
50
85
80 0
07/09/07 08/04/08 08/31/09
TOTAL Retail.Sales SEI Positive/NegativeRatio
7

8. SEI AND MARKETING RESPONSE MODELS
• To fully leverage the SEIsm for our clients, the task is to
understand its impact on their business.
• By incorporating SEIsm metrics into marketing response
(aka, mix) models, we can:
– Come to a better & more precise understanding of how
social media buzz affects a client’s business performance
– Understand the impact and interactions of the client’s
marketing and media as it affects social media
conversations about their brands.
– Provide strategic guidance as to the most effective ways
for monitoring and managing social media conversations
on brands

9. THE IMPACT OF SOCIAL MEDIA TONALITY
• A key insight we uncovered across clients is the difference between “positive” and
“negative” conversations about a brand. In absolute terms, the negative-toned
conversation has a significantly greater net impact. It is actually better for a firm
to try to manage a reduction in negative toned conversations than to increase
positive ones.
Total Retail Sales Elasticity of Response Total Retail Impact of 90% Change
(change in Sales versus change in Tone components of "Engagement") (assumes 100% Decrease in Negative is not realistic)
20% 20%
10% 15%
positive
Sales Impact
Sales Impact
0%
10%
negative +16.5%
-10%
5%
-20% +4.4%
-100% -50% 0% 50% 100%
0%
Change in "Engagement" by Tone Increase Positive Decrease Negative
9

10. THE IMPACT OF SEI, SALES AND MARKETING
• We also learned that a there is both a direct and an indirect effect from client media
& marketing. As you can see, the impact of Semantic Engagement is quite large.
The “indirect” effect of marketing is due to the impact on the SEI metric, itself.
Overall, this indirect effect is shown to greatly enhance the impact of the client’s
marketing and significantly improves marketing spend ROI.
Total Retail Sales Contribution
net driven by 6%
2%
marketing
8% +6% +13% = 27% Sub-model
6%
13%
37%
Engagement
55%
18%
Base Sales Direct Alpha Brand Media & Mktg Direct Social Media Mktg Blended Media Other.Media Other Base
10

11. PREDICTING POSITIVE CONSUMER ENGAGMENT (SEI)
We apply CART (Classification & Regression Trees) to score subject groups to determine what
conversational topics & issues are driving consumer engagement.
Brnd&Place
>14.29 The most important positive
Brnd&Place
=461 drivers are:
0.3%
>9.63 1. The Brand & Place
=320 Brnd&Place
2.1% <14.29 2. For Meeting People
Brnd&Place
>5.75 Brnd&Place
=298 3. The Beverage
1.8%
=219 <9.63 4. The Store Atmosphere
9.9% =192
7.8%
POS SEI 3.76
=100
Meeting
Brnd&Place People>5.84
<5.75 =229
3.6% Beverage>6.29
=87 =255 Brnd&Place>3.
90.1% Meeting 7.9% 64 = 121 Atmosphere
People<5.84 7.0% >5.25
=81 =209.6
Beverage<6.29
86.5% 1.3%
=79 Brnd&Place<3.
78.6% 64 = 72 Atmosphere
The tree starts with an average SEI score of 100; and each level 71.6% <5.25
indicates a higher or lower SEI based on an SEI score for a topic.. =70
The percent represents the percent of the sample in each segment. 70.3%

12. THE MOBILE RESEARCH ANGLE?
• Certain social media data companies can now
collect social media data for specific
geographies and demographics.
– www.gabacus.com
– www.decooda.com
• Social media as a tool for prediction:
– Bernardo Huberman, Hewlett Packard
• Predicting elections
• Test market analysis
– www.bzzagent.com

13. LESSONS LEARNED
• Through a number of case studies, BBDO’s “the Worth” has attempted to
push back the frontier on social media metrics and understanding
• By linking a metric of “Semantic Engagement” to client sales and demand,
we have shown that this approach shows great promise as a diagnostic for
understanding social media’s impact on a client’s business by including it
as an input into marketing response (aka, mix) models.
• Some of the key lessons that we have learned include:
– For a service-based hospitality client, the consumer’s rating of the quality of
the firm’s service, is in fact, the most important driver of that firm’s business.
– That negative brand conversations have a greater absolute impact on a client’s
business than corresponding positive reviews, i.e. negative word-of-mouth
travels wider & deeper than positive.
– That the direct impact of SEIsm on business is large and significant. A brand’s
marketing and advertising affects the SEI sm which in turn, affects sales.
– We learned that the value and ROI of marketing is greatly enhanced due to
the indirect effect it has on sales through its direct impact on Social Media
Engagement (SEIsm )
– That our SEIsm metric is no Holy Grail, but it shows much promise in delivering
un-matched insights on how social media conversations have a direct and
tangible impact on company performance.

14. Presented at:
Market Research in the
Mobile World
2nd International Conference | July 19 & 20, 2011 Atlanta
Organized by: Thank you to sponsors:
LinkedIn Group: Mobile MR
Upcoming Merlien Events: Merlien.org GreenBook Directory: GreenBook.org
Market Research Blog: GreenBookBlog.org New Qual Blog & Directory: NewQualitative.org
Get Inspired. Stay Informed. Cut Through the Clutter. Sign up for our free email newsletter