Social Media Measurement 101 + Day of aTONEment

SOCIAL MEDIA
MEASUREMENT 101
A Day of aTONEment
MRA ANNUAL CONFERENCE
JUNE 6, 2011

Alan Chumley
Senior Vice President
@alanchumley / @CARMA_Tweets
@ l h l @CARMA T t

1615 M Street, NW; Suite 750
Washington, DC 20036 USA Tel 1.202.842.1818
carma.com

ALAN CHUMLEY, SVP

• 14 years in the industry
• Recovering PR practitioner: ING, Bell Canada
R i PR titi ING B ll C d
• Former Director of Measurement @ Hill & Knowlton
• Masters in Communications & Culture:
• Research foci:
• media effects & uses
• reception analysis
reception analysis
• audience studies
• University‐level adjunct instructor of PR measurement
• LinkedIn early 2006
Li k dI l 2006
• Blogging / Tweeting since early 2007

carma.com 2
@alanchumley / #mra_ac

ABOUT CARMA

• Traditional & social
• Media monitoring & analysis
M di it i & l i
• Largely human‐based….on an automation journey

• In biz since 1984
• Head office here in DC
• 14 offices in 11 countries
14 offices in 11 countries

• Clients:
• PR f lk i bi
PR folk in big organizations
i ti
• Market research folks in big organizations are starting to emerge

carma.com 3

PART 1 (1.5 hrs): SOCIAL MEDIA MEASUREMENT
IN-ROOM STRETCH (5 mins)
PART 2 (30 mins): TONE ALONE
i )
PART 3: (35 mins): REAL DATA WITH MALCOM DE LEO NETBASE

carma.com

PART 1 (1.5 hrs): SOCIAL MEDIA MEASUREMENT

carma.com

ISN’T

• Not a Case Study

• Not a mention of Old Spice in sight

• Not a specific ‘how to’ / no one-size-fits all

• Doesn’t focus on a singular or few social
media platforms—looks at all holistically

carma.com


ISN’T IS

• Not a Case Study • An emerging / experimental philosophy /
approach / filter / lens through which to look
at measuring social media
• Not a mention of Old Spice in sight
• A high level connected / systems way of
• Not a specific ‘how to’ / no one-size-fits all approaching the problem

• Doesn’t focus on a singular or few social • About communities not an audience
b d
media platforms—looks at all holistically
• Beyond simple counts and sentiment

• More sociologist than publicist

• Influence potential
carma.com


Every hour more than 500,000 new
blog + micro-blog posts, status updates
micro blog posts
and comments are posted

carma.com

MEASUREMENT ACCOUNTS FOR 5 / 7 TOP SOCIAL MEDIA CHALLENGES

Why is this the case?
Gap between leap of
faith true believers and
C-suite’s need for
metrics to validate the
time/$?
$
Source: Harvard Business Review: the New Conversation: Taking Social Media from Talk to Action

Surveyed 2100 Harvard Biz Review subscribers

carma.com 9

SOCIAL MEDIA IS A GAME CHANGER… …& A SOURCE OF ANXIETY

Top Down
Command & Control
of the Message is Gone

• Transparency demanded

• Dialogue vs. Monologue

• Many-to-many not one-to-
many

• Conversation & community
NOT (strictly) coverage

• About relationships

• Relationship trust is co-
created by the
organization and its
communities of interest via
conversation

carma.com



Top Down There is no Passive
Command & Control Ready-to-Consume Audience’

• Transparency demanded • Prosumers not just consumers

• Dialogue vs. Monologue • Actively engaged in creating and
re-creating conversations and
g
• Many-to-many not one-to- communities
many
• Challenge in tracking &
• Conversation & community measuring something as
NOT (strictly) coverage nebulous as a community


created by the
conversation

carma.com



Top Down There is no Passive Disciplinary Lines
Command & Control Ready-to-Consumer ‘Audience’ are Eroding
• PA, PR, Marketing, CRM,
• Transparency demanded • Prosumers not just consumers Customer Service etc.
• Dialogue vs. Monologue • Actively engaged in creating and • Customer touch points
g
• Many-to-many not one-to- communities • Integrated cross-discipline
many re: strategy
• Conversation & community measuring something as • Integrate cross-discipline
NOT (strictly) coverage nebulous as a community re: measurement

created by the
conversation

carma.com



Top Down There is no Passive Disciplinary Lines
Command & Control Ready-to-Consumer ‘Audience’ are Eroding
• PA, PR, Marketing, CRM,
• Transparency demanded • Prosumers not just consumers Customer Service etc.
• Dialogue vs. Monologue • Actively engaged in creating and • Customer touch points
g
• Many-to-many not one-to- communities • Integrated cross-discipline
many re: strategy
• Conversation & community measuring something as • Integrate cross-discipline
NOT (strictly) coverage nebulous as a community re: measurement

C-Suite
C Suite is Intrigued but Anxious / Want Data to:
created by the
• Show why and how they should…
• Demonstrate the value of having done so…
conversation

carma.com


SIMPLE STATUS QUO

Counts:

Quick Quantity counts:
•Fans, Likes Follows, Etc.
•Fans Likes, Follows Etc
•Tantamount to reach in the offline or 1.0 world
•Reach does not equal Influence

Quality:
•Quick automated read on sentiment: but concerns re: accuracy

Rankings:

•Visibility, popularity, traffic, link love
Visibility,
•Technorati for blogs
•NOT RECOMMENDED: Twitrratr, Klout,
•BETTER: Twendz, Traackr, Twittalyzer, Twinfluence

•They rarely qualify people or content (the message)
The rarel q alif

•The message AND the medium are the measure

carma.com 14

LOT’S OF TOOLS OUT THERE

BEWARE OF THE SILOs AVOID USING (ONLY) THE NEW SHINEY FREEBIES

•Isolated / silo’d approach
•No one tool on it’s own is sufficient—not even one paid tool
it s sufficient not
•Have to connect the dots with a more relational, systems approach

Would you like your job performance
(read: raise/bonus) to be based only on a
Klout Score, for example?
carma.com


REACH SHOULD BE A RELATIVE NOT AN ABSOLUTE
& NOT ALL CONVERSATIONS ARE RELEVANT

Off topic / Irrelevant

Relevant
R l t Relevant “ ”
R l t “-”
Relevant “+”
Neutral

carma.com 16

REMEMBER: IT’S SOCIAL MEDIA

•It’s not coverage

•It’s not static
It s

•There isn’t an ‘audience’ of consumers

•Prosumers not just consumers:
•simultaneously receiving, interpreting, co-opting,
re-appropriating and creating content

•It’s more many-to-many di l
It’ t dialogue th one-to-many monologue
than t l

•So, traditional off line ways of measuring ‘coverage’ don’t cut it

•They don’t adequately account for the multi-dimensionality and dynamic
nature of a conversation and a community of interest in a network

•We need to think bigger
gg

carma.com 17

THINK BIGGER / INTEGRATE / CORRELATE

•Think Bigger / Broader Beyond PR’s Historical Silo –social Media’s Pushing us all that Way
•Think about all customer‐facing touch points
•Understand PR / Marketing / Business Metrics
g p
•Integrate Cross‐Discipline
•Report on the Critical Few
•Focus on /correlate with Outcomes
•Close the loop / connect the dots

Source: W b Sh d i k Measurement & Strategy Practice
S Weber-Shandwick M t St t P ti Source: Oliver Blanchard, S i l M di ROI
S Oli Bl h d Social Media

carma.com 18

THE MEASUREMENT MASHUP

carma.com 19

THINK BIGGER & BROADER
MOVE BEYOND ‘MOMENT-IN-TIME’ or ‘VANITY’ METRICS

Systems or N t
S t Networked
k d
Thinking:

Interdependent
relationships between
p
those in a network

Social Capital:

Connections within and
between those in a social
network as a source of
civic engagement among
a community leading to
cohesion; that has ‘value’
carma.com 20

FOCUSING ON CONTENT ANALYSIS AND NETWORK ANALYSIS
AS A WAY OF MEASURING INFLUENCE POTENTIAL

Systems or N t
S t Networked
k d
Thinking:

Interdependent
relationships between
p
those in a network

Social Capital:

Connections within and
between those in a social
network as a source of
civic engagement among
a community leading to
cohesion; that has ‘value’
carma.com 21

The 7Cs THINKING MORE LIKE A SOCIOLOGIST THAN PUBLICIST

Counting What some call search, syndication and site; stuff we can count like traffic, popularity, fans, followers,
retweets, etc.

Content Analysis that is.  Using content analysis, ideally a hybrid of some automation for volume and speed and
y g y , y y p
human analysts for context, nuance and depth to look for how your brand, messaging, topics, issues,
stakeholder voices are portrayed; the inclusiveness, accuracy and tone of the conversations.

Conversations And conversationships recognizing that social media is not strictly another communications channel
through which to get word out or reach and influence.  It is more about genuinely engaged dialogue
than a simple corporate monologue; not only a vertical, one to many conversation between an
than a simple corporate monologue; not only a vertical, one‐to‐many conversation between an
organization and stakeholders but a lateral, many‐to‐many conversation among stakeholders.

Cohesion The extent to which folks are agreeing with you, your position, your brand essences.  The extent to
which they are agreeing with each other and coalescing around and advocating a core theme, idea, or
some call to action either in support or opposition to your organization s position.
some call to action either in support or opposition to your organization’s position

Community The extent to which a core group of people with common interests are gathering and growing.
Measuring the change in community size, and volume and nature of chatter over time

Connectedness A look at how inter‐connected the highly engaged/key influencers/advocates are and how centrally
located they are in both the network and dialogue.    (See Appendix for the 5Ps of influence)

Conversion Stay tuned.  More to follow.

carma.com 22

THE 5Ps OF INFLUENCE

Popular Visible, vocal, has a substantial following, reach. In‐bound links, trackbacks, subscribers,
bookmarks, followers, friends, views, listens, saves, downloads, etc.

Polarized in tone Neutrality does little to drive influence way or the other. A clearly positive or
negative view will polarize readers/followers and is more likely to drive cohesion and
mobilize advocates and have those advocates coalesce around a core theme, idea, or call
to action.

Prolific / Relevant / Raw author contribution and # of on‐topic, related posts

Frequent
Prominent / Are they an idea starter or spreader; source or spider? They may be prolific but are they
prominent? Are they highly inter related inter connected and centrally located in the
Are they highly inter‐related, inter‐connected, and centrally located in the
Authoritative network? How engaged is this person’s following in a dialogue? How much dialogue is
there and what is its nature? Here we need to recognize, though, that authority is
contextual and topical. One might be an authority on PR measurement but not on 18th
century Russian literature.

Promoter / Loyal
/ l How many of th f ll
H f the followers/commentators active contributors advocating, endorsing,
/ t t ti t ib t d ti d i
advancing (or the opposite) your position? Are they adding links, tags. Is the nature of the
Advocate language they are using inter‐connective, expanded, clarifying, reinterpreting? RTs, digs,
fans, votes, buzz‐ups, up/downloads, shares, likes, invites, favorites, embeds. (More active
than the metrics in popularity)

carma.com 23

VARIETY OF OBJECTIVES & CAMPAIGN TYPES (NON-EXHAUSTIVE)

Objectives
Campaign Types
• Visibility
• Push (get bloggers to review / write)
• Visits
• Pull  (drive visitors on‐line)
• Volume
• CSR
• Velocity
• Product Launch
Product Launch
• Awareness
• Product Engagement
• Message delivery
• Marketing
• Link love
• Blogger Outreach
Dialogue vs Monologue
• Advocacy
• Generate dialogue / comments
• Issues Management
• Feedback
• Community building
• Using, sharing, pass‐a‐long
• Customer relations
Customer relations
• Embedding of interactive content
• Thought leadership
• Diggs, Delicious tags, stumbles, etc.
• Executive profiling
• Votes
• Etc., etc., etc.
Etc., etc., etc.
carma.com 24

SOME THINKING ON AN APPROACH
AN INDEX IS ONLY PART OF THE ANSWER

carma.com 25

COMPONENTS OF AN INDEX ?

Popularity + Relevance + Authority + Presence + Engagement + Advocacy/Loyalty + Cascade

x (+/‐) Sentiment
=
Influence Potential

• Based on an assumption that the components above are drivers of on-line influence, potential action

• They can be measured individually, but it’s important to measure them collectively as they are inter-
related

• The index is intended as a measure of potential to influence not actual real influence that is another
influence—that
research project all together.

• All components are important (but not equal) drivers of potential influence on-line
p p ( q ) p

carma.com 26

INDICATORS DEFINED

• Cascade High High •Authority
• Citation analysis / ripple index authority   authority •Of both poster and commentators
• Network analysis low AND •Idea starters or spreaders?
relevance relevance •Degree of inter connectedness / inter relatedness
•Degree of inter‐connectedness / inter‐relatedness
• Volume & velocity of message spread across the
V l & l i f d h
network •Connected but respected?
• Inter‐connectedness / inter‐relationships among: •How central is the influencer to the network?
Low High •Following: follower
• all authority relevance
• Only the influentials AND low
•RTs/1000 followers
relevance authority

• Advocacy (active)
y( )
• Are social media users advancing our cause? • Relevance
Rele ance
• % containing links • On topic?
• % active contributors • Author contribution:  how often / prolific?
• Tags added • Topical / textual / temporary

• Engagement (more coming)
Engagement (more coming) • Sentiment (more coming)
S ti t( i )
• Dialogue?  2‐way?  How much? How good?
• How many commentators?  Repeats?  Nature of • Multi‐level, multi‐author tone
the language • Post‐level, theme level, sentence level
• Network analysis:  inter‐connected/related • Gaps  in tone in originating content and resulting
discussion
• Entity Presence
Entity Presence
• Present?  Prominent?  Frequent? • Popularity (passive)
• # of + mentions
• Competitive share of original post • ‘Reach’  ?
• Competitive share of resulting discussion • Voice of the consumer stuff we can count
• Attribution recognition • Links, votes, shares, recommends, subscribers,
registrations, tweets, RTs, followers, fans, friends,
diggs, shares, likes, fav s, upvotes, buzzups,
diggs shares likes fav’s upvotes buzzups
embeds, tags, wish list saves, forwards, invites,
etc.
carma.com 27

NETWORK ANALYSIS

Source: K. Ognyanova, UCLA, 2010

How are these changing over time?

Might be part of many systems / networks?

Linking networks to networks

carma.com


(A QUICK ASIDE ON) SENTIMENT AS MARKET RESEARCH PROXY / PREDICTOR

Think here about social media monitoring and
analysis as:

•Real time situational awareness
•Listening at the point of need
•Listening close to the moment of influence

A real-time digital form of or proxy for market
research and CRM:

Mining conversations to tease out how people
‘feel’ about products/services/issues/causes
and perhaps even:

1) their intended (reported) behavior and what
motivates them to buy, (as cited in the
conversations), and

2) customer satisfaction

3)their actual behavior tracked via CRM 2.0 and
Web 2.0 analytics
y

1615 M Street, NW; Suite 750 Source: Crimson Hexagon
carma.com 29

BUT REMEMBER THAT OBJECTIVES DRIVE METRICS

Objectives
Campaign Types
• Visibility
• Push (get bloggers to review / write)
• Visits
• Pull  (drive visitors on‐line)
• Volume
• CSR
• Velocity
• Product Launch
Product Launch
• Awareness
• Product Engagement
• Message delivery
• Marketing
• Link love
• Blogger Outreach
Dialogue vs Monologue
• Advocacy
• Generate dialogue / comments
• Issues Management
• Feedback
• Community building
• Using, sharing, pass‐a‐long
• Customer relations
Customer relations
• Embedding of interactive content
• Thought leadership
• Diggs, Delicious tags, stumbles, etc.
• Executive profiling
• Votes
Etc., etc., etc.
carma.com 30

IMAGINE A DIFF’T. PYRAMID FOR DIFF’T. CAMPAIGN TYPES / OBJECTIVES

carma.com 31

AN INDEX WITH MOVABLE INDICATORS
FLEXIBLE ENOUGH TO ACCOUNT FOR DIFFERENT OBJECTIVES

Indicator Popularity Relevance Authority Brand Engagement Advocacy Cascade Sentiment
Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)

Popularity + Relevance + Authority + Presence + Engagement + Advocacy + Cascade X Sentiment = Influence Potential

• Each of the above indicators can be adjusted to account for different types of campaigns & contexts

• The weights would change. The scores don’t.

• For an organization, new to social media, perhaps they might only want to establish an on‐line presence

• So we’d dial or slide the weights of all but the popularity indicator down to zero

Has a significantly scaled back / rudimentary version of this.
This is inspired by but significantly expands on R6 thinking.
p y g y p g

carma.com 32


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 33


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 34


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 35


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 36


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 37


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 38


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 39


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 40


Presence

Score 10 10 10 10 10 10 10 10
Weight 1 2 3 4 5 6 7
(@ d f lt)
default)






p y g y p g

carma.com 41

WHAT DOES THE INDEX INDICATE?

• It’s a relative measure of the quality of social media coverage:
– One post relative to another and over time
– Many posts relative to a number of other similar clusters of postings
• It is a measure of both output and outtake, but not outcome
– Output:  how much and how good?
– Outtake:  how engaged, inter‐connected, inter‐related, volume & velocity of network /
community spread
– Outtake:  advocacy
– it does not claim to be a measure of outcome or impact
d l b f
• It’s a proxy for on‐line influence; it’s about potential to influence
– to measure actual influence, we’ve have to survey
• In a diagnostic, benchmarking sense the index can help inform an organization s social media
In a diagnostic benchmarking sense the index can help inform an organization’s social media
engagement strategy if tracked over time
• It’s tactical on the fly for mid‐campaign course correction and looking back over the long term
• It’s strategic looking forward: helps shape communications recommendations
• This data can be correlated with tangible outcomes—CARMA Connect.
• This data can be plugged into a market mix model to show causality…isolate for PR’s unique
contribution to the marketing mix.

carma.com 42

SO WHAT?
WHAT ABOUT
CONVERSION & ROI?

carma.com

WE’RE BACK TO THE PYRAMID THE TOP THIS TIME

carma.com 44

OUTCOMES: KINDS OF CONVERSION
SETTING THE STAGE FOR CORRELATION

• Conversion to a tangible like web traffic,
sales, donations, awareness, opinion,
Micro (steps toward destination) Macro (Destination) usage, brand preference, likelihood to
Usually an intermediate means to an end The end try/buy/recommend.

Non‐transactional (Engagement) Transactional
• Could also mean conversion toward any
Non‐financial Financial measurable marketing communication or
public affairs, issues, advocacy-based
Get to x clicks in a site, spend x time on
, p Sale, donation,
, ,
objective.
objective
site, download, lead generated, watch a (paid) subscription
video, created account, contacted us etc. etc.
• Here we could / should also be looking at
the extent to which the quantity and q
q y quality
y
of social media activity correlates with
these outputs.

carma.com 45

2 WEEKS IN THE SOCIAL MEDIA ECOSYSTEM…THEN…YOUR WEBSITE

Where were they? What drove them to your site?
What did they do once on your site? What are they ‘worth’ to you?

carma.com 46

SOCIAL MEDIA PERFORMANCE DATA + WEB ANALYTICS

Export Data to Your Web Analytics
•#of posts by date
•# of posts with key message included
# of posts with key message included
•# of post with brand mentioned X or more times
•# posts X average favorability by date
•# posts X average favorability X reach by date
p g y y

Or

Integrate web analytics into
your social media
monitoring platform

carma.com 47

LEVELS OF LINKING OUTPUT TO OUTCOMES

Level of Linkage Notes
Just graph ‘er up Plot # of articles vs. •No indication of
web traffic on a graph relationship other
than lines that
might trend
together

•Could be a fluke

Source: Olivier Blanchard, Social Media ROI

carma.com 48


than lines that
might trend
together

•Could be a fluke
Correlation •“R score” •Showing there is
S t t
•Some stats some relationship
l ti hi
•You can do this in beyond coincidence
excel
•Pearson’s Product •Not ‘proving’
Moment media exclusively
responsible for sales


carma.com 49


than lines that
might trend
together

•Could be a fluke
Correlation •“R score” •Showing there is
S t t
•Some stats some relationship
l ti hi
•You can do this in beyond coincidence
excel
•Pearson’s Product •Not ‘proving’
Moment media exclusively
responsible for sales


Causality •Heavy duty stats •Isolates
•You so can’t do this in (
(statistically
y
excel ‘proves’) for PR’s
•Sophisticated unique contribution
modeling to the
•Market Mix Modeling communications
•Multivariate mix
Regression Analysis
Regression Analysis
•Ton ‘o data, time, $
carma.com 50

TECH INDUSTRY EXAMPLE

Correlating opinion with
CARMA IQ (favorability +
reach)

carma.com 51

AUTO INDUSTRY EXAMPLE

Monthly Sales Results vs. Four Month CARMA Favorability:
Monthly Sales to Four Previous Months’ Favorability

carma.com 52

Toyota’s Heat Map:
CARMA Media IQ Metrics by Various Time Frames Correlated to
Monthly Sales Results
y

carma.com 53

UPPERCASE ‘ROI’ vs. lowercase ‘roi’ & Other Return-Offs

ROI roi
MBA PR
Financial Non‐financial
Non financial
Macro Micro (not all conversion events are ROI)
A mathematical equation Usually conceptual only; an equation in words only; can’t actually
that works
that works be calculated; something the industry tosses around too liberally
be calculated; something the industry tosses around too liberally

A business metric All‐too‐often a media or audience metric
Direct, clear A loose proxy or an intermediate metric (return on expectation,
return on objective, return on engagement)
return on objective return on engagement)


carma.com 54

THOSE USING TXT ANALYTICS ARE MEASURING ‘ROI’ NOT ‘roi’

Source: Text Analytics 2009: U
S T t A l ti 2009 User P Perspectives on S l ti
ti Solutions and P id
d Providers

carma.com


SOCIAL MEDIA ROI CALCULATOR(S) THE GOOD & THE BAD

carma.com


SOCIAL MEDIA BOOK BAG

• Social Media Metrics:
• Jim Sterne, (@jimsterne)

• Social Media ROI:
• Olivier Blanchard (@thebrandbuilder)

• Social Media Analytics:  Effective Tools for Building, Interpreting & Using Metrics:
• Marshal Sponder (@webmetricsguru)

• Measure What Matters:
M Wh M
• Katie Paine (@kdpaine)

• Using Web Analytics to Measure Impact of Earned Online Media…:
g y p
• Seth Duncan (www.instituteforpr.org)

carma.com 57

PART 1 (1.5 hrs): SOCIAL MEDIA MEASUREMENT
IN-ROOM STRETCH (5 mins)

PART 2 (30 mins): TONE ALONE
PART 3: (35 mins): REAL DATA WITH MALCOM DE LEO NETBASE

carma.com

AGENDA FOR PART 2 DEEP DIVE ON SENTIMENT

•Sentiment Analysis

•Why is it important/ why should we care?

•Sentiment as a Proxy / Predictor

•Applications

•Humans or Machines or Both?

carma.com 59





•Applications
•Why Automation?
•Situating Automated S
S Sentiment Analysis

•Defining Automated Sentiment Analysis

•Output

•Taxonomies etc.

•Automated Vendors

•Complications
p

carma.com 60





•Applications
•Why Automation?
•Situating Automated S
S Sentiment Analysis

•Defining Automated Sentiment Analysis

•Output
•Human vs. Automated
vs
•Taxonomies etc.
•Human-based vendors
•Automated Vendors
•Human-Based Sentiment Analysis
•Complications
p
•Favorability not (just) Sentiment

•Humans supplemented by automation

Linking outputs to outcomes revisited
carma.com 61

I AM NOT IAM

A COMPUTATIONAL LINGUIST
A SEMANTICIAN
COMPUTER ENGINEER
COMPUTER SCIENTIST
LOGICIAN
PSYCHOLINGUIST

BUT…CHECK
BUT CHECK OUT SETH GRIMES :
GRIMES’:
Text Analytics 2009: User Perspectives on Solutions and Providers
A number of excellent presentations on LinkedIn / Slide Share

I’ve borrowed liberally from both for this Part 2

carma.com 62

I AM NOT IAM

A COMPUTATIONAL LINGUIST
A RECOVERING PR FLACK
CO G C
A SEMANTICIAN
PR MEASUREMENT MOUTHPIECE
COMPUTER ENGINEER
WITH AN INTEREST IN MONITORING +
COMPUTER SCIENTIST
MINING ONLINE CONVERSATIONS
LOGICIAN
PSYCHOLINGUIST

BUT…CHECK
BUT CHECK OUT SETH GRIMES :
GRIMES’: Sentiment through the eyes
Text Analytics 2009: User Perspectives on Solutions and Providers
A number of excellent presentations on LinkedIn / Slide Share of a simpleton
I’ve borrowed liberally from both for this Part 2

carma.com 63

IMAGINE A DIFF’T. PYRAMID FOR DIFF’T. CAMPAIGN TYPES / OBJECTIVES

carma.com 64

WHY IS IT IMPORTANT? WHY SHOULD WE CARE?

Questions for biz & gov’t.:

•What are people saying?
Wh t l i ?
•What’s trending?
•How has opinion about x changed?
•Does it d e c a e to c a e
oes differ channel o channel
•How has sentiment correlated with
news / marketing / events etc.?
•Can we link opinions to transactions?

Computers are trading stock
based on automated sentiment
analysis of online content

carma.com 65

SENTIMENT ANALYSIS AS MARKET RESEARCH PROXY / PREDICTOR

Think here about social media monitoring and
analysis as:

•Real time situational awareness
•Listening at the point of need
•Listening close to the moment of influence

A real-time digital form of or proxy for market
research and CRM:

Mining conversations to tease out how people
‘feel’ about products/services/issues/causes
feel
and perhaps even:

1) their intended (reported) behavior and what
motivates them to buy, (as cited in the
conversations), and
ti ) d

2) their actual behavior tracked via CRM 2.0
and Web 2.0 analytics

1615 M Street, NW; Suite 750 Source: Crimson Hexagon
carma.com 66

APPLICATIONS

Call Centre Notes?
carma.com 67

SITUATING SENTIMENT ANALYSIS AUTOMATED

Text analytics:
•Applies linguistic +/or statistical
Artificial Intelligence techniques to extract concepts and
patterns

Text A l ti
T t Analytics •Transforms unstructured information into
data for analysis
•Unlocks meaning and relationships in
large volumes of information
Sentiment Analysis
•Adds semantic understanding of:
•Sentiment
•Opinions
Opinions
•Attitudes
•Entities
•Concepts
•Facts
Facts
•Abstract attributes: “expensive,”
carma.com
“comfortable”
68

AUTOMATED SENTIMENT ANALYSIS DEFINED

Sentiment analysis or opinion mining refers to the application of:
•Natural language processing
Linguistics Comp Science
•Computation linguistics
p g
•Text analytics…

…to identify and extract subjective information in content. Statistics

Natural Language Processing
• Grounded in statistical machine learning
• Much overlap with…
Computational Linguistics
Computational Linguistics
• Statistical or rule‐based modeling of language
Text Analytics
• Turning text into data
• Lexical analysis: word frequencies
• Entity recognition
• Pattern recognition
• Tagging/ annotation
Tagging/ annotation
carma.com


EXTRACTING ATTITUDES TURNING ATTITUDES INTO DATA

Generally speaking, sentiment analysis aims to determine the attitude of a speaker or a
writer with respect to some topic or the overall tonality of a document.

The attitude may be his or her:

• judgment or evaluation

• affective state (emotional state of the author when writing), or

• the intended emotional communication (emotional effect the author wishes to
have on the reader).

carma.com 70

HUMANS OR MACHINES OR BOTH?
ALGORITHM AL GORITHM

Can it be over ridden?
Does it learn?

THE ALGORITHMICS
carma.com 71

WHY AUTOMATION?

Time Spent
Analyzing Content
y g

w/out automation

Time Spent on
Ti S
Insights &
Implications

w/ automation

carma.com 72

WHY AUTOMATION?

Time Spent
Analyzing Content
y g

w/out automation

Time Spent on
Ti S
Insights &
Implications

w/ automation

carma.com 73

THE OUTPUT

Entity E t ti
E tit Extraction

Issues / Themes /
Messages

carma.com 74

TAXONOMIES…LEXICONS…DICTIONARIES…BAGS-O-WORDS…CARTRIDGES

• Words/phrases that
PRE‐SET are generally good or
are generally good or
bad (Hate. Love)
GENERAL • Caution!

carma.com 75


• Words/phrases that are
PRE SET
PRE‐SET generally good or bad
generally good or bad
(Hate. Love)
GENERAL • Caution!

• Financial / market news
PRE SET
PRE‐SET • Mergers & Acquisitions
• Competition
SPECIFIC • Better

carma.com 76


PRE‐SET GENERAL (Hate. Love)
• Caution!

PRE‐SET
PRE SET • Mergers & Acquisitions
SPECIFIC • Competition
• Better

CUSTOM
• Client‐specific context
CLIENT/PROJECT • Best
SPECIFIC

carma.com 77


PRE‐SET GENERAL (Hate. Love)
• Caution!

PRE‐SET
PRE SET • Mergers & Acquisitions
SPECIFIC • Competition
• Better

CUSTOM • Client‐specific context
CLIENT/PROJECT • Best
• Multiple cartridges
SPECIFIC

But! Still needs some human
intervention / validation /
feedback so it can ‘learn’ and
improve
carma.com 78

AUTOMATED TEXT ANALYTICS VENDORS

carma.com 79

1st GENERATION vs. 2nd GENERATION AUTOMATED SENTIMENT

Conceptual Search-Based
Key Word (only) Deeper linguistic level
p g
Based
Operates on a semantic level
Explicit Understands sentence grammar:
References
•Subject
Counting up the •Predicate
•Object
“+” & “-” •Who?
•Action
•Idea
•…and the relationships between these
Most studies show that key •…must contain these terms but in a particular
word matching is wrong relationship to one another
more than 50% of the time
Opinions, claims, arguments, facts, actions, concepts,
messages

Non-implicit: theme / spirit / i t t
N i li it th i it intent
carma.com 80

COMPLICATIONS & PROBLEMS w/ 1ST GEN AUTOMATED APPROACHES

•Sentiment may be of interest at multiple levels:
•Document
•Several Documents
•Statement / sentence
Statement
•Entity
•Topic or Theme
•Concept

•Language is chaotic:
•Jargon, slang, (slanguage)
•Colloquialism
•Forum-specific acronyms
•ambiguity,
•Anaphora (it, them)
Anaphora
•Irony, sarcasm
•Misspellings
•Poor grammar
•Variations in expression
•Synonyms
•Polysemy

•Context is key:
•Meaning can change depending on context.
•Need to look not only at the word but the connective tissues surrounding the word.

carma.com 81

CONTEXT IS KEY

carma.com 82

ANALYSING THE “CONNECTIVE TISSUE” OF A SENTENCE

carma.com 83

HUMAN VS. AUTOMATION BOTH IS BEST

•Most sentiment analysis algorithms rely on
HUMANS
humans using simple terms to express our sentiment AUTOMATION FOR
RELEVANCE,
FOR VOLUME
about a product or service. We don’t. & SPEED
DEPTH,
CONTEXT,
NUANCE

•Cultural f t
C lt l factors, li
linguistic nuances and diff i
i ti d differing
contexts make it extremely difficult to automate turning
Bridge the Gap with
a string of written text into a simple p or con
g p pro Human analysis
Human analysis
of a sample
sentiment.

•The fact that humans often disagree on the sentiment
of text illustrates how big a task it is for computers to
get this right.
right
carma.com 84

HUMAN (or hybrid)-BASED VENDORS

carma.com 85

Social Media Measurement 101 + Day of aTONEment

Social Media Measurement 101 + Day of aTONEment

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Social Media Measurement 101 + Day of aTONEment

Similar a Social Media Measurement 101 + Day of aTONEment (20)

Social Media Measurement 101 + Day of aTONEment