IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
The 2010 JDPA Sentiment Corpus for the Automotive Domain
1. The JDPA Sentiment Corpus
for the Automotive Domain
Miriam Eckert, Lyndsie Clark,
Nicolas Nicolov
J.D. Power and Associates
Jason S. Kessler
Indiana University
2. Overview
• 335 blog posts containing opinions about cars
– 223K tokens of blog data
• Goal of annotation project:
– Examples of how words interact to evaluate entities
– Annotations encode these interactions
• Entities are invoked physical objects and their
properties
– Not just cars, car parts
– People, locations, organizations, times
3. Excerpt from the corpus
“last night was nice. sean bought me caribou
and we went to my house to watch the baseball
game …
“… yesturday i helped me mom with brians
house and then we went and looked at a kia
spectra. it looked nice, but when we got up to it,
i wasn't impressed ...”
4. Outline
• Motivating example
• Overview of annotation types
– Some statistics
• Potential uses of corpus
• Comparison to other resources
5. John recently purchased a
had agreat a disappointing stereo,
and was
mildly
very grippy. He also considered a
which, while highly had a better
PERSON
Honda Civic.
CAR
engine,
CAR-PART CAR-PART
stereo.
CAR-PART
CARPERSON
BMW
It
CAR
REFERS-TO
priced
CAR-FEATURE
REFERS-TO
6. John recently purchased a
had agreat a disappointing stereo,
and was
mildly
very grippy. He also considered a
which, while highly had a better
PERSON
Honda Civic.
CAR
engine,
CAR-PART CAR-PART
stereo.
CAR-PART
CARPERSON
BMW
It
CAR
priced
CAR-FEATURE
TARGET TARGET TARGET
TARGET
TARGET
7. John recently purchased a
had agreat a disappointing stereo,
and was
mildly
very grippy. He also considered a
which, while highly had a better
PERSON
Honda Civic.
CAR
engine,
CAR-PART CAR-PART
stereo.
CAR-PART
CARPERSON
BMW
It
CAR
REFERS-TO
priced
CAR-FEATURE
REFERS-TO
PART-OF PART-OF
FEATURE-OF
PART-OF
8. John recently purchased a
had a great a disappointing stereo,
and was
mildly
very grippy. He also considered a
which, while highly had a better
PERSON
Honda Civic.
CAR
engine,
CAR-PART CAR-PART
stereo.
CAR-PART
CARPERSON
BMW
It
CAR
priced
CAR-FEATURE
DIMENSION
MORE
LESS
9. John recently purchased a
had a great a disappointing stereo,
and was
mildly
very grippy. He also considered a
which, while highly had a better
PERSON
Honda Civic.
CAR
engine,
CAR-PART CAR-PART
stereo.
CAR-PART
CARPERSON
BMW
It
CAR
REFERS-TO
PART-OF PART-OF
TARGET TARGET TARGET
TARGET
TARGET
priced
CAR-FEATURE
FEATURE-OF
DIMENSION
MORE
LESS
Entity-level
sentiment: positive
Entity-level
sentiment: mixedREFERS-TO
TARGET
10. Outline
• Motivating example
• Overview of annotation types
– Some statistics
• Potential uses of corpus
• Comparison to other resources
11. John recently purchased a Civic. It had a
great engine and was priced well.
John
PERSON
Civic It
Entity annotations
REFERS-TO
REFERS-TO
CAR
engine
CAR-PART
• >20 semantic types from
• ACE Entity Mention Detection Task
• Generic automotive types
priced
CAR-
FEATURE
12. Entity-relation annotations
Entity-level sentiment:
Positive
• Relations between entities
• Entity-level sentiment
annotations
• Sentiment flow between
entities through relations
• My car has a great engine.
• Honda, known for its high
standards, made my car.
Civic
CAR
engine
CAR-
PART
priced
CAR-
FEATURE
PART-OF FEATURE-
OF
13. Entity annotation type: statistics
• Inter-annotator
agreement
• Among mentions 83%
• Refers-to: 68%
• 61K mentions in corpus
and 43K entities
• 103 documents
annotated by around 3
annotators
A1: …Kia Rio…
A2: …Kia Rio…
MATCH
A1: …Kia Rio…
A2: …Kia Rio…
NOT A MATCH
15. Sentiment expressions
• Occurrences in corpus: 10K
• 13% are multi-word
• like no other, get up and go
• 49% are headed by adjectives
• 22% nouns (damage, good amount)
• 20% verbs (likes, upset)
• 5% adverbs (highly)
16. Sentiment expressions
• 75% of sentiment expression occurrences
have non evaluative uses in corpus
• “light”
– …the car seemed too light to be safe…
– …vehicles in the light truck category…
• 77% sentiment expression occurrences are
positive
• Inter-annotator agreement:
– 75% spans, 66% targets, 95% prior polarity
17. Modifiers -> contextual polarity
NEGATORS
not a good car
not a very good car
INTENSIFIERS
very good cara
kind of good cara
UPWARD
DOWNARD
NEUTRALIZERS
i
f
goodthe car is
I hope goodthe car is
COMMITTERS
sure goodthe car isI am
UPWARD
suspect goodthe car isI
DOWNWARD
18. Other annotations
• Speech events (not sourced from author)
–John thinks the car is good.
• Comparisons:
–Car X has a better engine than car Y.
–Handles a variety of cases
19. Outline
• Motivating example
• Overview of annotation types
– Some statistics
• Potential uses of corpus
• Comparison to other resources
20. Possible tasks
• Detecting mentions, sentiment expressions,
and modifiers
• Identifying targets of sentiment expressions,
modifiers
• Coreference resolution
• Finding part-of, feature-of, etc. relations
• Identifying errors/inconsistencies in data
21. Possible tasks
• Exploring how elements interact:
– Some idiot thinks this is a good car.
• Evaluating unsupervised sentiment systems or
those trained on other domains
• How do relations between entities transfer
sentiment?
– The car’s paint job is flawless but the safety record
is poor.
• Solution to one task may be useful in solving
another.
22. But wait, there’s more!
• 180 digital camera blog posts were annotated
• Total of 223,001 + 108,593 = 331,594 tokens
23. Outline
• Motivating example
– Elements combine to render entity-level
sentiment
• Overview of annotation types
– Some statistics
• Potential uses of corpus
• Comparison to other resources
24. Other resources
• MPQA Version 2.0
– Wiebe, Wilson and Cardie (2005)
– Largely professionally written news articles
– Subjective expression
• “beliefs, emotions, sentiments, speculations, etc.”
– Attitude, contextual sentiment on subjective
expressions
– Target, source annotations
– 226K tokens (JDPA: 332K)
25. Other resources
• Data sets provided by Bing Liu (2004, 2008)
– Customer-written consumer electronics product
reviews
– Contextual sentiment toward mention of product
– Comparison annotations
– 130K tokens (JDPA: 332K)
26. Thank you!
• Obtaining the corpus:
– Research and educational purposes
– ICWSM.JDPA.corpus@gmail.com
– June 2010
– Annotation guidelines:
http://www.cs.indiana.edu/~jaskessl
• Thanks to: Prof. Michael Gasser, Prof. James
Martin, Prof. Martha Palmer, Prof. Michael
Mozer, William Headden