Web 2.0 applications for social networking provide data about users’ mood and opinions in almost real time. Many applications are taking advantage of these data to derive business intelligence. However, the volume of data makes it hard and error-prone to classify sentiments and opinions manually. The combination of data mining techniques and a pipeline to process data from Web 2.0 applications, such as Twitter, Facebook, and Wordpress, makes it possible to apply natural language processing and machine learning techniques to automate partially this task. Therefore, the amount of manual classification is reduced, as the incoming data has already a classification tag that can be easily changed, feeding back the classifier. There is room for improvements and a Brazilian Portuguese Corpus was created to do the initial training of the classifier. The code used for this testing was based on open source libraries and is available as a test bed for different corpora and new algorithms.

1. Semi Automatic Sentiment Analysis
Results from a case study in Brazilian Portuguese web 2.0 sites
Gleicon Moraes, Marco Aurélio Gerosa
gleicon@gmail.com, gerosa@ime.usp.br

2. Introduction
• Popular Web 2.0 applications are based on social
networking: Facebook, Twitter, Orkut, Flickr, LinkedIn
• Status messages, user information, wall posts, like/unline
votes, scraps, recommendations are created and exchanged
between users.
• Symetric and Assymetric relationships broadcast these
messages between friends (direct connections) and friends
of friends.
• Sentiment and opinions might be objective (up/down votes,
recommendations) or subjective (free text)
Gleicon Moraes, Marco Aurélio Gerosa 2/20

3. Sentiment Classification
• Find out what users in a social network think about
product, tendency or brand.
• Compute or help to compute the Return of Investment of a
marketing campaign
• Create or compose product and services recommendations
to other users
• To measure user satisfaction and experience about a
service.
Gleicon Moraes, Marco Aurélio Gerosa 3/20

4. Goals
• Opinion mining / subjectivity and sentiment analysis review
[1]
• Automate opinion classification (tweet, scrap, message, wall
post) using Machine Learning and Information Retrieval
techniques.
• To apply a Bayesian filter (and also try a SVM classifier) to
identify Positive and Negative sentiment on brazilian
Portuguese texts.
• To build a corpus to train and test the classifiers
• To find out how to measure the filter efficiency.
[1] Pang e Lee - Opinion Mining and Sentiment Analysis
Gleicon Moraes, Marco Aurélio Gerosa 4/20

5. Related work
• Thumbs Up? Sentiment Classification using Machine Learning
Techniques: Bayesian filter, Maximum Entropy filter and SVM filter. Training
was made with Movielens dataset, splitting between 70% corpus to training and
30% to test. This corpus is already marked as positive and negative. Conclusion
was that sarcasm on opinions made it difficult to classify the sentiments. There
was no smaller text classification (e.g. A tweet/140 chars) and feedback with
outside text to the classifiers. [1]
[1] Pang, B. Lee L., Cornell University, Vaithyanathan S, IBM: Thumbs Up? Sentiment Classification using Machine Learning
Techniques
Gleicon Moraes, Marco Aurélio Gerosa 5/20

6. Related work
• Content-based book Recommendation Using Learning for Text
Categorization and information extracted from the internet to train a classifier,
with a database per user. The combination between collaborative filtering and
content filtering complete each other and help improve the results. [1][2]
[1] Mooney R. J., Roy L., “Content-Based Book Recommendation Using Learning for Text Categorization” (Proceedings of
ACM Conference on Digital Libraries, 2000)
[2] Dˇzeroski S., Zenko B. “Is Combining Classifiers Better than Selecting the Best One?”
Gleicon Moraes, Marco Aurélio Gerosa 6/20

7. Semi-Automatic Sentiment Classification
• Trained Bayesian Filter on two categories: “positive” and “negative”
• Feedback feature so false positives and false negatives could be
trained back to improve the filter
• Problem: There is not a brazilian portuguese data matching text to
sentiment to do the initial classificator training.
• Problem: Text composition varies between social networks and
groups within these networks. Feeding back data to keep the
classificator database updated is fundamental
Gleicon Moraes, Marco Aurélio Gerosa 7/20

8. Semi-Automatic Sentiment Classification
• English language training corpus uses movie reviews in
most papers, associated with ratings to tell what that text
block express [1]
• An initial training corpus was made using consumer review
data from Brazilian websites like iVox, ReclameAqui,
opiniões do MercadoLivre
• After scrapping each opinion and its rating (stars, rating, or
positive/negative indication), stored it on folders ranging
from 0.0 to 5.0, each opinion a file inside the proper folder
[1] MovieLens dataset: http://www.grouplens.org/node/73
Gleicon Moraes, Marco Aurélio Gerosa 8/20

9. Training composition findings
• Number of words in negative opinions is bigger than on
positive opinions: 67.575 words in 712 positive opinions
versus 81.747 words in 507 negative opinions.
• Distribution of reviews between minimum and maximum
ratings: more opinions on the extremes (0.0 to 0.5 and 4.5
to 5.0).
Gleicon Moraes, Marco Aurélio Gerosa 9/20

10. Composição da base de treinamento - iVox
Gleicon Moraes, Marco Aurélio Gerosa 10/22

11. Domain
• Language domain varies between communities/sites
Gleicon Moraes, Marco Aurélio Gerosa 11/20

12. Opinion Sample (Mercadolivre)
positivo (rating 5):
"Este alto-falante faz o baile tremer... comprei para montar uma mini-
saveiro”
negativo (rating 1):
"Apesar de custar muito barato recomendo economizar e comprar
falantes de marcas conhecidas. Bravox, Selenium.
O produto parece recondicionado, e não tem 90Wats nem na china,
meu triaxial Pionner de 60Wats aquenta muito mais grave que esse
Unlike.
Não faça besteira economize mais R$60,00 e compre um Kit 2 vias
Selenium ou até Sony ou Bomber que custa quase o mesmo aqui no
Mercado Livre"
Gleicon Moraes, Marco Aurélio Gerosa 12/20

13. Opinion Sample (iVox)
positivo (rating 5.0):
"Economica não tem Adquiri uma web.evo Sundown,à moto é bonita,gostei tanto
da Sundown que adquiri mais uma moto Sundow a hunter 90cc. estou com 2
motos e estou muito satisfeito. Quanto ao pessoal da grappa, todos sem exceção
sempre bem atenciósos comigo; só tenho a agradecer. "
negativo (rating 1):
"Contra Todas Não sei o motivo de sua defesa a esta empresa, pois fui
enganado a pouco tempo e o engraçado é que liguei para reclamar,
bem na hora que o vendedor estava enganando outra pessoa, por um
deslize do mesmo o cliente verificou o numero e me ligou dizendo que
também havia sido enganado. Entramos com denúncia conjunta na
DECON do DF. Razoável Muito Ruim Razoável Muito Ruim"
Gleicon Moraes, Marco Aurélio Gerosa 13/20

14. Opinion Sample (Reclame Aqui)
positivo:
"Olá, estou passando apenas para parabenizar ao ótimo e sério trabalho da
equipe do site reclameaqui.com.br, pois já fui atendido em duas ocasiões
reclamadas no site e foi algo bem melhor e mais rápido do que partir para outras
atitudes. Parabéns e que cada vez mais possamos ter meios iguais para
podermos agilizar o processo de negociação.
Obrigado,"
negativo:
"Fiz 2 reclamações contra a MOTOROLA DO BRASIL por propaganda
enganosa em seu site www.motorola.com.br sobre o aparelho V3m que
no site diz ACOPMPANHA cartão enquanto no meu aparelho nao veio
NADA !!! Eles me ligaram e tiram o deles da reta dizendo que a culpa é
da VIVO ! MAis perai quem faz o aparelho nao é eles ??? A VIVO so
revende !!!! Ah MOTOROLA POR FAVOR NE !!!!! QUERO MEU
CARTAO !!!" Gleicon Moraes, Marco Aurélio Gerosa 14/20

15. Domain
• Language Domain [1]: "go read a book” has different meaning related
to each social network. In a book related network might be a positive
meaning. In others might mean a negative sentiment.
• Feeding back data also helps to keep the database updated with new
slangs and combinations that also might cover sarcasm expressions.
• Events like world cup and television shows might introduce new words
and expressions.
[1] Pang e Lee - Opinion Mining and Sentiment Analysis
Gleicon Moraes, Marco Aurélio Gerosa 15/20

16. Training
• Split the database between negative (rating: 0.0) and positive (rating
5.0). Later steps added ratings 4.5, 4.0 to positive while negative rating
kept the same.
• Training/Classifying applied on raw data and on data processed a
pipeline of taking out stop words and extracting the stem of remaining
words
• Raw data biased towards negative sentiment, processed data biased
towards positive sentiments.
Gleicon Moraes, Marco Aurélio Gerosa 16/20

17. Results – raw data
iVox ReclameAqui False results
Ratings Negative/Positive Negative Positive Negative Positive
No training No messages 1635 268 0 0
0.0 e 5.0 506/720 1634 6 262 1
0.0 e 4.5 + 5.0 506/873 1587 99 169 48
0.0 e 4.0 + 4.5 + 5.0 506/973 1365 165 105 270
Gleicon Moraes, Marco Aurélio Gerosa 17/20

18. Results – filtered data
iVox ReclameAqui False results
Ratings Negative/Positive Negative Positive Negative Positive
No training No messages 1635 268 0 0
0.0 and 5.0 506/720 1635 0 268 0
0.0 and 4.5 + 5.0 506/873 0 261 0 1627
0.0 and 4.0 + 4.5 + 5.0 506/973 0 268 0 1635
Gleicon Moraes, Marco Aurélio Gerosa 18/20

19. Measuring efficiency
• Metrics: Accuracy, Precision Recall
• Token extraction: words (bag of words) and bigrams.
• Test between languages and domain: trained and tested the same
classifiers and extractors with the Movielens dataset [1]
[1] The MovieLens dataset: http://www.grouplens.org/node/73
Gleicon Moraes, Marco Aurélio Gerosa 19/20

20. Efficiency
Movie Review (en)
Feature Extractor Accuracy Positive Negative Positive Negative
Precision Precision Recall Recall
Bag of Words 0.7280 0.6516 0.9597 0.9800 0.4760
Bigrams 0.8240 0.7613 0.9263 0.9440 0.7040
Consumer Opinion (pt_br)
Feature Extractor Accuracy Positive Negative Positive Negative
Precision Precision Recall Recall
Bag of Words 0.5984 1.0000 0.5100 0.3099 1.000
Bigrams 0.7049 1.0000 0.5862 0.4930 1.000
Gleicon Moraes, Marco Aurélio Gerosa 20/20

21. Conclusion
• Consumer review database helped on initial training.
• O keep the messages as is helps makes the database richer with
different forms of the same expression
• Token extraction influences the end result
• Feeding back helps to keep the database up to date
• To combine classifiers helps the end results and the precision
• Contribution: Brazilian portuguese database and scripts used to extract
data and to reproduce the experiment at: https://github.com/gleicon/
sentiment_analysis
Gleicon Moraes, Marco Aurélio Gerosa 21/20