I gave this presentation at Workshop on Interactive Language Learning, Visualization, and Interfaces / ACL 2014 in Baltimore, MD on June 27, 2014. http://nlp.stanford.edu/events/illvi2014/index.html ABSTRACT Everyday on Twitter, there are millions of thoughts that are captured and shared to the world in the form of 140-character messages, or Tweets. There are many things we could learn from these thoughts if we could figure out a way to digest this gigantic dataset. Visualization is one of the many ways to extract information from these Tweets. In this presentation, I will talk about several visualizations based on Tweets, as well as share experiences and challenges from working with Tweet data.

1. Making Sense of
Millions of Thoughts
Finding
patterns
in the
Tweets
“Knowing comes from learning, from seeking.”
“What we call chaos is just we haven't recognized.”
“I am looking for a needle haystack.”
“140-character text messages, called ”
Krist Wongsuphasawat
(50 characters)
(58 characters)
(42 characters)
(42 characters)

2. X-Men

3. Prof. X
Ability: Telepathy
(mind reading)

4. Cerebro
Enhance telepathy
Prof. X

5. Cerebro

6. With this power…

7. What are you thinking?

8. What are people thinking
about x?
Product Event
Person
etc.

9. Reality

10. Cerebro

11. Internet

12. Platform
thought
thought
thought
thought
thought
crowdsourcing
social networks
Data

13. Twitter
tweet
tweet
tweet
tweet
tweet
Tweets

14. Tweets
• 140 characters
• text + media
• geo
• time

15. Twitter
tweet
tweet
tweet
tweet
tweet
Tweets

16. What can we learn
from these Tweets?

17. visual-insights@twitter
@miguelrios @philogb @trebor @kristw

18. World Cup
Election
Oscars
Pure Curiosity
Grammy
TV Shows
New Year
Breaking news
Earthquake

19. Insights, Stories
(Tweets)
DATA
with limited time
Audience: general public

20. Tools
• Hadoop
• Apache Pig
• Vertica
• node.js, python
• d3 & co.

21. Pig

22. Insights, Stories
(Tweets)
DATA

23. Insights, Stories
(Tweets)
Filter
DATA

24. Having all Tweets
How people think I feel.

25. Having all Tweets
How people think I feel. How I really feel.

26. Filter data
Good news:
Bad news:
Want only relevant Tweets
Have all Tweets
Too many Tweets

27. Filter data (2)
• #hashtags — e.g. #world-cup
• easy to filter
• hashtags must be presented
• typo?

28. Filter data (2)
• #hashtags — e.g. #world-cup
• easy to filter
• hashtags must be presented
• keywords — e.g. goal
• broader
• can be ambiguous

29. Filter data (3)
• Combine with other attributes
• Time
• during the first half of World Cup final

30. Filter data (3)
• Combine with other attributes
• Time
• during the first half of World Cup final
• Location
• Tweets from Brazil
• Not every Tweet is geotagged.

31. Filter data (4)
• Languages
• Sometimes use only English Tweets
• Future
• Translation?

32. Insights, Stories
(Tweets)
Filter
Clean
DATA

33. Clean data
• Typo (Mobile input)
• Abbreviation (due to 140-character limit)
• Exaggeration (e.g. GOOOOALLLL)
• Twitter specific e.g., Old-style retweet “RT …”
• Inappropriate content

34. Insights, Stories
(Tweets)
Filter
Clean
Visualize
DATA

35. (+ media)
photos, videos
What?
Where? When?
GEO TIME
TEXT
DATA

36. What?
Where? When?
GEO TIME
TEXT
Visualize Data

37. What?
Where? When?
GEO TIME
TEXT
Visualize Data

38. TIME Tweets/second

39. TIME Tweets/second

40. TIME Tweets/second + Annotation
http://www.flickr.com/photos/twitteroffice/5681263084/

41. TIME Tweets/second + Annotation
Manual
To automate
Top tweets
(most Retweets, Favs)

42. What?
Where? When?
GEO TIME
TEXT
Visualize Data

43. GEO
Heatmap
Low density
High density

44. GEO
New York City
flickr.com/photos/twitteroffice/8798020541

45. GEO
San Francisco
flickr.com/photos/twitteroffice/8798020541

46. GEO
San Francisco
Rebuild
the world
based on
tweet volumes
twitter.github.io/interactive/andes/

47. What?
Where? When?
GEO TIME
TEXT
Visualize Data

48. TIME + GEO
blog.twitter.com/2011/global-pulse
youtu.be/SybWjN9pKQk
Japan Earthquake 2011

50. TIME + GEO Tweet pattern [Rios & Lin 2012]
Night
Late night
Daytime
Night
Late night
Daytime

51. What?
Where? When?
GEO TIME
TEXT
Visualize Data

52. TEXT Trends

53. TEXT
www.wordle.net
Some samples from
World Cup

54. TEXT Word cloud of Tweets right after the 1st goal
www.wordle.net

56. TEXT WordTree [Wattenberg & Viégas 2008]
www.jasondavies.com/wordtre
www.jasondavies.com/wordtree

57. TEXT
• Now
• Derived information: Sentiment, Topic
• Combine with other information (geo & time) + context
• Future
• Better technique + involves more NLP e.g. key phrases, etc.

58. TEXT Descriptive Keyphrases [Chuang et al. 2012]

59. TEXT
• Challenge
• Scale

60. What?
Where? When?
GEO TIME
TEXT
Visualize Data

61. GEO + TEXT Real-time Tweet map

62. GEO + TEXT Real-time Tweet map

63. GEO + TEXT Real-time Tweet map
most
frequent
term

64. GEO + TEXT Real-time Tweet map
Gmail went down
Jan 24, 2014

65. GEO + TEXT Real-time Tweet map
Nelson Mandela
passed away
Dec 5, 2013

66. GEO + TEXT Real-time Tweet map
• Next:
• Involves more NLP
• Tokenization - Languages without space between words
• etc.
• Challenge:
• Real-time

67. GEO + TEXT
www.yelp.com/wordmap
Yelp Wordmap

68. What?
Where? When?
GEO TIME
TEXT
Visualize Data

69. TIME + TEXT
http://www.babynamewizard.com/voyager
Baby Name Voyager

70. TIME + TEXT
http://www.babynamewizard.com/voyager
Baby Name Voyager

71. TIME + TEXT
UEFA Champions League
Biggest Tournament for European soccer clubs
Many Tweets during the matches

72. TIME + TEXT UEFA Champions League
Dortmund Bayern Munich
Count Tweets mentioning
the teams every minute
Team 1 Team 2

73. TIME + TEXT UEFA Champions League

74. TIME + TEXT UEFA Champions League
+ “goal” count
+ context

75. TIME + TEXT UEFA Champions League
+ “offside”

76. TIME + TEXT UEFA Champions League
+ players

77. A B C D
A C
C
Competition Tree
vs vs
vs

78. A B C D
A C
C
Competition Tree
+
vs vs
vs

79. A B C D
A C
C
Competition Tree
+ =
uclfinal.twitter.com
vs vs
vs

80. TIME + TEXT UEFA Champions League
• Challenges
• Filter relevance tweets
• Multiple matches at the same time
• Ambiguous words: “goal”, “red”, “yellow”
• Tweets mentioning both teams e.g. “#GER 2-2 #GHA”

81. What?
Where? When?
GEO TIME
TEXT
Visualize Data

82. TIME + GEO + TEXT State of the Union
twitter.github.io/interactive/sotu2014

83. TIME + GEO + TEXT State of the Union
1) timeline + topic from Tweets
4) Density map of
Tweets about
selected topic
3) Volume of Tweets
by topics
during selected
part of the SOTU
2) context
(speech)
twitter.github.io/interactive/sotu2014

84. TIME + GEO + TEXT New Year 2014

85. TIME + GEO + TEXT New Year 2014

86. TIME + GEO + TEXT New Year 2014
twitter.github.io/interactive/newyear2014/

87. Recap

88. What can we learn
from these Tweets?
many, many things.

89. better
the examples
in this talk
imagine…
DATA
(Tweets)

90. Insights, Stories
(Tweets)
Filter
Clean
Visualize
DATA

91. (Tweets)
Insights, Stories
Filter
Clean
Process &
Visualize
DATA

92. (Tweets)
Insights, Stories
Filter
Clean
Process &
Visualize
DATA
NLP

93. TEXT
What?
Where? When?
GEO TIME
Visualize data

94. (Tweets)
Insights, Stories
Filter
Clean
Process &
Visualize
DATA
Research

95. Working
together
Raw data
Human

96. Working
together
Raw data
Human
Computer (One machine, Cloud, MapReduce, etc.)

97. Working
together
Raw data
Human
Ignored informationProcessed information
Computer (One machine, Cloud, MapReduce, etc.)

98. Working
together
Raw data
Human
Aggregated information
Ignored informationProcessed information
Computer (One machine, Cloud, MapReduce, etc.)

99. Working
together
Raw data
Human
Aggregated information
Ignored informationProcessed information
Computer (One machine, Cloud, MapReduce, etc.)
NLP Make computers
think more like Human.

100. Working
together
Raw data
Human
Aggregated information
Ignored informationProcessed information
VIS
Help people consume information.
Computer (One machine, Cloud, MapReduce, etc.)
NLP Make computers
think more like Human.

101. Working
together
Raw data
Human
Aggregated information
Ignored informationProcessed information
VIS
Help people consume information.
Computer (One machine, Cloud, MapReduce, etc.)
NLP Make computers
think more like Human.
HCI
User interactions
or
Provide feedback
Bridge the gap. Connect human & computer.

102. Advanced techniques
vs.
Scalability

103. LifeFlow => Flying Sessions
Research System at Twitter

104. Summary
• Thoughts are captured in the Tweets: what, where, when
• Finding patterns from: text + geo + time
• Opportunities for NLP + HCI + VIS collaboration
• Better technique vs. Scalability + Real-time
@kristw / interactive.twitter.com

105. Questions?

106. Thank you