1. Twitter: The Tweet, the Whole Tweet,
and Nothing but the Tweet #2
charsyam@naver.com

2. Data Mining

3. Discover New Knowledge

4. Discover New Knowledge
from Existing Information

5. What do #TeaParty and
#JustinBieber
have in common

6. Tools: Pymongo, MongoDB
apt-get install python-dev
pip install pymongo

7. Get Tweets
from pymongo.connection import Connection
import sys
import tweepy
connection = Connection("localhost")
db = connection.foo
import tweepy
api = tweepy.API()
tweets = api.search('#JustinBieber', rpp=100)
for tweet in tweets:
db.foo.save(tweet.__getstate__())

8. Insert TO MongoDB
from pymongo.connection import Connection
import sys
import tweepy
connection = Connection("localhost")
db = connection.foo
import tweepy
api = tweepy.API()
for num in range(1,16):
tweets = api.search('#JustinBieber', rpp=100, page=num)
for tweet in tweets:
db.foo.save(tweet.__getstate__())

9. Count Frequency in mongo
MAP
map = function(){
words = this.text.split(' ');
for ( i in words ){
emit({ key: words[i] }, {count: 1});
}
};

10. Count Frequency in mongo
REDUCE
reduce = function (key, values) {
var count = 0;
values.forEach(function (v) {count += v.count;});
return {count:count};
}

11. Count Frequency in mongo
EXECUTE
res = db.foo.mapReduce( map, reduce, {out: "mystring"});

12. Count Frequency in mongo
RESULT
{ "_id" : { "key" : "#1000ADay" }, "value" : { "count" : 1 } }
{ "_id" : { "key" : "#1000aday" }, "value" : { "count" : 1 } }
{ "_id" : { "key" : "#500ADay" }, "value" : { "count" : 1 } }
{ "_id" : { "key" : "#500aday" }, "value" : { "count" : 1 } }
{ "_id" : { "key" : "#AutoFollow" }, "value" : { "count" : 1 } }
{ "_id" : { "key" : "#Bieber" }, "value" : { "count" : 1 } }

13. Get From MongoDB
from pymongo.connection import Connection
import sys
import tweepy
connection = Connection("localhost")
db = connection.foo
cursor = db.mystring.find()
for d in cursor:
print d

14. What Entities Co-Occur
Most Often with
#JustinBieber and
#TeaParty Tweets?

15. intersection
import sys
from sets import Set
if __name__=='__main__':
r1 = open( sys.argv[1] )
r2 = open( sys.argv[2] )
s1 = Set()
s2 = Set()
for line in r1.readlines():
key = line.split()
if( len(key) > 0 ):
s1.add(key[0])
for line in r2.readlines():
key = line.split()
if( len(key) > 0 ):
s2.add(key[0])
s3 = s1.intersection(s2)
print len(s1)
print len(s2)
print len(s3)

16. On Average, Do #JustinBieber
or #TeaParty Tweets Have
More Hashtags?

17. Which Get Retweeted More
Often: #JustinBieber or
#TeaParty?

18. How Much Overlap Exists
Between the Entities of
#TeaParty and
#JustinBieber Tweet?

19. Thank You!