This episode of tutorial teaches you how to download tweets that include a set of keywords.

1. Five Steps to Search and Store
Tweets by Keywords
• Created by The Curiosity Bits Blog (curiositybits.com)
• With the support from Dr. Gregory D. Saxton
(http://social-metrics.org/ )

2. The output you will get…
Let’s say I want to study Twitter discussions of the missing Malaysian airliner
MH370. I plan to gather all tweets that include the keywords MH370 or
Malaysian.
You will get an ample amount of metadata for each tweet. Here is a breakdown
of each metadata type:
name Def.
tweet_id The unique identifier for a tweet
inserted_date When the tweet is downloaded into your database
language language
retweeted_status Is the tweet a RETWEET?
content The content of the tweet
from_user_scree
n_name
The screen name of the tweet sender

3. name Def.
from_user_followers_count The number of followers the sender has
from_user_friends_count The number of users the sender is following
from_user_listed_count How many times the sender is listed
from_user_statuses_count The number of tweets sent by the sender
from_user_description The profile bio of the sender
from_user_location The location of the sender
from_user_created_at When the Twitter account is created
retweet_count How many times the tweet is retweeted
entities_urls The URLs included in the tweet
entities_urls_count The number of URLs included in the tweet
entities_hashtags The hashtags included in the tweet
entities_hashtags_count The number of hashtags in the tweet
entities_mentions The screen-names mentioned in a tweet

4. name Def.
in_reply_to_screen_name The screen name of the user who is replied to
by the sender
in_reply_to_status_id The unique identifier of a reply
entities_expanded_urls Complete URLs extracted from short URLs
json_output The ENTIRE metadata in JSON format,
including metadata not parsed into columns
entities_media_count NA
media_expanded_url NA
media_url NA
media_type NA
video_link NA
photo_link NA
twitpic NA

5. Step 1: Checklist
• Do you know how to install necessary Python
libraries? If not, please review pg.8 in
http://curiositybits.com/python-for-mining-the-social-web/python-
tutorial-mining-twitter-user-profile/
• Do you know how to browse and edit SQLite
database through SQLite Database Browser? If not,
please review pg.10-14 in http://curiositybits.com/python-for-
mining-the-social-web/python-tutorial-mining-twitter-user-profile/
Download the code
https://drive.google.com/file/d/0Bwwg6GLCW_I
Pdm1mcHNXeU85Nkk/edit?usp=sharing

6. Have you installed these necessary
Python libraries?
Step 1: Checklist

7. Step 1: Checklist
Most importantly, we need to install a Twitter mining
library called Twython
(https://twython.readthedocs.org/en/latest/index.html)

8. Step 2: enter the search terms
You can enter multiple search terms, separated by comas. Please notice
that the last search term ends by a coma.
You can enter non-English search terms. But make sure the Python
script starts by the following block of code:

9. Step 3: enter your API keys
API Key
API secret
Access token
Access token secret
Enter the key inside the quotation marks

10. Step 3: enter your API keys
• Set up your API keys - 1
First, go to https://dev.twitter.com/, and sign in your
Twitter account. Go to my applications page to create
an application.

11. Step 3: enter your API keys
• Set up your API keys - 2
Enter any name that makes sense to you
Enter any text that makes sense to you
you can enter any legitimate URL, here, I put in the URL of my institution.
Same as above, you can enter any legitimate
URL, here, I put in the URL of my institution.

12. Step 4: change the parameter
result_type defined by the Twitter API Documents. Now, we
set it to recent, we can also set it to mixed or popular.

13. Step 4: change the parameter
Here is a list of parameters you can tweak or add:
https://dev.twitter.com/docs/api/1.1/get/search/tweets
For example, if you want to limit the search to Chinese, you
can add lang = ‘zh’

14. Step 4: change the parameter
For another example, if you want to limit the search to all
tweets sent until April 1 of 2014. You can add until = ‘2014-
04-01’

15. Step 5: set up SQLite database
• When you type in just a file name, the database will be
saved in the same folder with the Python script. You can
use a full file path such as
sqlite:///C:/xxxx/xxx/MH370.sqlite.

16. Hit RUN!

17. If you run the script daily or twice a day, you should be
good enough to cover all tweets generated on that day,
and tweets a few days old.
But, historical tweets are EXPENSIVE! Tweets older than
a week can be purchased through http://gnip.com/
Are we getting all the tweets?