Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Consuming the Twitter Streaming API with Ruby and MongoDB
1. Consuming the Twitter Streaming API
with Ruby and MongoDB
Jeff Linwood
Lone Star Ruby Conference V
August 12, 2011
http://www.jefflinwood.com
@jefflinwood
Friday, August 12, 2011
2. Goals
• Watch for any
tweet that contains
certain keywords
• Store those tweets
into the MongoDB
database
Friday, August 12, 2011
4. Twitter Streaming API+
Mongo DB
• Twitter Streaming API
• MongoDB
• Not a web application
Friday, August 12, 2011
5. The many APIs of
Twitter
• Twitter Streaming API
• User Streams
• Site Streams (for the big boys)
• REST API
• Search API
Friday, August 12, 2011
6. Twitter Streaming API
• Keywords
• Users
• Locations
Great photo is by rachel_thecat
http://www.flickr.com/photos/23209605@N00/2786126623/
Friday, August 12, 2011
7. TweetStream gem
by Michael Bleigh
https://github.com/intridea/tweetstream
Friday, August 12, 2011
8. Connecting to the
Twitter Streaming API
• JSON responses
• HTTP Basic Authentication
• One stream per account (dev/prod)
• Leave it open!
• Don’t constantly reconnect, and if you do,
back off
Friday, August 12, 2011
9. Limitations of Twitter
Streaming API
• 400 Keywords
• 5,000 User Ids
• 25 Location Boxes
• Can ask Twitter for increased access
Friday, August 12, 2011
11. Intro to MongoDB
• NoSQL - what does that mean?
• Great fit for JSON-oriented applications
• If you don’t know your schema in advance
• Query language
• Map Reduce
Friday, August 12, 2011
12. Storing data in MongoDB
• Native format of MongoDB is BSON,
similar to JSON
• Connect to a database (similar to MySQL)
• Connect to a collection (created if it
doesn’t exist)
• Insert JSON (in our case, a tweet)
Friday, August 12, 2011
14. Considerations for
MongoDB
• Tweets - very verbose JSON
• Date format in Tweets not same as
MongoDB
• May want to pre-process Tweets
• Can use both MongoDB and MySQL in
same app if you want
Friday, August 12, 2011
15. Here’s the code
• Okay, the whole thing is really done in
about three lines.
Friday, August 12, 2011
16. Where do you run this?
• Command line
• Your own server
• Heroku + MongoLab
• Other cloud services
Friday, August 12, 2011