The story and process behind visualizing Twitter data as told by TweetStats creator, Damon Cortesi (@dacort).

1. TWITTER
AWESOMENESS
Damon P. Cortesi

2. @DACORT
By Day By Night
Founder/Director of Twitter Addict, including
Alchemy Security
TweetStats
Web app security Twitter Timeline Viz
Security Product Dev TweepSearch
Twitter Profile Search
Firewall Visualization

3. TWEETSTATS?
GRAPHING TWITTER SINCE 2007
Visualize your Twitter timeline - tweetstats.com
Tweets per month/hour
Top replies and apps
TweetCloud
Trend monitoring since September ’08
Global Twitter usage since December ’08
Over 250M tweets (no content) logged
Thanks, Gnip (http://gnip.com)

4. CHALLENGES
1. Data, data, data
Twitter allows retrieval of up to 3,200 tweets/user
Whitelisting allows 20,000 API queries/hour
Not uncommon now for 20k + tweets
Disparate - timeline vs. friends/followers
2. User experience
It takes time to gather data, graph

5. TWEETSTATS V1
OMG, SCREENSCRAPING

6. TWEETSTATS V2
BEAUTIFUL GRAPHS

7. GRAPH LIBRARIES
JAVASCRIPT
JavaScript
Honestly...
Kind of ugly
Limited functionality
Limited support

8. GRAPH LIBRARIES
FLASH
Flash - Interactive, beautiful
TweetStats currently uses FusionCharts Free edition
Gantt charts in disguise!!

9. BUT WAIT...
Raphaël JS - Hot new JavaScript graphing library
Vector Library - SVG/VML
Impressive capabilities
http://raphaeljs.com

10. TWEETSTATS V2
TWEETCLOUD

11. TWEETSTATS V2
TWEETCLOUD VIA WORDLE

12. USER EXPERIENCE
Average Twitter user attention span = seconds
Keep them informed
Queues and Quotes and Status Bars, oh my!

13. USER EXPERIENCE
BACKGROUND QUEUES
Generate XML data
asynchronously
Activate charts via JavaScript

14. USER EXPERIENCE
K.I.S.S.
Tons of data, easy to get lost
Lots of features TweetStats _could_ have...
Important to keep the experience simple
And this doesn’t pay the bills ... yet. ;)
TweetStats v3 - More features? More analytics?

15. SCALING TWEETSTATS
The story of the Blockheads ...

16. SCALING TWEETSTATS
WHY IT BROKE?
One server (affectionately named after @darkgracie)
Web server + Rails stack
Database w/150,000 tables
= #FAIL
Various other applications
Synchronous DB queries
Thousands of NKOTB Fans

17. SCALING TWEETSTATS
TYPICAL TWITTER TRAFFIC
Peaks and valleys
Influential tweet = immediate need for resources

18. SCALING TWEETSTATS
“FIXING” IT
Nods to:
Amazon EC2
Scalr.net
Architecture: Scaled!
Wallet: Empty.

19. POST-NKOTB
New avatar for @TweetStats ->

20. DATA MINING
Other services - TweepSearch, TweetSum
Background Queues are King
TweetStats - BackgroundRB
TweepSearch - Workling
TweetSum - BackgroundJob
Why so many? Can’t judge without experience.

21. DATA MINING
VOLUME
TweepSearch - ~4.5M Twitter Profiles Indexed
Just a small fraction of actual users
TweetStats - Global twitter usage via Gnip
Previously - 2M updates/day
Now - ~10M updates/day
~9,000 updates/minute
aka ... my servers hate me.

22. DATA MINING
OPTIONS
Brute Force (historical)
Crawl profiles, timelines
Twitter Search - 1,500 results or 20-30(??) days
Official Twitter Streaming API
“spritzer”, “gardenhose”, “firehose” streams
Gnip - Not just Twitter!

23. Other Glances
Friends and Followers

24. Other Glances
Friends and Followers

25. Other Glances
Friends and Followers

26. Other Glances
Friends and Followers

27. Other Glances
Friends and favorites

28. Other Glances
Friends and favorites

29. Other Glances
Friends and favorites

30. THANKS
TweetStats - Graph your Twitter Timeline
TweepSearch - Profile Search Awesomeness
Damon P. Cortesi
@dacort
dacort@tweetstats.com