Trunk.ly launched as a half baked product when Yahoo announced its plan to "sunset" delicious. Within 2 days, 5000 users sign up with the service. In order to deal with the huge demands, we spun off 10+ EC2 instances to crunch through the links. Alex rewrote the crawler architecture using gevent and redis. Reducing the total number of servers to 2. Saving $3000 per month. This talk is about the framework and architecture used in the rewrite.

1. Writing a distributed crawler system using gevent and redis Alex Dong @alexdong

3. Async 101

4. Gevent: the monkey king

5. Redis: data structure server

6. Lessons learned

8. How many links today?

9. What was the project Google employee #1 working on?

11. DNS resolution

12. Request HTTP Header

13. Download full content

14. Store to local file store, database and index

15. Scheduling, throttling, status monitoring, scale up by flicking on more servers.

17. Yield on IO/socket, but

19. Context switch overhead

21. Cooperative multitasking

22. epollfd = epoll_create ();

23. epoll_ctl ( epollfd , EPOLL_CTL_ADD, listen_sock , &ev)

24. epoll_wait ( epollfd , events , MAX_EVENTS, -1)

28. Working: Hashtable - HSET, HDEL, HEXISTS

29. One and only one: Sets - SADD, SPOP

32. 25% code for status update and monitoring

34. DNS is down?

35. ISP's bandwidth?

36. Large file download?

37. Scheduler re-submit tasks?

40. Carefully profile to make sure socket.recv becomes the main bottleneck.

41. The get_title crisis

42. Rule of thumb: load average < 1 to saturate 10M bandwidth

43. Twitter: @alexdong trunk.ly/?q= from:alexdong + gevent