QCon NYC 2012: Twitter's Real Time Architecture

1. real-time delivery architecture @raffi qcon nyc 2012

2. http://twitpic.com/135xa - There’s a plane in the Hudson. I’m on a ferry going to pick up the people. Crazy. 15 Jan 09 Janis Krums @jkrums

4. this is barack - let's get this started! -bo 24 May 12 The White House @whitehouse

5. what are the goals? ⇢ evolve from being solely a web stack

6. ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock + Haplo Monorail Darkwing Flock(s)

7. what are the goals? ⇢ evolve from being solely a web stack ⇢ isolate responsibilities and concerns ⇢ site speed and reliability ⇢ developer innovation speed

9. Pull Push Targeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.) Queried Search API Track / Follow Streams

13. Write API Ingester Fanout Batch Compute Timeline Cache Push Compute Search Cache HTTP Push Redis Redis Redis Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Service

14. Write API Social Ingester Fanout Graph Service insert Batch Compute Timeline Cache Push Compute Search Cache HTTP Push Redis Redis Redis Redis Hadoop Earlybird Redis ⇢ keyed off Mobile Push “recipient” Timeline ⇢ pipelined 4k Blender Service “destinations” at a time ⇢ replicated

15. Write API Ingester Fanout using redis Batch Compute Timeline Cache Push Compute Search Cache Tweet IDPush User ID HTTP Bits Redis Redis Redis Redis Hadoop Earlybird Redis 8 bytes 8 bytes 4 bytes ⇢ native list Mobile Push structure Timeline ⇢ RPUSHX to Blender Service only add to cached timelines

16. Write API Ingester Fanout using redis Batch Compute Timeline Cache Push Compute Search Cache Tweet IDPush User ID HTTP Bits Redis Redis Redis Redis Tweet ID User ID Bits Hadoop Earlybird Redis ⇢ native list Mobile Tweet ID User ID Bits Push structure Tweet ID User ID Bits Tweet ID User ID Bits Timeline ⇢ RPUSHX to Blender Tweet ID User ID Bits Service Tweet ID User ID Bits only add to Tweet ID User ID Bits cached Tweet ID User ID Bits timelines Tweet ID User ID Bits

17. Write API Ingester Fanout Batch Compute Timeline Cache Push Compute Search Cache HTTP Push Redis Redis Redis Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Service

19. Write API Ingester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Service

20. Write API Ingester Fanout blender Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis ⇢ queries one Mobile Push replica of all Blender Timeline Service indexes ⇢ merges & ranks results

24. http push / hosebird ⇢ maintains persistent connections with end clients ⇢ processes tweet & social graph events ⇢ event-based “router”

25. Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Follow event propagation ⇢ write API sends all events into hosebird; sees content creation events, social graph changes, etc. ⇢ different queues for public tweets, protected tweets, social events, etc.

26. Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Follow event cascading ⇢ bandwidth management ⇢ simultaneous connection management (~1m long lived & open connections to this cluster)

27. Hosebird Firehose Write API Hosebird User Streams Hosebird Track / Follow ﬁrehose ⇢ edge machine simply outputs the public tweet queue ⇢ only allow a limited number of ﬁrehoses per hosebird box for bandwidth management

28. Hosebird Firehose Write API Hosebird Track / Follow Hosebird User Streams track / follow ⇢ simple query based on tweet content ⇢ keeps list of terms / users of interest ⇢ parses public tweets at the edge, and if term matches a token, or user is of interest, then route

29. Hosebird Firehose Write API Hosebird Track / Follow Hosebird User Streams user streams ⇢ replicate home timeline experience ⇢ upon login, obtain “following” list ⇢ keep cached following list coherent by seeing social graph updates ⇢ route tweet if from a followed user

33. Write API Ingester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Social Mobile Graph Push Service Timeline Blender Service

34. Write API Ingester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Social Mobile Graph Push Service Timeline Blender Service

39. Synchronous Path Write API Ingester Fanout Asynchronous Path Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Query Path Service

42. Write API Read Path Write Path Ingester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Service

43. Write API Read Path Write Path Ingester Fanout Batch Compute Timeline Cache Push Compute Search Index HTTP Push Redis Redis Earlybird Redis Hadoop Earlybird Redis Mobile Push Timeline Blender Service

44. things we’re trying...

46. Write API Ingester Fanout Timeline Cache Redis Search Index Redis Earlybird Redis Earlybird Redis

47. Write API Ingester Fanout Timeline Cache Redis Search Index Redis Earlybird Redis Earlybird Redis search index fanout index ⇢[‘hello’,‘world’] ⇢[@danadanger, ...]

48. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi

49. User Intent Query Expansion “Hello, world” “Hello” AND “world” user_timeline:nelson @raffi’s home timeline OR user_timeline:danadanger

50. Write API fan-in fan-out ⇢ O(1) write ⇢ O(n) write Ingester Fanout ⇢ O(n) read ⇢ O(1) read Timeline Cache Redis Search Index Redis Earlybird Redis Earlybird Redis

51. User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi

52. User Intent Query Expansion “Hello, world” “Hello” AND “world” home_timeline:raffi @raffi’s home timeline OR user_timeline:taylorswift13

53. streaming compute ⇢ continuous computation ⇢ driven by the events that come into twitter ⇢ generalizing the push mechanism

55. timeline query statistics ⇢ >150m active users worldwide ⇢ 300k qps poll-based timelines @ 1ms p50 / 4ms p99 ⇢ 30k qps search-based timelines

56. tweet input ⇢ ~340m tweets per day ⇢ ~4K/sec daily average ⇢ ~6K/sec daily peak ⇢ >10K/sec during large events

58. followed by following

59. timeline delivery statistics ⇢ 26b deliveries / day (~18m / min) ⇢ 3.5 seconds @ p50 to deliver to 1m ⇢ ~300k deliveries / sec

60. thanks!

QCon NYC 2012: Twitter's Real Time Architecture

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (6)

Más de Raffi Krikorian

Más de Raffi Krikorian (20)

Último

Último (20)

QCon NYC 2012: Twitter's Real Time Architecture