2. WHAT IS FLITE?
• Display advertising platform
• Create and publish rich,
interactive ads
• Serve ad impressions
• Collect and analyze metrics,
batch AND realtime
3. WHAT IS FLITE?
• Display advertising platform
• Create and publish rich,
interactive ads
• Serve ad impressions
• Collect and analyze metrics,
batch AND realtime
with
4. REALTIME YOU SAY?
• Realtime monitoring of ad performance
• Debugging of ads, with instant feedback on triggered events
6. WRITING TO REDIS
Persistence: Up to 4 hours for most data
Amount: ~30 writes per event, pipelined
Granularity: By second, minute, hour
Write Types:
• HSET - JSON blobs (Event body data)
• LPUSH - Lists of events (Events by session)
• HINCRBY - Simple counters (Events by ad)
• ZINCRBY - Sorted set counters (To retrieve “Top 100”)
• PUBSUB - Stream event data (Debugger)
7. READING FROM REDIS
Redis transactions are used extensively like multi, exec
Read types:
• HGETALL - get all data for event
• HGET - get event counts by ad
• LRANGE - list of all events by session
• ZREVRANGE - top 100 ads with highest number of events
• SUBSCRIBE
8. AND ALL OF THIS AT SCALE
• Daily traffic peaks: 100k - 200k events per minute
• Peaks are really plateaus that last for hours
• Read load is negligible by comparison, but reads must be fast
• In fact, everything must be fast: <1 sec latency for debugger to work
9. WHAT’S WRONG WITH THIS PICTURE?
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js
10. WHAT’S WRONG WITH THIS PICTURE?
Bottleneck!
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js
13. A QUICK SOLUTION?
• MOAR Megahurtz!!: m2.2xl is already about as fast as Amazon gets.
• Redis-As-A-Service: Expensive, not fast enough for even our usual load.
• twemproxy: Twitter’s sharding solution.
Doesn’t support all commands: PING, MULTI, INFO, MGET, etc…
14. WHAT ABOUT JEDIS’S NATIVE SHARDING?
• No pipelining
• No pubsub
• Complicated consistent hashing mechanism makes reading in other
environments more difficult
15. LET’S ROLL OUR OWN!
Goals: Speed, speed, speed
Not Goals: Fault tolerance, redundancy, resiliency
16. HOW HARD CAN IT BE?
Sharding method: Java hashCode of key for every item written
JVMad
event items
node.js
Write to many Read from one
17. WHAT HAPPENED?
Before Sharding After Sharding
Items per Event 30 30
Items written per Event per Redis 30 10
Redis Connections per Event 1 3
Connections per second per Redis box n n
Reality: When n gets to around 500, Redis maxes out CPU and starts rejecting connections.
Theory: Since Redis claims to be able to handle 70k connections per second, the amount of
data being sent per connection is the problem.
19. TAKE TWO
Sharding method: Java hashCode of EVENT key for every item written.
A single key now lives on multiple Redis boxes
JVMad
event items
node.js
Write to one Read from many
20. BETTER!
Before Sharding After Take 1 After Take 2
Items per Event 30 30 30
Items written per Event per Redis 30 10 30
Redis Connections per Event 1 3 1
Connections per second per Redis box n n n/3
More load can be easily accommodated by adding boxes
21. CODING CHALLENGES
Java
• Managing multiple connection pools
• Managing multiple pipelines
• Automatic health checks
node.js
• Finding hashing function that works in different environments
• Managing multiple pipelines
• Fanout requests and merging response once pipeline is
executed
22. SINGLE-REDIS JEDIS WORKFLOW
On application startup:
1. Initialize jedisPool with connection info
Every time:
1. Jedis jedisClient = jedisPool.getClient();
2. Pipeline pipeline = jedisClient.pipelined();
3. State your business
4. pipeline.sync();
5. jedisPool.returnResource(jedisClient);
23. SHARDED JEDIS WORKFLOW
On application startup:
1. Initialize n jedisPools with connection info
Every time:
1. Jedis jedisClient = jedisPool.getClient();
2. Pipeline pipeline = jedisClient.pipelined();
3. State your business
4. pipeline.sync();
5. jedisPool.returnResource(jedisClient);
24. SHARDED JEDIS WORKFLOW
On application startup:
1. Initialize n jedisPools with connection info
Every time:
1. Jedis jedisClient = jedisPool.getClient(); Which pool?
2. Pipeline pipeline = jedisClient.pipelined(); Which client?
3. State your business To whom?
4. pipeline.sync(); Which pipeline?
5. jedisPool.returnResource(jedisClient); Return what where?