Games in social networks offer interesting scaling challenges, as the user base can grow explosively once they become popular. When wooga started work on the facebook game 'Happy Hospital' in early 2010, we set ourselves an ambitious goal: "Let's try to scale it to a million players per day -- with as few servers as possible." This constraint led us to select Redis as the main backend storage, running on dedicated (but cheap, consumer-grade) hardware. As an in-memory database, Redis offers an order-of-magnitude reduction in query latency, but also introduces new challenges in the area of durability. This case study will recount how our backend architecture has evolved over time, highlight the difficulties we ran into, describe how they shaped our final setup ... and how we came up with a new Redis persistence model (diskstore) on the way. Find out how many machines you *really* need for a high-traffic game backend -- even using a 'slow' language like Ruby.

1. Redis to the Rescue?
Berlin Buzzwords
2011-06-07

2. Who
• Tim Lossen / @tlossen
• backend developer at wooga
• Berlin, Germany

4. apps.facebook.com/happy-hospital

5. Goals
• 1 million players per day
(“DAU = daily active users”)

6. Goals
• 1 million players per day
(“DAU = daily active users”)
• small production cluster
- less operational overhead
- cheaper

7. Architecture

9. Data Model
• user = single Redis hash
- each entity stored in hash field
(serialized to JSON)
• custom Ruby mapping layer
(“Remodel”)

10. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

11. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

12. Setup A (July 2010)

13. Setup B (Nov 2010)

14. Setup B (Dec 2010)

15. Analysis
• Redis virtual memory not
compatible with:
- persistence
- replication

16. Setup B’ (Dec 2010)

17. Setup C (Dec 2010)

18. Setup D (May 2011)

19. Status
• peak traffic:
- 3000 requests/s
- 35000 Redis operations/s
• average backend response time:
- 10 ms

20. appdata.com

21. Goals
• 1 million DAU
• small production cluster

22. Production Cluster
Text
(1x intel core i7 X980,
6 cores, 24 gig ram)

23. Goals
• 1 million DAU
• small production cluster

24. Redis Highlights
• high sustained throughput /
performance
• very stable
• active & helpful community

25. Redis Gotchas
• memory consumption
- fragmentation!
• durability

26. Q&A

27. redis.io
github.com/tlossen/remodel
wooga.com/jobs