4. Wooga has dedicated game teams
PHP Ruby Ruby
SQL SQL + Redis Redis
Cloud Cloud Bare Metal
5. Wooga has dedicated game teams
PHP Ruby Ruby
SQL SQL + Redis Redis
Cloud Cloud Bare Metal
6. Both games are similar in their setup
Flash client Ruby backend
7. Both games have similar problems
High traffic
• 200+ million h=p requests / day
High DB throughput
• 100.000 DB operaDons / second
Hardest problems to solve
• DB migraDons, DB failover, DB performance, ...
8. What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
9. Big cloud vs. bare metal
Big cloud: Amazon EC2
• new server in 5 minutes
• automated deployment
• load based setup
10. Big cloud vs. bare metal
Big cloud: Amazon EC2 Bare metal: Hetzner
• new server in 5 minutes • new server in 2-‐14 days
• automated deployment • automated deployment
• load based setup
11. What kind of app server are available?
XL CPU instance
• 8 virtual cores
• 8 GB RAM
12. Bare metal servers are more powerful
XL CPU instance EQ10 server
• 8 virtual cores • 12 “processing threads”
• 8 GB RAM • 24 GB RAM
13. Bare metal servers are more powerful
XL CPU instance EQ10 server
• 8 virtual cores • 12 “processing threads”
• 8 GB RAM • 24 GB RAM
=> 390,-‐ € / month => 200,-‐ € / month
14. And how is the network?
min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms
15. Want a fast network? Get your own!
min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms
max ping: 7.8 -‐ 59 ms max ping: 0.3 ms
avg ping: 0.8 -‐ 4.4 ms avg ping: 0.2 ms
16. Want a fast network? Get your own!
min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms
max ping: 7.8 -‐ 59 ms max ping: 0.3 ms
avg ping: 0.8 -‐ 4.4 ms avg ping: 0.2 ms
22 DB calls per hfp calls saved 13 -‐ 92 ms latency
17. What’s befer? Depends on your needs!
Big cloud is great if you
• don’t know your load
• scale up / down quickly
18. What’s befer? Depends on your needs!
Big cloud is great if you Bare metal is great if you
• don’t know your load • want to save money
• scale up / down quickly • need stable performance
19. What’s befer? Depends on your needs!
Big cloud is great if you Bare metal is great if you
• don’t know your load • want to save money
• scale up / down quickly • need stable performance
Say: “Don’t worry” Say: “Think ahead”
20. What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
21. Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis) Redis only
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
22. If MySQL is a truck, ...
Fast enough for reads
Can store on disk
Robust replicakon
h=p://www.flickr.com/photos/erix/245657047/
23. If MySQL is a truck, Redis is a Ferrari
Fast enough for reads Super fast reads/writes
Can store on disk Out of memory => dead
Robust replicakon Fragile replicakon
h=p://www.flickr.com/photos/erix/245657047/
24. Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis) Redis only
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
25. Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis) Redis only
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
MySQL has its hard limits Works fine if done right
26. How many tables to store a user in?
22 MySQL tables
7 Redis hashes
27. How many tables to store a user in?
22 MySQL tables 1 Redis Hash
7 Redis hashes
28. How many tables to store a user in?
22 MySQL tables 1 Redis Hash
7 Redis hashes
Easy to implement Less top level keys
Faster for small changes Export and import is easy
29. What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
30. Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
31. Redis storage type
Redis works in-‐memory Redis used vm
Dumps slow down master Dumps to disk impossible
• Dumping on slaves only • Own system to store
JSON files to disk
32. Redis storage type
Redis works in-‐memory Redis used vm
Dumps slow down master Dumps to disk impossible
• Dumping on slaves only • Own system to store
JSON files to disk
Redis became a cache only
33. Redis storage type
Redis works in-‐memory Redis now uses diskstore
Dumps slow down master
• Dumping on slaves only
34. Redis storage type
Redis works in-‐memory Redis now uses diskstore
Dumps slow down master • Experimental branch
• Dumping on slaves only • Stores async. to disk
35. Redis storage type
Redis works in-‐memory Redis now uses diskstore
Dumps slow down master • Experimental branch
• Dumping on slaves only • Stores async. to disk
Redis became a DB again
36. There are many inackve users
hourly 0,3M hourly 0,1M
daily 1,3M daily 0,4M
monthly 8,0M monthly 3,0M
lifetime 20,8M lifetime 6,0M
0M 8M 16M 24M 0M 2M 4M 6M
37. Not all users play at the same day
In-‐memory DB hierarchy
• Redis in-‐memory
• Redis diskstore on disk
100% of users in RAM 3% of users in RAM
38. What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
39. So, what didn’t work?
Big cloud Bare metal
Hybrid (SQL + Redis) Redis only
In-‐memory “only” DB hierarchy
40. So, what didn’t work?
Big cloud Bare metal
Hybrid (SQL + Redis) Redis only
In-‐memory “only” DB hierarchy
41. Let’s dig a bit deeper
1,300,000 daily users 400,000 daily users
5000 hfp calls/s peak 2500 hfp calls/s peak
100 ms response kme 10 ms response kme
35-‐70 servers 11 servers
43. There are 2 main reasons for higher cost
Databases Other
Other: 3x higher
DB: 20x higher
44. There are 2 main reasons for higher cost
Databases Other
Other: 3x higher
No DB hierarchy!
DB: 20x higher
(both RAM and cost)
45. So, what didn’t work?
Big cloud Bare metal
Hybrid (SQL + Redis) Redis only
In-‐memory “only” DB hierarchy
46. So, what didn’t work?
ds ...
Big cloud dep en Bare metal
Hybrid (SQL + Redis) Redis only
In-‐memory “only” DB hierarchy
47. So, what didn’t work?
ds ...
Big cloud dep en Bare metal
Hybrid (SQL + Redis) Redis only
nev er
In-‐memory “only” in! DB hierarchy
a ga
48. So, what didn’t work?
ds ...
Big cloud dep en Bare metal
Hybrid (SQL + Redis) er, Redis only
er, ev
In-‐memory “only” n ev n! DB hierarchy
ag ai
ev er
49. Next game will do things befer
Start in cloud and then go metal?
Redis only
DB hierarchy
50. BTW, wooga is hiring!
Q & A
Jesper Richter-‐Reichhelm
@jrirei
slideshare.net/wooga wooga.com/jobs
51. BTW, wooga is hiring!
Thank you
Jesper Richter-‐Reichhelm
@jrirei
slideshare.net/wooga wooga.com/jobs
