How to Handle 1,000,000 Daily Users Without Using a Cache (RailsWayCon 2012)

HOW TO HANDLE
1,000,000 DAILY USERS
(without using a cache)

Jesper Richter-‐Reichhelm, @jrirei
Tuesday, June 5, 2012

The overall architecture is not that complex

Flash client Backend



Flash client

Game Session
Asynch. Communica2on



Backend

State Changes
ValidaMon
Persistence


But the scale is interesMng

14 billion requests / month




>100,000 DB operaMons / second




>100,000 DB operaMons / second

>50,000 DB updates / second


A journey to 1,000,000 daily users

Start of the journey
6 weeks of pain
Paradise
Conclusion


October 2009: wooga’s ﬁrst simulaMon game


Instead of PHP we used Ruby


Our database was MySQL


Our database was MySQL

even user ids odd user ids


And we went into the cloud


Master-‐slave replicaMon for DBs worked ﬁne

lb

app app app

db db


We added a few applicaMon servers over Mme

lb

app app app app app app app app app

db db


250K daily users and no problems
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"

#!!$!!!"
Life was good

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


Life was well and I went on a nice vacaMon
TO DO

<picture: Jesper in clot
canyon>


Our bane: MySQL hiccups

(!!"#

'!"#

&!"#

%!"#

$!"#

!"#
!# )# (!# ()# $!# $)# *!# *)# %!#


SQL queries generated by Rubyamf gem

AMF responses to Flash client



Wrong conﬁg...
... so associated data was included, too



Wrong conﬁg...
... so associated data was included, too

=> Easy to ﬁx


More traﬃc using the same cluster

lb


db db


Config tweaks brought us to 300K DAU
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"

#!!$!!!"
Config fixes

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


AcMveRecord’s checks caused 20% extra DB

Checking connecMon state
MySQL process list full of ‘status’ calls


AcMveRecord’s checks caused 20% extra DB

Checking connecMon state
MySQL process list full of ‘status’ calls

=> Fixed by 1 line of code


I/O on MySQL masters sMll was the bojleneck

New Relic: 60% of all UPDATEs on ‘Mles’ table


Tiles are part of the core game loop

Core game loop
1) plant
2) wait
3) harvest


We started to shard on model, too

Adding new shards

old old
master slave



Adding new shards
1) Setup new masters as slaves of old ones

old old new
master slave master



Adding new shards
1) Setup new masters

old old new new
master slave master slave



Adding new shards
2) Start using new masters

old old new new



Adding new shards
3) Cut replicaBon

old old new new



Adding new shards
3) Cut replicaBon
4) Truncate

old old new new


4 DB masters and a few more servers

lb

app app app app app app app app

app app app app app app app app

Bles Bles
db db
db db


Sharding by model brought us to 400K DAU
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"

#!!$!!!"
Shard by model

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


We improved our MySQL setup

RAID-‐0 of EBS volumes




Using XtraDB




Using XtraDB

Tweaking my.cnf


Sharding gem circumvented AR’s internal cache

AcMveRecord caches SQL queries...




... only in our development environment!




... only in our development environment!

=> Fixed by 2 lines of code


I/O sMll was not fast enough

If 2 + 2 is not enough, ...


I/O sMll was not fast enough

If 2 + 2 is not enough, ...

… perhaps 4 + 4 masters will do?


It’s no fun to handle 8+8 MySQL DBs

lb



Bles Bles
db db
db db


It’s no fun to handle 8+8 MySQL DBs

lb



Bles Bles Bles Bles
db db db db
db db db db


At 500K DAU we were at a dead end
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


I/O remained bojleneck for MySQL UPDATEs

Each DB master could do
about 1000 DB write/s.


I/O remained bojleneck for MySQL UPDATEs

Each DB master could do
about 1000 DB write/s.

That’s not enough!


Pick the right tool for the job!


Redis is fast but goes beyond simple key/value

Redis is a key-‐value store
Hashes, Sets, Sorted Sets, Lists
Atomic operaBons like set, get, increment


Redis is fast but goes beyond simple key/value

Redis is a key-‐value store
Hashes, Sets, Sorted Sets, Lists
Atomic operaBons like set, get, increment

50,000 transacMons/s on EC2
Writes are as fast as reads


We could learn from another team using Redis


Shelf Mles : An ideal candidate for Redis

Shelf 2les:
{ plant1 => 184,
plant2 => 141,
plant3 => 130,
plant4 => 112,
… }


Shelf Mles : An ideal candidate for using Redis

Redis Hash
HGETALL
HGETSET
HINCRBY
…


On-‐demand migraMons from MySQL to Redis


Typical migraMon throughput over 3 days


Migrate on the ﬂy -‐ and clean up later

1. Let migraMon run unMl everything cools down



2. Migrate the rest manually



3. Remove migraMon code



4. Wait unMl no fallback necessary



4. Wait unMl no fallback necessary
5. Remove SQL table


A journey to 1,000,000 daily users

Start of the journey
6 weeks of pain
Paredise (or not?)
Conclusion


Again: Tiles are part of the core game loop

Core game loop
1) plant
2) wait
3) harvest


Size majers for migraMons



MigraMon check overload



MigraBon only on startup




Overlooked an edge case




Only migrate 1% of users




Only migrate 1% of users
ConBnue if everything is ok


In-‐memory DBs don’t like dumping to disk

Dumping to disk



Dumping to disk
SAVE is blocking



Dumping to disk
SAVE is blocking
BGSAVE needs free RAM



Dumping to disk
SAVE is blocking

Latency increase by 100%



Dumping to disk
SAVE is blocking

Latency increase by 100%

=> BGSAVE on slaves every 15 minutes


Redis replicaMon starts with a BGSAVE

StarMng up a new slave by replicaMon



BGSAVE on master



BGSAVE on master
Slave imports dumped ﬁle



BGSAVE on master
Slave imports dumped ﬁle

=> No RAM means no new slaves


Redis had a memory fragmenMon problem

44 GB

in 8 days

24 GB



38 GB
in 3 days
24 GB



2.2
in v
xe d
F i 38 GB
in 3 days
24 GB


If MySQL is a truck
Fast enough
Disk based
Robust

Fast enough disk based robust


If MySQL is a truck, Redis is a race car
Super fast
RAM based
Fragile

Super fast RAM based fragile


Big and staMc data in MySQL, rest goes to Redis

256 GB data 60 GB data
10% writes 50% writes
h"p://www.ﬂickr.com/photos/erix/245657047/


Lots of boxes, but automaMon helps a lot!

lb lb

app app app app app app app app app app app app app



db db db db db redis redis redis redis redis


We reached 1 million daily users!
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"
1,000,000 -‐ Big party!

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


We started archiving inacMve users
&$!!!$!!!"

%$#!!$!!!"

50% DB
%$!!!$!!!"
reduc2on

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


We even survived a complete data center loss
&$!!!$!!!"

EBS no
%$#!!$!!!"
more!

%$!!!$!!!"

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


We improved our MySQL schema on-‐the-‐ﬂy
&$!!!$!!!"

30% DB
%$#!!$!!!"
reduc2on

%$!!!$!!!"

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


Meanwhile we have more than 2M daily users
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


EvoluMon every week

EVOLUTION
of sotware
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


EvoluMon every week

REVOLUTION
of sotware
&$!!!$!!!"

%$#!!$!!!"

%$!!!$!!!"

#!!$!!!"

!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


Works for teams ...


Each new game is a revoluMon


Works for teams and for companies

!""#$%&"'()"*+,

Ques2ons?

Jesper Richter-‐Reichhelm
@jrirei

slideshare.net/wooga
wooga.com/jobs


How to Handle 1,000,000 Daily Users Without Using a Cache (RailsWayCon 2012)

Recomendados

Recomendados

Más contenido relacionado

Más de Wooga

Más de Wooga (20)

Último

Último (20)

How to Handle 1,000,000 Daily Users Without Using a Cache (RailsWayCon 2012)