www.nimbl3.com

1. How to scale your
platform when you
are not Facebook
2013 Carlos Herrera
caherrerapa@gmail.com

2. CTO - Binumi, former Head of Product and IT –
Lazada TH (Rocket Internet)
MBA and B. Computer Science
7 countries
10 years in tech. Contributed to some OSS
Product Management, Software dev, ethical
hacking, corporate governance
Coca Cola FEMSA, Infosys, among others
Who am i?

3. Be humble
“Normal” scale platforms are different.
–Customer paying vs Like a cat
–Sensitivity*
–Structured information vs Unstructured
–Concurrency
–Reach across regions
–GBs vs Several TB or PB
Why I shouldn’t do exactly as fb?

4. You scale a platform not a language*
Language selection drivers
– Problem
– Maturity
– Community
– Talent pool
Nice to experiment but focus on the
problem (yes, im talking about you
mongodb guys)
Different solutions / Creativity

5. No silver
bullet

6. Scaling in 7
steps

7. How does it start?
Everything on the
same server
Reading the DB
all the time
JS, CSS, Images
served from your
server

8. 1. Define, measure, benchmark
b. Measure
Munin
Icinga
Statsd
New Relic $
Pingdom $
c. Benchmark
Siege
Jmeter
Sysbench
a. Baremetal vs Cloud (Amazon)
Physical vs Virtualized
Powerful vs Flexible
Normal skills vs Hard core skills
Support vs On your own
Outages?
Avoid anything with Cpanel for God’s sake
International gateway

9. Cache?
Set content in memory/drive
Faster than DB
Key = Value
TTL
Memcached (> 1 server), APC (1
server)
Other Common use: sessions
Extremely important in cloud
Live without cache
Constant hits to DB
DB easily the bottleneck
Wasting $$ you paid for memory
2. Cache system

10. 2. Cache system
a. Customer goes to your website’s homepage
b. The page requested needs to load a list of
products
c. Is the list of products in the cache by the key
“XXXXX”?
a. Yes. Retrieve from cache using key
“XXXXX”, use it and return page.
b. No. Go to the Database, perform the query
and save it in cache for later, use the info
and return the page.
Introducing
Cache

11. Content Delivery
Network
Static files (CSS, JS, HTML,
JPG)
Amazon, Rackspace,
Cloudflare, Akamai
CDN
< Page Load time
< Server load (important
in cloud)
Inexpensive
> Automation
3. Content Delivery Network

12. 4. Decouple and revisit your
web node Separation of concerns
Webnodes apart from DB
More visibility
Easier to scale
Horizontal
Vertical
Evaluate Nginx, Puma(RoR)
PHP-FPM

13. 5. Adding web nodes / load balancing
Load Balancer
Sends traffic to internal web nodes
Easier to pay. ELB*, Rackspace
No budget? Nginx, HAProxy
Location depends if you pay or you
build it*
www.yourapp.com
web01.yourapp.com
web02.yourapp.com

14. Beware of the session
management
If you don’t use sticky sessions
be careful you can end up not
remembering your customers as
the load balancer sends traffic
to the less busy
5. Adding web nodes / load balancing

15. 6. Scaling the database:
vertical
Vertical
Bigger server
RDS (downtime)
Easier
But there is a limit

16. 7. Scaling the database:
horizontal
Master / Slave
More of effort for sysadmin
Master
Important reads
Always writes
Slave
Only Reads
Consider delay
Enough for
1000RPM per
node*

17. Need more?
Separate memcache servers
Sharding
RAIDs
If you are doing heavy search add
Apache Solr
Revisit your problem
Do you need Hadoop
Can you use NoSQL (Cassandra,
Mongo)?

18. Something is not working?
Revisit your database design
Indexes anyone?
Revisit your app design
Pessimistic locks?
Lack of good algorithms?

20. Bottleneck => web nodes
We added one web and db
RPM
Used historical performance data
Rackspace (no choice)
Reduce loaded elements on frontpage
Standby - Monitor APDEX
We did well. No downtimes
TV and BTS at Lazada

22. 20-25% of the traffic just to one page
That page was on a small server we
controlled in TH
2 hours handling our own end of the
world
12/12/12 Campaign

24. Tech communication with business
Monitoring
VPS
Static files
Cache
What could be done better?
Solution
Delay traffic / Business guys
EC2
Cloudfront
Cache and fix code
New relic

25. 1M video clips
20K+ video clips per month
Heavy search
Streaming
Fast quick preview. Faster than Animoto or
Wevideo
Rendering

27. Some
lessons
learnt

28. 7 DO’s

29. 1. No fear

30. 2. Iterate fast

31. 3. Decouple and
create interfaces

32. 4. Track and
Monitor

33. 5. Version and
manage
branches

34. 6. Use the right tools

35. …”mmm I will need a
bigger drill”

36. 7. Code
conventions /
Process

37. 7 DON’Ts

38. 1. Don’t do
“Temporary fixes”
Bad code is like
Karma

39. 2. “DRY”*

40. 3. Don’t
modify
“Live” files
or
databases
directly

41. 4. Don’t
forget
testing
environment

42. 5. Don’t use self-
managed
servers

43. 6. Don’t be the
last to know

44. 7. Don’t scale
too soon just be
prepared

45. Where do I see how the big guys do?
https://github.com/blog/530-how-we-made-
github-fast
http://nerds.airbnb.com/
http://www.facebook.com/Engineering
http://engineering.twitter.com/
http://highscalability.com/blog/2012/4/16/ins
tagram-architecture-update-whats-new-with-
instagram.html

46. Interesting projects
Apache Mesos (Distributed apps).
Cassandra (Database)
Kentrel (Queue)
BERT (RPC)
Apache Hadoop
Apache Zookeeper
HipHopVM

47. Do you code in PHP?
I’m Hiring
caherrerapa@gmail

48. “Scaling is like replacing
all components on a car
while driving it at 100
mph.”
“Initial scaling won’t be
glamorous”