Facebook has over 1 billion active users, processes trillions of actions per day, and stores billions of photos. It scales horizontally across many servers and caches to handle this large traffic. The interconnection of user data means failures can cascade, so Facebook focuses on reliability through techniques like shedding load during outages to minimize impacts and doing post-mortems to prevent recurrences.

2. Million Active Users
M
M
M
M
M
M
M

3. # property on the Internet (time on site)
s of billions of monthly page views
> trillion feed actions processed per day
s of millions of cache queries per second
Over trillion objects cached
Over million active users, half log in every day
billion photo files stored
Over billion minutes spent every Day
Over billion pieces of content uploaded every week

4. Move Fast
Server Scaling
Reliability

5. Million Active Users
M
M
M
M
M
M
M

6. Over million active users per engineer
, ,
,
,
, ,
,
,
,
, ,
Facebook Google* Amazon* Microsoft*
* Conservative estimates based on publicly available data

7. Few external deadlines... but

8. The site can’t go down

9. Frequent small changes
Never a delay waiting for a push
Easier to isolate bugs
Major changes dark launched

12. Traditional websites
Bob’s data
Bob

13. Traditional websites
Bob’s data Beth’s data
Bob Beth
Julie’s data Sue’s data
Julie Sue
Dan’s data Erin’s data
Dan Erin

14. Facebook the data is interconnected
Bob Beth Erin
Servers

17. Scale Horizontally

18. Web Server
Memcache
Database

19. Web Server
Memcache
Database

20. Web Server
Memcache
Database

21. Web Server
Memcache
Database

22. Network Incast
Memcache Memcache Memcache Memcache
Switch
PHP Client

23. Network Incast
Memcache Memcache Memcache Memcache
Switch
Many Small
Get Requests
PHP Client

24. Network Incast
Memcache Memcache Memcache Memcache
Switch
Many big
data packets
PHP Client

25. Network Incast
Memcache Memcache Memcache Memcache
Switch
PHP Client

26. Network Incast
Memcache Memcache Memcache Memcache
Switch
PHP Client

27. Network Incast
Memcache Memcache Memcache Memcache
Switch
PHP Client

28. Reliability

29. Single Points of Failure

30. Single Points of Failure
Software can be a SPOF

31. Don’t make small problems big

32. Don’t make small problems big
Don’t push problems upstream

33. Don’t make small problems big
Don’t push problems upstream
Be wary of “smart” failover

34. Don’t make small problems big
Don’t push problems upstream
Be wary of “smart” failover
Shed load when you’re in trouble

35. If you lose half of your machines
you’re doing well if you’re serving
half of your traffic

36. Measure Everything

37. p vs p

39. Culture
Always do a post-mortem
Release often
Control and Responsibility

40. Facebook Platform

41. Lessons Learned
Federate everything
Keep failures contained
Measure distributions
Understand every problem
Make a person responsible

42. (c) Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. .