Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1Th5Qe3. Yongsheng Wu talks about how to build highly-resilient systems at scale. He covers 5 highly fault-tolerant, battle-tested systems: dynamic service discovery, real-time configuration management, caching, persistent storage, and event processing pipeline. Wu presents also failure cases that prompted engineers at Pinterest to build such systems, and how they actually test these systems. Filmed at qconsf.com. Yongsheng Wu is an early engineer on infrastructure team at Pinterest, where he helped in making the system available, scalable, reliable, and performant. He is currently leading the storage & caching team; previously he led the effort in making it easy to use, build and run services at Pinterest.

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations/
pinterest-resilient-systems

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Highly Resilient Systems at Pinterest
Yongsheng Wu
Engineering Manager of Storage & Caching, Pinterest
Email: yongsheng@pinterest.com
Pinterest: www.pinterest.com/yswu
Nov 17, 2015

5. Our mission is to help people
discover and do what they love

6. 50+ billion Pins
categorized by 100+ million Pinners into
> 1 billion Boards

7. 10s of 1,000s of
Instances on AWS

8. ~100 Services

9. Some Systems
Handles ~1M QPS

10. Failures Happen All the
Time

11. Dynamic Service Discovery
Realtime Configuration
Caching
Persistent Storage
Async Processing

12. Dynamic Service Discovery
Realtime Configuration
Caching
Persistent Storage
Async Processing

13. Client Retries
Clients
foo
1
bar
1
bar
2
bar
3
bar 1, 2, 3

14. Client Retries
Clients
foo
1
bar
1
bar
2
bar
3
bar 1, 2, 3

15. Client Retries
Clients
foo
1
bar
1
bar
2
bar
3
bar 1, 2, 3

16. Client Retries
Clients
foo
1
bar
1
bar
2
bar
3
bar 1, 2, 3
bar
4
?

17. Reactive Manifesto

18. Zookeeper
• Highly reliable distributed coordination
• Hierarchical namespace: ZNode
• Persistent Node
• Ephemeral Node
• Sequence Node
• Watch
• Node Children Changed
• Node Created
• Node Data Changed
• Node Deleted

19. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2
bar 1
Capacity Addition

20. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2
bar 1
Capacity Addition

21. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2
bar 1, bar2
Capacity Addition

22. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2
bar 1, bar 2
Capacity Reduction

23. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 1, bar 2
Capacity Reduction

24. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 1
Capacity Reduction

25. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2
bar 1, bar 2
Zookeeper Failure

26. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2
bar 1, bar 2
Observers
/discovery
bar
prod
bar 1
bar 2
Zookeeper with Observers

27. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2
bar 1, bar 2
Observers
/discovery
bar
prod
bar 1
bar 2
Zookeeper with Observers Failure

28. Avoid Complete reliance on any
single system, even if it is a highly
reliable distributed system

29. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2 bar 1, bar 2
Observers
/discovery
bar
prod
bar 1
bar 2
ZUM bar
bar 1, bar 2
Zookeeper with Observers Protected by Local Persistent ServerSet

30. Dynamic Service Discovery
bar
1
Zookeeper
/discovery
bar
prod
bar 1
foo
foo apps
bar
2
bar 2 bar 1, bar 2
Observers
/discovery
bar
prod
bar 1
bar 2
ZUM bar
bar 1, bar 2
Zookeeper with Observers Protected by Local Persistent ServerSet

31. Challenges
• Rapid planned capacity reduction
• Gradual loss of instances

32. Dynamic Service Discovery
Realtime Configuration
Caching
Persistent Storage
Async Processing

33. Typical Service Setup
Config
ServerClient
Cache
DB

34. Realtime Configuration
• Decider
• Experiment
• Rate limiting
• Failover
• … …

35. Realtime Configuration
Zookeeper
/config
decider
setting_1: 53
foo
foo apps
{“setting_1”: 53,
… …. }
setting_2: 0
setting_3: 100
setting_4: 78
… …
Config
Admin
Console
Zookeeper Powered Realtime Configuration Management

36. Realtime Configuration
foo
foo apps
v0: {“setting_1”: 53,
… …. }
Observers
decider
v0: {“setting_1”: 53,
… …. }
Zookeeper
/config
decider: v0
AWS S3
Config
Admin
Console
/config
decider: v0
ZUM
Zookeeper, Observers, S3 Based Realtime Configuration Management

37. Realtime Configuration
foo
foo apps
v0: {“setting_1”: 53,
… …. }
Observers
ZUM decider
v0: {“setting_1”: 53,
… …. }
Zookeeper
/config
decider: v0
AWS S3
Config
Admin
Console
/config
decider: v0
v1: {“setting_1”: 54,
… …. }
v1
Zookeeper, Observers, S3 Based Realtime Configuration Management

38. Realtime Configuration
foo
foo apps
v0: {“setting_1”: 53,
… …. }
Observers
ZUM decider
v0: {“setting_1”: 53,
… …. }
Zookeeper
/config
decider: v1
AWS S3
Config
Admin
Console
/config
decider: v0
v1: {“setting_1”: 54,
… …. }
v1
Zookeeper, Observers, S3 Based Realtime Configuration Management

39. Realtime Configuration
foo
foo apps
v0: {“setting_1”: 53,
… …. }
Observers
decider
v0: {“setting_1”: 53,
… …. }
Zookeeper
/config
decider: v1
AWS S3
Config
Admin
Console
/config
decider: v1
v1: {“setting_1”: 54,
… …. }
v1
ZUM
v1: {“setting_1”: 54,
… …. }
Zookeeper, Observers, S3 Based Realtime Configuration Management

40. Realtime Configuration
foo
foo apps
v1: {“setting_1”: 54,
… …. }
Observers
decider
v0: {“setting_1”: 53,
… …. }
Zookeeper
/config
decider: v1
AWS S3
Config
Admin
Console
/config
decider: v1
v1: {“setting_1”: 54,
… …. }
v1
ZUM
v1: {“setting_1”: 54,
… …. }
Zookeeper, Observers, S3 Based Realtime Configuration Management

41. Realtime Configuration
foo
foo apps
v1: {“setting_1”: 54,
… …. }
Observers
decider
v1: {“setting_1”: 54,
… …. }
Zookeeper
/config
decider: v1
AWS S3
Config
Admin
Console
/config
decider: v1
v1: {“setting_1”: 54,
… …. }
v1
ZUM
v1: {“setting_1”: 54,
… …. }
Zookeeper, Observers, S3 Based Realtime Configuration Management

42. Realtime Configuration
foo
foo apps
v1: {“setting_1”: 54,
… …. }
Observers
decider
v1: {“setting_1”: 54,
… …. }
Zookeeper
/config
decider: v1
AWS S3
Config
Admin
Console
/config
decider: v1
ZUM
Zookeeper, Observers, S3 Based Realtime Configuration Management

43. Challenges
• Staggered rollout
• Configuration of huge size

44. Dynamic Service Discovery
Realtime Configuration
Caching
Persistent Storage
Async Processing

45. Typical Service Setup
Config
ServerClient
Cache
DB

46. Caching
bar
cache
1
bar
cache
m
proc1
… …
prock
consistent hash
ring
Foo 1
… …

47. Caching
bar
cache
1
bar
cache
m
proc1
… …
prock
Foo 2
… …
proc1
… … prock
Foo 1
… …
proc1
… …
prock
Foo n
Too many
connections!

48. Twemproxy
bar
cache
1
bar
cache
m
… …
proc1
… …
prock
Foo 1
… …
proc1
… …
prock
Foo 2
proc1
… …
prock
Foo n
NutcrackerNutcracker Nutcracker

49. Cache Inconsistency
Nutcracker
… …foo: 1
bar cache 1 bar cache 2 bar cache n
set foo 2
foo: 2

50. Cache Inconsistency
Nutcracker
… …foo: 1
bar cache 1 bar cache 2 bar cache n
foo: 2

51. McRouter
… …
cache 1 cache 2 cache n
McRouter

52. McRouter
… …
cache 1 cache 2 cache n
McRouter
No ring reshuffle

53. McRouter
Pros
• No inconsistency caused by node joining/leaving the pool
• No cascading failures in case of excessive load caused by hot keys
Cons
• Cache misses

54. Replicated Pools - Reads
McRouter
cache
1
cache
2
cache
n…
cache
1’
cache
2’
cache
n’…
ac Pool de Poolgetfoo

55. Replicated Pools - Reads
McRouter
cache
1
cache
2
cache
n…
cache
1’
cache
2’
cache
n’…
ac Pool de Poolgetfoo

56. Replicated Pools - Invalidation
McRouter
cache
1
cache
2
cache
n…
cache
1’
cache
2’
cache
n’…
ac Pool de Pooldeletefoo

57. Replicated Pools - Invalidation
McRouter
cache
1
cache
2
cache
n…
cache
1’
cache
2’
cache
n’…
ac Pool de Pooldeletefoo
Log

58. Replicated Pools - Invalidation
McRouter
cache
1
cache
2
cache
n…
cache
1’
cache
2’
cache
n’…
ac Pool de Pooldeletefoo
Log

59. Replicated Pools - Invalidation
McRouter
cache
1
cache
2
cache
n…
cache
1’
cache
2’
cache
n’…
ac Pool de Pooldeletefoo
LogSinger
kafka
tailer
PinLater

60. Challenges
• Build the feedback loop from persistent layer to
caching layer
• Move to multiple geographic regions

61. Dynamic Service Discovery
Realtime Configuration
Caching
Persistent Storage
Async Processing

62. Typical Service Setup
Config
ServerClient
Cache
DB

63. Sharding
Shard Id Type Local Id
64 bits

64. Clients
DataServices
1
… …
Master
1
Slave
1 … …
Master
m
Slave
m
Clients
Clients
DataServices
2
DataServices
n
Read from Slave

65. Clients
DataServices
1
… …
Master
1
Slave
1 … …
Master
m
Slave
m
Clients
Clients
Read from Slave
DataServices
2
DataServices
n
Read from slave after
master failing health
check over a certain
period of time.

66. Master
1
Slave
1 … …
Master
m
Slave
m
Failover
DataServices Proc
ZUM ShardConfig
ShardConfig
Changes
Realtime Configuration Management Powered Failover

67. Other Persistence Stores
UMetaStore
• Key value store based on HBase
Zen
• Graph store: nodes and edges
• Flexible schema
• Custom index
• Both HBase and MySQL
Future
• Rocksdb

68. Challenges
• Automated failover with MySQL replica set
• HBase 1.x Upgrade
• Move to multiple geographic regions

69. Replication and failover are the
key ingredients for building highly
resilient storage and caching
systems

70. Dynamic Service Discovery
Realtime Configuration
Caching
Persistent Storage
Async Processing

71. Async Processing
Use Cases
• Acknowledge success with non-time-sensitive actions taken at
later time
• Schedule and execute large number of jobs
Benefits
• Faster response time
• More resilient to dependent system failures

72. Pyres Limitations
• No mechanism for success acknowledgement
• No visibility into status of individual job types
• No support for scheduled job execution at a
specific time in the future
• Rate limiting and retries are hard to manage
• Redis as only supported storage backend

73. PinLater
Asynchronous Processing System
Clients
Pinlater
Servers
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pools
Master Slave
Clients
Clients
Pinlater
Servers
PinLater
Servers
Storage Backend
Enqueue
Dequeue/ACK

74. PinLater Job State Transition
Pending Running
Succeeded
Failed
Done
Failed (no more retries)
Dequeued
Failed (retries left)/Claim Timeout

75. PinLater Job Requirements
Idempotency
Commutativity

76. PinLater Dashboard

77. Clients
PinLater
Server
1
PinLater
Server
2
PinLater
Server
n
… …
Master
1
Slave
1
… …
Master
m
Slave
m
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
1
… …
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
k
Clients
Clients
Enqueue
PinLater
Asynchronous Processing System

78. Clients
PinLater
Server
1
PinLater
Server
2
PinLater
Server
n
… …
Master
1
Slave
1
… …
Master
m
Slave
m
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
1
… …
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
k
Clients
Clients
Enqueue
PinLater
Asynchronous Processing System

79. Clients
PinLater
Server
1
PinLater
Server
2
PinLater
Server
n
… …
Master
1
Slave
1
… …
Master
m
Slave
m
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
1
… …
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
k
Clients
Clients
Enqueue
PinLater
Asynchronous Processing System

80. Clients
PinLater
Server
1
PinLater
Server
2
PinLater
Server
n
… …
Master
1
Slave
1
… …
Master
m
Slave
m
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
1
… …
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
k
Clients
Clients
Enqueue
PinLater
Asynchronous Processing System

81. Clients
PinLater
Server
1
PinLater
Server
2
PinLater
Server
n
… …
Master
1
Slave
1
… …
Master
m
Slave
m
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
1
… …
Infra
Worker
Pool
Infra
Worker
Pool
Worker
Pool
k
Clients
Clients
Enqueue
PinLater
Asynchronous Processing System

83. Challenges
• Multi-tenancy with fair failure isolation
• Fault-tolerant async job enqueuer

84. Use async processing as much as
possible to deliver faster
response time and make request
handling more robust

85. Dynamic Service Discovery
Realtime Configuration
Caching
Persistent Storage
Async Processing

86. Learnings
• Avoid Complete reliance on any single system, even
if it is a highly reliable distributed system
• Replication and failover are the key ingredients for
building highly resilient storage and caching
systems
• Use async processing as much as possible to
deliver faster response time and make request
handling more robust

87. Failure Testing
• Be explicit with scope
• Failure Modes
• Sandbox testing
• Manual testing
• Automated simulation
• Testing on production
• AWS is doing it for us all the time
• Simian Army

89. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/pinterest-
resilient-systems