The ‘Black Friday fail’ is the greatest fear of every major online retailer. Since downtime equals money, and in Black Friday it means quite a lot of money. But the sad truth is that a failure of a service is inevitable, and not only on black Friday. So how can we survive a failure of a service when it inevitably fails? In this lecture I will share our approach for SRE. Why we all have misconceptions on how a major website failure unfolds. and how to use tools like chaos testing, gradual rollout, circuit breakers and automatic fallback to protect your system.

1. Surviving Black Friday
Omri Fima
ROME - APRIL 13/14 2018

2. SURVIVING
BLACK FRIDAYA STORY ABOUT SHOPPING & SOFTWARE ENGINEERING

9. I’m not going to talk
about that kind of
Surviving
Black Friday

17. #BlackFridayFail

18. Omri Fima
• Technical Lead at
Data & ML Group
• Resilience engineer
• Maker

19. Social E-Commerce
We give our members a
place to connect, discover,
and shop with people who
share their passion.

21. My Story

22. Typical E-Commerce Architecture

23. Well, not really

24. It’s probably more like this

26. or more accurately,
like this

27. Long
GC
Load Balancer
cluster cluster
Table
lockSlow
API
Server
in japan
Limited network
CDN

28. So lets see what happens
when a server fails

30. Or in real life

32. But even this is not what
happens most of times.

33. Usually it’s more like this

34. Under
Load

35. Under
Load

36. Choking Choking Choking
Under
Load
Choking

37. Dead Dead Dead
Yay!
No
Load
Dead

38. Oy Vey…

39. Bazillion servers on the cloud will not make it
better.
Under certain scenarios it might even make it
worse.
Sad Truth #1

40. It won’t happen only in black Friday.
Other good scenarios
Good marketing campaign, Bad code, API
load, Failure in external services, Hurricanes.
Sad Truth #2

41. Even 99.99% reliability on 30 servers
is 2HR of downtime per month
Sad Truth #3

42. 2 Hours on Black Friday
can be worth millions of $

44. You can accept it
or...

46. Resilience
Engineering

47. our job is to stop
failure from cascading
throughout the system

48. OK, nice but where do I start?

49. Just for you I have …

52. First you need to understand your SLA
What are our most important flows?
What it means when they fail?
How much can we allow them to fail?

53. Your page

54. Your “essential” page

55. Your “most essential” page

56. We can agree on SLA?
GREAT!
Because this is usually the hardest part.

58. You don’t know if a service is fault
tolerant if you don’t test for faults

59. Simulate and test
Hydra – simulate service latency and error
JMETER - for testing our system under load.

60. Why not to just kill machines?
Dead machines are
The least interesting.

61. Why under load?
Failure is a game of
percentiles.

62. Why under load?
Latency failures
usually cascade
under stress.

63. Hydra + JMETER = Great fun

64. Now we know what to solve first,
and can easily re-test it!

65. Let’s pause for a second

66. Focus

70. Preventing latency
cascade through the
system.

71. Latency is
UX killer

73. Latency is
Server killer

74. Dependency A Dependency B Dependency C Dependency D
UserRequest
UserRequest
UserRequest

75. Dependency A Dependency B Dependency C Dependency D
UserRequest
UserRequest
UserRequest

76. Dependency A Dependency B Dependency C Dependency D
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest
UserRequest

77. Aggressive
Timeouts

78. What is a good
number for a timeout?

79. it depends

80. I’m OK with
0.05% of my
users getting
timeouts

81. Limit queues

82. Bulkhead pattern
20 Threads 10 Threads 15 Threads
UserRequest
UserRequest
UserRequest
Dependency A Dependency B Dependency C Dependency D
5 Threads
Command A Command B Command C Command D

83. Ok,
so now we are protecting our servers
while annoying our users.

85. Remember this?

86. Fail Silently

87. A little bit too aggressive…

88. In the real world

89. Degrade to less
accurate service

90. Failover to another
experience

91. Use stale data
from cache

92. Return Empty data

93. Defer writes

94. And if I cannot fail silently?

95. At least fail gracefully

97. What is not failing gracefully?

98. What is not failing gracefully?

99. What is not failing gracefully?

100. What is not failing gracefully?

102. Feature Toggles
• Have the option to kill your features when
needed.
• Have the option to return your feature back
to life gradually.
0%
50%
100%

103. Gradual rollout
AB Testing
Targeting
Smart bulkheads.
Added Bonus

105. Ok, so we have timeout of 100ms.
The day we were all afraid of has
come.
And it works!
Nobody waits more than 100ms!
But everybody waits 100ms.
And fail

106. You knew you going to fail!
So why wasting everybody
time on trying?

109. Break if you are
likely to
“burn the house”

110. Tools
• Netflix Hystrix (Java)
• Polly & TPL (.Net)
• HystrixJS (Node)
• CircuitBreakerJS (Node)
• Dyno (Python)

112. Measure everything
• when you have failed.
• latency of dependencies.
• when fallbacks were triggered.
• when fallbacks were “almost” triggered.
• when a service failure has “leaked” to the
user.

113. Visualize everything

114. Tools we are using
• StatsD – statistical analysis on metrics
• Graphite – graphing metrics
• Graphana – graphing just about anything
• Skyline – alert on anomalies on grpahs.
• Oculus - find relations between graphs.

117. Sounds trivial

118. Not so much.
Measuring everything is easy
clearing the noise is hard.

119. But guess what?
clearing the noise makes the
system more fault tolerant!

122. • Know your priorities
• Simulate, Test, Fix, Repeat.
• Fail Fast
• Fail Silent
• Turn broken stuff off
• Turn broken stuff off (automatically)
• Measure everything

123. THANK YOU
MORE ABOUT US @ SEARS.CO.IL