Failures are the bane of scaling a modern web service and can cause serious pain for your end users! Lucky for us, there are techniques that can help protect your product handle failures in subsystems gracefully. This talk will dive into one of these in depth, the Circuit Breaker pattern, and explore the options it gives us for keeping all our users safe.

1. Protect your users with
Circuit Breakers
@scott_triglia
Jim’s

2. Yelp’s Mission:
Connecting people with great
local businesses.

3. @scott_triglia
I work with 💵💶

4. Let’s talk Circuit Breakers

7. CC BY-NC 2.0 by zerial

10. Our goals today:
introduce a basic circuit
breaker

11. Our goals today:
a modular circuit breaker

12. Our goals today:
test it out on several
scenarios

15. 15
1
2
34

16. the fundamental rule:
your systems will fail
what’s your response?

17. 1
2

18. 1
2
3

19. 1
2
34

20. Nygard’s circuit
breaker

25. Circuit Breaker States:
* Healthy (or “closed”)
* Recovering (or “half-open”)
* Unhealthy (or “open”)

27. Recovery:
* Wait for recovery_timeout seconds
* Send a trial request, trust its results

28. Before a circuit breaker

29. Assume the kitchen gets slow

30. Kitchen’s backlog grows

31. Diners wait much longer to get food

32. New diners make issues worse

33. And your entire system collapses

34. And your entire system collapses

35. With a circuit breaker

36. CB sees backlog, stops orders

37. Fewer frustrated diners

38. Reduced load on the kitchen

39. A well defined failure mode

40. How can we do better?

41. Improvements

42. 1) Detecting Unhealthiness

43. 2) Mitigating downtime

44. 3) Recovering Effectively

45. Detecting Unhealthiness
Component 1:

47. def signal_overload(cb):
if len(jobs) > THRESH:
cb.mark_unhealthy()

48. New Behavior:
* CB gets signals from anywhere
* Signal combining logic

49. * Allows many (many) new signals
* Must combine signals
* Adds complexity to system

50. Mitigating Downtime
Component 2:

55. New Behavior:
* Code can check in advance about
healthiness of system
* Automatic monitoring!

56. * Build features on top of system
health status
* Requires a single source of truth?

57. Recovering Effectively
Component 3:

59. Dark launch:
* Reject but process normally
* Dangerous with side effects
Block User
Request
Try to process anyway!

60. Synthetic:
* Dark launching with fake requests
* Not necessarily representative
Block User
Request
Process fake requests

61. New Behavior:
* Traffic determines health
* Removal of recovery timeouts

62. * Representative recovery
* No timeout tuning required
* Dark launching not always possible
* Synthetic can be unrepresentative

63. in summary

64. Your system will fail, have a plan!

65. The basic CB is better than nothing

66. Questions to ask:
* What is “unhealthy” for my system?
* How should I react to unhealthiness?
* How do we recover?

67. Questions to ask:
* What is “unhealthy” for my system?
* How should I react to unhealthiness?
* How do we recover?

68. Questions to ask:
* What is “unhealthy” for my system?
* How should I react to unhealthiness?
* How do we recover?

69. Questions to ask:
* What is “unhealthy” for my system?
* How should I react to unhealthiness?
* How do we recover?

70. …and much more!

71. Much comes down to your use case

72. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp
striglia@yelp.com
@scott_triglia

74. Downtime options
besides rejecting
requests?

75. http://techblog.netflix.com/
2011/12/making-netflix-api-
more-resilient.html

76. How do I safely test out a
new circuit breaker?

77. https://engineering.heroku.com/
blogs/2015-06-30-improved-
production-stability-with-circuit-
breakers/