Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1iyFyTo. Avleen Vig presents some of the most unexpected, confusing, hilarious and face-palming events during Etsy's outages to show what can be learnt from their problems to build more resilient systems. Filmed at qconlondon.com. Avleen Vig is a Staff Operations Engineer at Etsy, where he spends much of his time growing the infrastructure for selling knitted gloves and cross-stitch periodic tables. Before joining Etsy he worked at several large tech companies, including EarthLink and Google, as well as a number of small successful startups.

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/resiliency-etsy

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Building resilience
How outages shaped Etsy’s systems

5. Act 1

6. Quick! Be resilient!
http://www.flickr.com/photos/niaid/11854196633/sizes/l/

7. Quick! Be resilient!
• Actually, it’s a slow process
• Iterative
• Introspective
• Horizontal and vertical
development

8. Quick! Be resilient!
http://www.flickr.com/photos/ogcodes/6091644301/sizes/l/

9. Quick! Be resilient!
http://www.flickr.com/photos/studio360/1150744342/sizes/o/

10. Quick! Be resilient!
http://www.flickr.com/photos/studio360/1150744368/sizes/o/

11. Quick! Be resilient!
http://www.flickr.com/photos/ogcodes/6091644301/sizes/l/

12. Quick! Be resilient!
Current generation
Next generation

13. Quick! Be resilient!
http://www.flickr.com/photos/jurvetson/8671257096/

14. Quick! Be resilient!
http://cudebi.wordpress.com/2012/09/19/tah-pagh-tahbe-o-el-reconocimiento-de-william-shakespeare-en-el-universo-de-star-trek/

15. Resilience Engineering
http://www.flickr.com/photos/freefoto/728651045/sizes/o/

16. Resilience Engineering
• “To Engineer is Human”
“To Forgive Design”
- Henry Petroski
• “The Field Guide to Understanding Human Error”
“Just Culture”
- Sidney Dekker

17. Act 2

18. Building resilience at Etsy
• Continuous deployment
• Metrics, metrics, metrics
• Peer review
• Postmortems

19. Building resilience at Etsy
• Continuous deployment
• Metrics, metrics, metrics
• Peer review
• Postmortems
}Culture

20. Or: How to win at failing
Postmortems

21. • No blame
• Open discussion
• Focus on improvements
Constructive cultures

22. • No blame
• Open discussion
• Focus on improvements
}Culture
Constructive cultures

23. –Japanese proverb
“The nail that sticks up,
gets hammered down”
Destructive cultures

24. The result?

25. • #23: Fortune’s “Top 50 best small and medium
businesses to work for”
• Rapid code iterations and deploys
• Lasting relationships
• Generousity of spirit
• …and much more

26. Act 3

27. Doing postmortems?
Get Morgue
http://github.com/etsy/morgue

28. Morgue

29. Morgue

30. Morgue

31. Forkistan
• Mean time to detect: 0 min
• Mean time to recover: 10 mins

32. Yo Dawg, I Heard You
Like Errors..
• Mean time to detect: 2 mins
• Mean time to recover: 15 mins

33. Smashing INT for Fun
and Profit
• Mean time to detect: 0 min
• Mean time to recover: 4 hrs 52 mins

34. Apache Amnesia
• Mean time to detect: 2 hours
• Mean time to recover: 5 mins

35. Continuously Upgrading
Databases
• Mean time to detect: 2 mins
• Mean time to recover: 1 hour (but, not really..)

36. Q & A
Avleen Vig
Staff Operations Engineer
Etsy, Inc
@avleen

38. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/resiliency
-etsy