Development, Deployment & Collaboration at Etsy

•

5 likes•1,633 views

Daniel Schauenberg

Technology

http://www.ﬂickr.com/photos/brianglanz/1095706242

How comfortable
are you deploying
a change right
now?

@mrtazzhttp://www.ﬂickr.com/photos/renaissancechambara/2349811492
small change

“If this is your first
day at Etsy, you
deploy the site”

@mrtazz
Developer VMs
• KVM
• Every engineer has one
• Fully Chef’d with the Etsy Stack
• Different sizes and Chef roles

@mrtazz
Continuous Integration
• Run set of tests before each deploy
• Full QA suite
• Princess/Production smoker tests
• Try (yup, there is one)

http://www.ﬂickr.com/photos/egfocus/6962179321

@mrtazz
The Bobs
• LXC virtualized hosts
• 14/physical hosts
• Spread over 3 SSDs
• Most of them attached to try

@mrtazz
Deployinator
• 2 Buttons, no ambiguity
• Overview of current state of deploy
• Links to Logwatcher and Dashboards
• Easy to add stacks for new tools to deploy

http://www.ﬂickr.com/photos/jbgeronimi/6363087361

@mrtazz
Monitoring
• Devs do their feature monitoring
• Everybody can access all the graphs
• Dashboard All The Things!
• Stream All The Logs!

If you are writing
code, you are
on-call

@mrtazz
On-Call Schedules
• ops on-call
• dev on-call
• payments on-call
• support on-call

@mrtazz
Dev On-Call
• On-call for 3 days
• All developers who are not in another
rotation
• L1 and L2 escalations
• L1 if it’s your first time

@mrtazz
Incident Response
• “This graph looks funny”
• “Hey I just got paged for elevated error rate
after deploys”
• “Supergrep is going crazy!!”

@mrtazz
#warroom
• only outage related conversations
• coordinate investigation, communication,
countermeasures and monitoring
• good place to lurk for new engineers

@mrtazz
Summary
• These are things that work for *us*
• Culture is an on-going effort
• Share everything
• Encourage learning/teaching

@mrtazz
Summary
• Lunch ’n learns
• DC visits
• On-call for a day
• Bootcamps/Senior rotations

codeascraft.com

etsy.com/codeascraft/talks

etsy.github.com

etsy.com/careers

Development,
Deployment and
Collaboration at Etsy
Daniel Schauenberg
dschauenberg@etsy.com

What's hot

Concourse and DatabaseGwenn Etourneau

Practical Continuous Deployment - Atlassian - London AUG 18 Feb 2014Matthew Cobby

VersaPay's Tools for Happyfficient DevelopersPhilippe Creux

Monitoring in the cloud with PuppetKris Buytaert

My Top 5 Favorite GemsJimmy Ngu

PuppetConf 2017: Using Puppet Enterprise APIs with PowerShell- Jason Johnson,...Puppet

Data + Visualization Toronto Meetup 4: The Google Maps APImattholtom

Railsconf 2014 - Deploying Rails is Easier Thank It Lookstalkingquickly

Update Smart: Use a Staging Site!Meagan Hanes

PuppetConf 2017: No Server Left Behind - Miguel Di Ciurcio Filho, InstructPuppet

Dev-Friendly OpsJosh Schramm

What's New in JHipsterLand - DevNexus 2017Matt Raible

The Ruby workflowClemens Helm

Stripe con 2021 UI stackMaximeRainville1

From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015Corley S.r.l.

Modern Front End Tools & WorkflowAdir Amsalem

Background processes and tasks in an async worldparticlebanana

Anton Boyko "The future of serverless computing"Fwdays

React for .net developersmacsdickinson

Anton Boyko, "The evolution of microservices platform or marketing gibberish"Sigma Software

What's hot (20)

Concourse and Database

Practical Continuous Deployment - Atlassian - London AUG 18 Feb 2014

VersaPay's Tools for Happyfficient Developers

Monitoring in the cloud with Puppet

My Top 5 Favorite Gems

PuppetConf 2017: Using Puppet Enterprise APIs with PowerShell- Jason Johnson,...

Data + Visualization Toronto Meetup 4: The Google Maps API

Railsconf 2014 - Deploying Rails is Easier Thank It Looks

Update Smart: Use a Staging Site!

PuppetConf 2017: No Server Left Behind - Miguel Di Ciurcio Filho, Instruct

Dev-Friendly Ops

What's New in JHipsterLand - DevNexus 2017

The Ruby workflow

Stripe con 2021 UI stack

From Chef to Saltstack on Cloud Providers - Incontro DevOps 2015

Modern Front End Tools & Workflow

Background processes and tasks in an async world

Anton Boyko "The future of serverless computing"

React for .net developers

Anton Boyko, "The evolution of microservices platform or marketing gibberish"

Similar to Development, Deployment & Collaboration at Etsy

Continuous Integration, the minimum viable productJulian Simpson

Continuous TestingCrevise Technologies

Continuous DeploymentBrian Moon

Pragmatic Continuous Delivery - ReaktorDevDay 2012Neeme Praks

Untangling spring week10Derek Jacoby

Devops For DrupalKris Buytaert

Maintaining Retainers as a WordPress DeveloperDevinVinson

Great Tools Heavily Used In Japan, You Don't Know.Junichi Ishida

Production Ready WordPress #WPLDNEdmund Turbin

Devops Devops Devops, at FrosconKris Buytaert

Take control of your Jenkins jobs via job DSL.Łukasz Proszek

Continuous Delivery of Puppet ManifestsKris Buytaert

Frontend Engineer ToolboxYnon Perek

Essential Plone development tools - Plone conf 2012Kim Chee Leong

Vagrant for Effective DevOps CultureVaidik Kapoor

stackconf 2022: Infrastructure Automation (anti) patternsNETWAYS

Infrastructure as Code PatternsKris Buytaert

Task runners + theming automating your workflowJoshua Gilmer

Ship It ! with Ruby/ Rails EcosystemYi-Ting Cheng

Lessons Learnt in 2009pratiknaik

Similar to Development, Deployment & Collaboration at Etsy (20)

Continuous Integration, the minimum viable product

Continuous Testing

Continuous Deployment

Pragmatic Continuous Delivery - ReaktorDevDay 2012

Untangling spring week10

Devops For Drupal

Maintaining Retainers as a WordPress Developer

Great Tools Heavily Used In Japan, You Don't Know.

Production Ready WordPress #WPLDN

Devops Devops Devops, at Froscon

Take control of your Jenkins jobs via job DSL.

Continuous Delivery of Puppet Manifests

Frontend Engineer Toolbox

Essential Plone development tools - Plone conf 2012

Vagrant for Effective DevOps Culture

stackconf 2022: Infrastructure Automation (anti) patterns

Infrastructure as Code Patterns

Task runners + theming automating your workflow

Ship It ! with Ruby/ Rails Ecosystem

Lessons Learnt in 2009

Recently uploaded

The Metaverse: Are We There Yet?Mark Billinghurst

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel

ECS 2024 Teams Premium - Pretty SecureFemke de Vroome

AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379

ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...FIDO Alliance

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl

WebAssembly is Key to Better LLM PerformanceSamy Fodil

Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge

10 Differences between Sales Cloud and CPQ, Blanka DoktorováCzechDreamin

Demystifying gRPC in .Net by John StaveleyJohn Staveley

SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...CzechDreamin

THE BEST IPTV in GERMANY for 2024: IPTVreelreely ones

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin

The UX of Automation by AJ King, Senior UX Researcher, OcadoUXDXConf

Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore

Buy Epson EcoTank L3210 Colour Printer Online.pdfEasyPrinterHelp

Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomCzechDreamin

PLAI - Acceleration Program for Generative A.I. StartupsStefano

Connecting the Dots in Product Design at KAYAKUXDXConf

Recently uploaded (20)

The Metaverse: Are We There Yet?

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx

ECS 2024 Teams Premium - Pretty Secure

AI presentation and introduction - Retrieval Augmented Generation RAG 101

ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

WebAssembly is Key to Better LLM Performance

Enterprise Knowledge Graphs - Data Summit 2024

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

Demystifying gRPC in .Net by John Staveley

SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...

THE BEST IPTV in GERMANY for 2024: IPTVreel

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

The UX of Automation by AJ King, Senior UX Researcher, Ocado

Extensible Python: Robustness through Addition - PyCon 2024

Buy Epson EcoTank L3210 Colour Printer Online.pdf

Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom

PLAI - Acceleration Program for Generative A.I. Startups

Connecting the Dots in Product Design at KAYAK

Development, Deployment & Collaboration at Etsy

1. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz

3. @mrtazz Etsy Stats

4. @mrtazz Etsy Stats

5. @mrtazzItem by TheBackPackShoppe

6. http://www.ﬂickr.com/photos/brianglanz/1095706242

7. avg 50 deploys/ day

8. avg n > m deploys/ day

9. How comfortable are you deploying a change right now?

10. @mrtazzhttp://www.ﬂickr.com/photos/renaissancechambara/2349811492 small change

11. Config Flags Item by RocajoStudio

12.

13. “If this is your first day at Etsy, you deploy the site”

14. Developer VMs

15. @mrtazz Developer VMs • KVM • Every engineer has one • Fully Chef’d with the Etsy Stack • Different sizes and Chef roles

16.

17. Continuous Integration

18.

19. @mrtazz Continuous Integration • Run set of tests before each deploy • Full QA suite • Princess/Production smoker tests • Try (yup, there is one)

20. http://www.ﬂickr.com/photos/egfocus/6962179321

21. @mrtazz The Bobs • LXC virtualized hosts • 14/physical hosts • Spread over 3 SSDs • Most of them attached to try

22.

23. Item by decomodwalls

24. Deployinator

25. @mrtazz Deployinator • 2 Buttons, no ambiguity • Overview of current state of deploy • Links to Logwatcher and Dashboards • Easy to add stacks for new tools to deploy

26. http://www.ﬂickr.com/photos/jbgeronimi/6363087361

27.

28. Monitoring

29. @mrtazz shouldigraphit.com

30. @mrtazz Monitoring • Devs do their feature monitoring • Everybody can access all the graphs • Dashboard All The Things! • Stream All The Logs!

31.

32.

33.

34. On Call

35. If you are writing code, you are on-call

36. @mrtazz On-Call Schedules • ops on-call • dev on-call • payments on-call • support on-call

37.

38. @mrtazz Dev On-Call • On-call for 3 days • All developers who are not in another rotation • L1 and L2 escalations • L1 if it’s your first time

39. Incident Response

40. @mrtazz Incident Response • “This graph looks funny” • “Hey I just got paged for elevated error rate after deploys” • “Supergrep is going crazy!!”

41. Is the site down?

42.

43. #warroom

44. @mrtazz #warroom • only outage related conversations • coordinate investigation, communication, countermeasures and monitoring • good place to lurk for new engineers

45. Post Mortems

46. blameless

47. Everybody’s invited

48. Learning Opportunity

49. Summary

50. @mrtazz Summary • These are things that work for *us* • Culture is an on-going effort • Share everything • Encourage learning/teaching

51. @mrtazz Summary • Lunch ’n learns • DC visits • On-call for a day • Bootcamps/Senior rotations

52. codeascraft.com etsy.com/codeascraft/talks etsy.github.com etsy.com/careers

53. Questions?

54. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com