Human Factors and PostMortems

•

2 recomendaciones•1,362 vistas

Our daily work takes place in a myriad of systems. They are comprised of software, hardware and humans. And everybody who has worked with complex systems at any scale knows: Failure is not an option, it's inevitable. At Etsy we are embracing the fact that failures happen and that the only way to understand how the accident happened is to investigate it without blaming the humans involved. This is why we have a blameless postmortem for every outage that occurs. It is an open meeting and everybody is invited to join and find out what happened and how we can make the system safer. This talk will explain how postmortems at Etsy are conducted and how we maintain and scale the process as the team grows and new people start. It will go over the tools we built and utilize to make postmortems efficient and also share the learnings from each one with all the people in the company.

Tecnología

Human Factors &
PostMortems
Daniel Schauenberg
d@etsy.com
@mrtazz

Erkenntnis und Irrtum
fließen aus denselben
psychischen Quellen; nur
der Erfolg vermag beide
zu scheiden.
— Ernst Mach, Erkenntnis und Irrtum (p. 116)

People don't come to
work to do a bad job

So I always get off the
hook whatever I do?

There is a difference
between explaining and
excusing human
performance.
— Sidney Dekker, The Field Guide to Understanding
Human Error (p. 196)

Look out for
indicators of Old View
thinking

- she should have
- if he would have
- if they just had
- you failed to

Hindsight Bias
Confirmation Bias
Outcome Bias

incorporate learning
and takeaway from
the meeting

"Hey all, I just ran rm -rf $DIR/
and since the variable was empty
I deleted my whole VM. This
would have been bad in
production. Don't do that."

It is also worth pointing
out that the bias towards
investigating failures
rather than success itself
represents a trade-off.
— Erik Hollnagel, The ETTO Principle: Efficiency-
Thoroughness Trade-Off

Human Error is where
you stopped looking

codeascraft.com
etsy.com/codeascraft/talks
etsy.com/careers

Más contenido relacionado

Development, Deployment & Collaboration at Etsy

Mobile CI at Etsy

A Whirlwind Tour of Etsy's Monitoring Stack

Development, Deployment and Collaboration at Etsy

Scaling Deployment at Etsy

Feature Flagging your Infrastructure for Fun and Profit

DevTools at Etsy

Scaling Deployment at Etsy

StatsD Workshop Monitorama 2013

Etsy chef-workflow

Último

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

Architecting Cloud Native Applications

WSO2

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

When you’re building (micro)services, you have lots of framework options. Spring Boot is no doubt a popular choice. But there’s more! Take Quarkus, a framework that’s considered the rising star for Kubernetes-native Java. It always depends on what's best for your situation, but how to choose the best solution if you're comparing 2 frameworks? Both Spring Boot and Quarkus have their positives and negatives. Let us compare the two by live coding a couple of common use cases in Spring Boot and Quarkus. After this talk, you’ll be ready to get started with Quarkus yourself, and know when to select Quarkus or Spring Boot.

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Jago de Vreede

ICT role in 21st century education and its challenges

rafiqahmad00786416

Human Factors and PostMortems

1. Human Factors & PostMortems Daniel Schauenberg d@etsy.com @mrtazz

3. We deploy quite a lot

4. MTTR

5. MTTR > MTBF

10.

11.

12. realtalk: things break

13. New View

14. Complex Socio- Technical Systems

15. Erkenntnis und Irrtum fließen aus denselben psychischen Quellen; nur der Erfolg vermag beide zu scheiden. — Ernst Mach, Erkenntnis und Irrtum (p. 116)

16. Things made sense at the time

17. People don't come to work to do a bad job

18. Nietzschean Anxiety

19. So I always get off the hook whatever I do?

20. There is a difference between explaining and excusing human performance. — Sidney Dekker, The Field Guide to Understanding Human Error (p. 196)

21. Blameless Postmortems

22. Open Meeting

23. Everybody is Invited

24. What happened?

25. Timeline

26. Describe the past Don't excuse it away

27. The Facilitator

28. Guide the Discussion

29. Look out for indicators of Old View thinking

30. Counterfactuals

31. - she should have - if he would have - if they just had - you failed to

32. Biases

33. Hindsight Bias Confirmation Bias Outcome Bias

34. there are many more

35. Who is in charge?

36. Etsy School

37. Taught Facilitator Course

38. 3 x 90 minutes

39. Remediation Items

40. incorporate learning and takeaway from the meeting

41.

42. turn surprises into known factors

43. MORGUE

44.

45.

46.

47.

48.

49.

50.

51. https://github.com/ etsy/morgue

52. Near Miss

53. Pre Mortem

54. "Hey all, I just ran rm -rf $DIR/ and since the variable was empty I deleted my whole VM. This would have been bad in production. Don't do that."

55. Architecture Reviews

56. Operability Reviews

57. It is also worth pointing out that the bias towards investigating failures rather than success itself represents a trade-off. — Erik Hollnagel, The ETTO Principle: Efficiency- Thoroughness Trade-Off

58. Investigate Success

59. Why did it work?

60. Human Error is where you stopped looking

61. peakscale.com/ postmortems

62. codeascraft.com etsy.com/codeascraft/talks etsy.com/careers

63. Questions?

64. Human Factors & PostMortems Daniel Schauenberg d@etsy.com @mrtazz