The Postmortem – It doesn’t have to be messy
Your company has had a really, really bad day. Bad Days happen. It is when, not if. In a typical scenario, the top executives come in yelling and screaming and fire whoever they think may have had some part in the Bad Day’s events. After most of the yelling has stopped, then the executives want to know what happened. There is no doubt that autopsies can be messy and emit unpleasant odors, but they are a necessary evil and effective learning tool. But there is a right way and a wrong way to conduct a postmortem analysis of your incident. The obvious objective is to learn what happened so that it doesn’t happen again. The focus should be on the root cause and conducting an efficient analysis of the problem, but there is so much more that can be learned. Using the Incident Command System (ICS), this session provides an overview of the same postmortem process used by Fire Departments across the country.
Recomendados
Recomendados
Más contenido relacionado
Más de Chef Software, Inc.
Más de Chef Software, Inc. (20)
Último
Último (20)
The Postmortem – It doesn’t have to be messy
- 1. © 2013 Blackrock 3 Partners LLC 1 A Good Post Mortem . . . . Includes a Pulse Check! 2 2013 Events…so far 2 You are part of Critical Infrastructure! 3 Postmortem Typical Worst Case (the Usual Case) Yelling & screaming Fighting & biting Firing Everyone hides Blame finding Many resumes updated Stock option date reviewed Nothing is fixed Best Case (Utopia) Process oriented root cause(s) analysis Team effort Focus on positive Focus on prevention Mistakes kept in perspective Completed by non- involved personnel
- 2. © 2013 Blackrock 3 Partners LLC 2 4 EVENT A Typical Post Mortem Goes Like This Failure Operations Process Software Hardware Response Responsibilities 5 Why the Pulse Check? Cause: Factors that produce an effect Reason: An explanation or justification Y 6 Looking for “Some Guy” Failure Operations Process Software Hardware Response Responsibilities
- 3. © 2013 Blackrock 3 Partners LLC 3 7 AAR Marketing/Sales want products faster than Engineering/Operations can design/build/ implement them Investors want faster growth than management can deliver Early technology architecture/equipment decisions never get replaced R&D teams love the newest technology Operations teams hate the newest technology 8 It’s a Matter of Perspective A wake-up call A near miss for a larger, more devastating incident An opportunity to stop and look at the organization – because you never have time to stop and look because the business is running so fast You don't know what you don't know Don’t let a good crisis go to waste! 9 When done right The process allows you to look at risk differently Positive Failure!
- 4. © 2013 Blackrock 3 Partners LLC 4 10 TECH Possible vs. Likely Operations Copyright 2013 BlackRock 3 Partners LLC Operations Normal Technological Environm ental H um an Not Normal C rim inal 11 TALENT Training Accountability Leadership Empowerment Notification Trust/Temptation Experts or Saviors 12 IMS Incident Commander Unified Command Applications App 1 App 2 Database DBA - 1 DBA - 2 Continued Operations Customer Service Executive Liaison Plans Disaster Recovery
- 5. © 2013 Blackrock 3 Partners LLC 5 13 Change – Failure – AAR Maybe we shouldn’t talk about failure and death together! Failure AAR Change After Action Review (AAR) Post Incident Evaluation (PIE) Post Incident Analysis (PIA) Incident Review (IR) Reason- Cause – Analysis (RCA) 14 EVENT Operations Process Software Hardware Response Responsibilities Sales Marketing Growth Technology Environment Communication Human After Action Review 15 AAR Tradeoff between different groups: Marketing Sales Engineering Operations Finance Management Team Board Investors
- 6. © 2013 Blackrock 3 Partners LLC 6 16 Establishing a Culture Things WILL go wrong There should be a process to address them Incident Management System (IMS) – Handling of an occurrence – Handling of an issue – Having a really bad day RCA with a new perspective Retrospective and prospective The key is to understand the reason 17 www.blackrock3.com Chris Hawley – Chris@blackrock3.com Rob Schnepp – Rob@blackrock3.com Ron Vidal – Ron@blackrock3.com San Francisco & Baltimore Blackrock 3 Partners LLC