SlideShare una empresa de Scribd logo
1 de 17
Who Broke Prod?
Growing teams who can fail without fear
Emma Button
Co-founder, nubeGO.io
@growerofawesome
InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
culture-blameless-failure
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com
BAD STUFF
HAPPENS
SELF-DEFENCE
Photo: Silvia Izquierdo
Respond positively to feedback
Stop seeking blame
SELF-DEFENCE
Resilience
• Re-live a painful failure. Find one new way of responding in a painful situation, write
it down and consciouslychoose to use that response next time
• Respond with “Thank you for the feedback”
Stop Seeking Blame
• Correct yourself and others when we find ourselves asking “Who is to blame for
this?” or “What caused this to happen?”– replay the question as “What are the
factors that contributed to the failure?”
• Periodically remind ourselves, our peers and our team “It’s OK to fail”
Improvement Katas – Responding to Failure
INCIDENT
Brutal transparency
Collaboration
& Shared Accountability
INCIDENT Improvement Katas – Incident Response
Transparency
• Use Slack to share and keep a quick, lightweight, internal record of all actions we
take, record our assumptions
Collaboration & Shared Accountability
• Practice Pair-Incident-Management
• Pair when formulating customer-facing responses
• Pair when diagnosing logs & metrics
• Pair when applying changes
• Get into the habit of using the word “We” instead of They/I/You/Them
POST-MORTEM
Blameless Post-Mortem
Learning Review
Beyond Blame; Learning from Failure and Success by Dave Zwieback
POST-MORTEM Improvement Katas – Blameless Post-Mortem
Do them!
VISIBILITY
Make failure visible
Know your normal
VISIBILITY Improvement Katas – Make Failure Visible
Visualise Failure
• Install an information radiator. Put the results of your CI/CD
pipeline onto the screen somewhere that everyone can see.
Know Your Normal
• Draw a flow map of data as it travels through your system on
a big wall or whiteboard.
EXPERIMENT
Make time for experiments
Try things you DO NOT know
Practice game days
REWARD
Never punish people for trying
Reward positive behaviours
Change starts with US
REWARD
Re-enforceBlameless Behaviours
• At least twice a week, make a point of publicly singing people’s praises for
demonstrating behaviours that support:-
• Experimentation
• Collaboration
• Accountability
• Go for lunch as a team to celebrate a honest, but difficult retrospective
Improvement Katas – Reward, don’t Punish
Reward, Don’t Punish
• Practice using supportive language:
• Say “thank-you”to someone who shares their account of what happened
• Say “well done” to someone who tries a new approach to solve a problem, whether
or not it works
• Tell someone their honesty in a post-mortem is appreciated
• Reward honest and blameless contribution to post-mortem with a gift - laptop stickers
Emma Button
emma@emmabutton.co.uk
linkedin.com/in/emma-button
@growerofawesome
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
culture-blameless-failure

Más contenido relacionado

Más de C4Media

Más de C4Media (20)

Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CD
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine Learning
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at Speed
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep Systems
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.js
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly Compiler
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix Scale
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's Edge
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home Everywhere
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing For
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data Engineering
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 
Navigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery TeamsNavigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery Teams
 
High Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in AdtechHigh Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in Adtech
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/await
 
Opportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven UtopiaOpportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven Utopia
 
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/DayDatadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
 
Are We Really Cloud-Native?
Are We Really Cloud-Native?Are We Really Cloud-Native?
Are We Really Cloud-Native?
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL Database
 
A Dive into Streams @LinkedIn with Brooklin
A Dive into Streams @LinkedIn with BrooklinA Dive into Streams @LinkedIn with Brooklin
A Dive into Streams @LinkedIn with Brooklin
 

Último

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Último (20)

Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 

Who Broke Prod? - Growing Teams Who Can Fail without Fear

  • 1. Who Broke Prod? Growing teams who can fail without fear Emma Button Co-founder, nubeGO.io @growerofawesome
  • 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ culture-blameless-failure • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
  • 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
  • 5. SELF-DEFENCE Photo: Silvia Izquierdo Respond positively to feedback Stop seeking blame
  • 6. SELF-DEFENCE Resilience • Re-live a painful failure. Find one new way of responding in a painful situation, write it down and consciouslychoose to use that response next time • Respond with “Thank you for the feedback” Stop Seeking Blame • Correct yourself and others when we find ourselves asking “Who is to blame for this?” or “What caused this to happen?”– replay the question as “What are the factors that contributed to the failure?” • Periodically remind ourselves, our peers and our team “It’s OK to fail” Improvement Katas – Responding to Failure
  • 8. INCIDENT Improvement Katas – Incident Response Transparency • Use Slack to share and keep a quick, lightweight, internal record of all actions we take, record our assumptions Collaboration & Shared Accountability • Practice Pair-Incident-Management • Pair when formulating customer-facing responses • Pair when diagnosing logs & metrics • Pair when applying changes • Get into the habit of using the word “We” instead of They/I/You/Them
  • 9. POST-MORTEM Blameless Post-Mortem Learning Review Beyond Blame; Learning from Failure and Success by Dave Zwieback
  • 10. POST-MORTEM Improvement Katas – Blameless Post-Mortem Do them!
  • 12. VISIBILITY Improvement Katas – Make Failure Visible Visualise Failure • Install an information radiator. Put the results of your CI/CD pipeline onto the screen somewhere that everyone can see. Know Your Normal • Draw a flow map of data as it travels through your system on a big wall or whiteboard.
  • 13. EXPERIMENT Make time for experiments Try things you DO NOT know Practice game days
  • 14. REWARD Never punish people for trying Reward positive behaviours Change starts with US
  • 15. REWARD Re-enforceBlameless Behaviours • At least twice a week, make a point of publicly singing people’s praises for demonstrating behaviours that support:- • Experimentation • Collaboration • Accountability • Go for lunch as a team to celebrate a honest, but difficult retrospective Improvement Katas – Reward, don’t Punish Reward, Don’t Punish • Practice using supportive language: • Say “thank-you”to someone who shares their account of what happened • Say “well done” to someone who tries a new approach to solve a problem, whether or not it works • Tell someone their honesty in a post-mortem is appreciated • Reward honest and blameless contribution to post-mortem with a gift - laptop stickers
  • 17. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ culture-blameless-failure