SlideShare una empresa de Scribd logo
1 de 45
DevOps for the Database
Baron Schwartz • QConSF 2018
InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
devops-database
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com
Introduction
I’ve been focused on databases for about
two decades, rst as a developer, then a
consultant, and now a startup founder.
I’ve written several books (High
Performance MySQL)… and created a lot of
open source software, mostly focused
around database monitoring, database
operations, and database performance:
innotop, Percona Toolkit, Percona Monitoring
Plugins among others.
@xaprb 2
Agenda
Overview of database DevOps
How companies succeed
Tooling, culture, process, people
How companies fail
Challenges to database DevOps
Resources, etc
Slides are online at xaprb.com.
My Twitter is @xaprb, my email is baron@vividcortex.com
@xaprb 3
Three Database
DevOps Stories
1. One DBA, hundreds of developers,
growing the DBA team
@xaprb 4
Three Database
DevOps Stories
1. One DBA, hundreds of developers,
growing the DBA team
2. One DBA, from 20 to 100 developers
@xaprb 4
Three Database
DevOps Stories
1. One DBA, hundreds of developers,
growing the DBA team
2. One DBA, from 20 to 100 developers
3. Two database ops folks, seventeen
developers
@xaprb 4
Benefits of Database
DevOps
DevOps brings the same bene ts to the
database as everywhere else.
@xaprb 5
Benefits of Database
DevOps
DevOps brings the same bene ts to the
database as everywhere else.
For software delivery performance in
particular:
Faster, better, cheaper—pick all three
stability -> speed -> stability -> speed
(See the 2018 State of DevOps Report!)
@xaprb 5
Detriments of Lacking
DevOps
Without DevOps, speed and quality suffer:
Mismatched responsibility and authority
Overburdened database operations
personnel
Broken feedback loops from production
Reduced developer productivity
@xaprb 6
What Is Database DevOps?
Attributes I’ve seen in companies that apply DevOps to their database:
Developers own database schema, workload, and performance
Developers debug, troubleshoot, and repair their own outages
Schema and data model as code
A single fully-automated deployment pipeline
App deployment includes automated schema migrations
Automated pre-production refresh from production
Automation of database operations, to an RDS-like level
@xaprb 7
From the 2018 State of DevOps Report
Database changes are often a major source of risk and delay when
performing deployments… integrating database work into the software
delivery process positively contributed to continuous delivery… good
communication and comprehensive con guration management that
includes the database matter. Teams that do well at continuous delivery
store database changes as scripts in version control and manage these
changes in the same way as production application changes… when
changes to the application require database changes, these teams discuss
them with the people responsible for the production database
Emphasis mine. See p. 57
@xaprb 8
Bringing DevOps to the Database
Core Elements
1. People
2. Culture
3. Structure and Process
4. Tooling
@xaprb 10
Tooling: Deploy/Release
You need frequent, automated deploys
Eliminate manual work (toil)
Continuous integration and deployment
@xaprb 11
Tooling: Monitoring and
Observability
Instrumentation, telemetry, analytics, monitoring, observability
Monitoring: the Seven Golden Signals (CELT + USE)
Observability is built on these foundations
@xaprb 12
Tooling: Shared Knowledge and
Process
DBRE processes, mentality,
rubrics
Deploy con dence
procedures
Documentation to share SME
experience & skill
Noti cations should link to
runbooks
@xaprb 13
Structure: Team Orientation
Teams work best when they:
Are service- or product-oriented
Are loosely coupled, autonomous
Are highly aligned and trusted
Own what they build
My personal experience: Conway’s Law is true.
@xaprb 14
Process: First, Do No Harm
Stabilize the patient, then
transport
Protect production (no holes
below the waterline)
@xaprb 15
Process: Plan And
Roadmap
Work is work—maintain a single
backlog
Embrace DevOps in small chunks
and build on success
Lay out the progression in stages
Emphasize what’s not changing, too
@xaprb 16
Process: Getting Started
Pick a place to start.
One team
One app, service, or product
First new, then established/legacy
@xaprb 17
Culture: Change
Culture is emergent; you can’t
operate on it directly
Create a new path of least resistance
Include everyone in consequences
and bene ts
@xaprb 18
Culture: Leadership Support
Exec mandate sometimes works; so does attraction
Starve the old way
Leadership isn’t just top of org chart
@xaprb 19
Culture: Communication
and Trust
Align around a North Star: a simple,
compelling why
Customer-centric culture, customer
empathy
Psychological safety enables risk-taking
@xaprb 20
People: You Need Experts
You do need database expertise
You do not need a database caretaker
Engineers can learn database competency
@xaprb 21
Pathways To Failure
Tooling FAIL
Fragile or too-ambitious automation
Lack of automation / accepting manual toil
Two routes to production—code vs DB
@xaprb 23
Culture FAIL
Any friction in the way of change
Failure to create incentives to change
Relying on a vendor to bring culture
Insisting on adherence to One True Way
Clinging to legacy DBA roles and duties
@xaprb 24
Aside: Legacy DBA Roles
In a traditional sense, the job of the
DBA means she is the only person
with access to the servers that host
the data, the go-to person to create
new database cluster for new
features, the person to design new
schemas, and the only person to
contact when anything database
related breaks in a production
environment.
— Silvia Botros, SendGrid
@xaprb 25
Leadership FAIL
Underinvesting in experience and skill
Lack of management support
Micromanagement, failure to manage up
See also my Kafka Summit talk on advocating technical decisions.
@xaprb 26
Planning FAIL
All-or-nothing
Too much too fast
Velocity over resilience
@xaprb 27
What’s The Hardest Part?
Challenge: Politics
Selling DevOps to the DBA
Selling DevOps to leadership
Creating culture change
@xaprb 29
Challenge: Tooling
Legacy databases aren’t cloud-native
Data tier ops tooling isn’t as mature
Schema changes
Integration with e.g. canary deploys, feature ags
@xaprb 30
Challenge: HA, Scale, Performance
Failover and recovery
Locking/blocking
Nonblocking schema changes at scale
@xaprb 31
Whither The DBA?
Become a DBRE instead of a DBA
Focus on data platform and architecture
Be the subject matter expert supporting product teams
@xaprb 32
The Rewards
The outcomes
The process itself
Individual bene ts
@xaprb 33
Acknowledgments
Many people contributed to this talk. Thank you.
(When I get their OK I will add their names.)
@xaprb 34
Slides and Contact Information
Slides are at
https://www.xaprb.com/talks/ or you
can scan the QR code.
Contact: baron@vividcortex.com,
@xaprb
@xaprb 35
Appendix: Survey
I asked my Twitter followers to respond to an informal, nonscienti c
survey. The following two slides summarize some of the
quantitative feedback.
@xaprb 36
Survey Results
How important are each of the following in your view of “Database DevOps”?
Automation
Is it important to be DevOps-y with databases?
Visibility and measurement
CI, CD, etc
Tight feedback loops
Developers own DB performance of their code
Avoiding specialized roles/privileges/human bottlenecks
Autonomy
Infrastructure as code
Shared ownership, alignment
0 15 30 45 60
Very important Somewhat important Not important
@xaprb 37
Survey Results Cont’d
How do you rate the importance of these factors in making progress?
Cultural change was required
Buy-in from engineers
Tooling
Individuals championing the cause
Leadership from above
Databases are hard to DevOps
0 15 30 45 60
Very important Somewhat important Not important
@xaprb 38
Resources
Slides/Video: How to Monitor a Database including the Seven Golden
Signals (CELT+USE)
The Phoenix Project
Database Reliability Engineering
The 2018 DORA State of DevOps Report
Three Steps To Psychological Safety
My Kafka Summit talk on advocating technical decisions
@xaprb 39
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
devops-database

Más contenido relacionado

Más de C4Media

Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDC4Media
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine LearningC4Media
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at SpeedC4Media
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsC4Media
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsC4Media
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerC4Media
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleC4Media
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeC4Media
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereC4Media
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing ForC4Media
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data EngineeringC4Media
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
 
Navigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery TeamsNavigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery TeamsC4Media
 
High Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in AdtechHigh Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in AdtechC4Media
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/awaitC4Media
 
Opportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven UtopiaOpportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven UtopiaC4Media
 
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/DayDatadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/DayC4Media
 
Are We Really Cloud-Native?
Are We Really Cloud-Native?Are We Really Cloud-Native?
Are We Really Cloud-Native?C4Media
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseC4Media
 
A Dive into Streams @LinkedIn with Brooklin
A Dive into Streams @LinkedIn with BrooklinA Dive into Streams @LinkedIn with Brooklin
A Dive into Streams @LinkedIn with BrooklinC4Media
 

Más de C4Media (20)

Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CD
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine Learning
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at Speed
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep Systems
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.js
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly Compiler
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix Scale
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's Edge
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home Everywhere
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing For
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data Engineering
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 
Navigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery TeamsNavigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery Teams
 
High Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in AdtechHigh Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in Adtech
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/await
 
Opportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven UtopiaOpportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven Utopia
 
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/DayDatadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
 
Are We Really Cloud-Native?
Are We Really Cloud-Native?Are We Really Cloud-Native?
Are We Really Cloud-Native?
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL Database
 
A Dive into Streams @LinkedIn with Brooklin
A Dive into Streams @LinkedIn with BrooklinA Dive into Streams @LinkedIn with Brooklin
A Dive into Streams @LinkedIn with Brooklin
 

Último

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 

Último (20)

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 

DevOps for the Database

  • 1. DevOps for the Database Baron Schwartz • QConSF 2018
  • 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ devops-database
  • 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  • 4. Introduction I’ve been focused on databases for about two decades, rst as a developer, then a consultant, and now a startup founder. I’ve written several books (High Performance MySQL)… and created a lot of open source software, mostly focused around database monitoring, database operations, and database performance: innotop, Percona Toolkit, Percona Monitoring Plugins among others. @xaprb 2
  • 5. Agenda Overview of database DevOps How companies succeed Tooling, culture, process, people How companies fail Challenges to database DevOps Resources, etc Slides are online at xaprb.com. My Twitter is @xaprb, my email is baron@vividcortex.com @xaprb 3
  • 6. Three Database DevOps Stories 1. One DBA, hundreds of developers, growing the DBA team @xaprb 4
  • 7. Three Database DevOps Stories 1. One DBA, hundreds of developers, growing the DBA team 2. One DBA, from 20 to 100 developers @xaprb 4
  • 8. Three Database DevOps Stories 1. One DBA, hundreds of developers, growing the DBA team 2. One DBA, from 20 to 100 developers 3. Two database ops folks, seventeen developers @xaprb 4
  • 9. Benefits of Database DevOps DevOps brings the same bene ts to the database as everywhere else. @xaprb 5
  • 10. Benefits of Database DevOps DevOps brings the same bene ts to the database as everywhere else. For software delivery performance in particular: Faster, better, cheaper—pick all three stability -> speed -> stability -> speed (See the 2018 State of DevOps Report!) @xaprb 5
  • 11. Detriments of Lacking DevOps Without DevOps, speed and quality suffer: Mismatched responsibility and authority Overburdened database operations personnel Broken feedback loops from production Reduced developer productivity @xaprb 6
  • 12. What Is Database DevOps? Attributes I’ve seen in companies that apply DevOps to their database: Developers own database schema, workload, and performance Developers debug, troubleshoot, and repair their own outages Schema and data model as code A single fully-automated deployment pipeline App deployment includes automated schema migrations Automated pre-production refresh from production Automation of database operations, to an RDS-like level @xaprb 7
  • 13. From the 2018 State of DevOps Report Database changes are often a major source of risk and delay when performing deployments… integrating database work into the software delivery process positively contributed to continuous delivery… good communication and comprehensive con guration management that includes the database matter. Teams that do well at continuous delivery store database changes as scripts in version control and manage these changes in the same way as production application changes… when changes to the application require database changes, these teams discuss them with the people responsible for the production database Emphasis mine. See p. 57 @xaprb 8
  • 14. Bringing DevOps to the Database
  • 15. Core Elements 1. People 2. Culture 3. Structure and Process 4. Tooling @xaprb 10
  • 16. Tooling: Deploy/Release You need frequent, automated deploys Eliminate manual work (toil) Continuous integration and deployment @xaprb 11
  • 17. Tooling: Monitoring and Observability Instrumentation, telemetry, analytics, monitoring, observability Monitoring: the Seven Golden Signals (CELT + USE) Observability is built on these foundations @xaprb 12
  • 18. Tooling: Shared Knowledge and Process DBRE processes, mentality, rubrics Deploy con dence procedures Documentation to share SME experience & skill Noti cations should link to runbooks @xaprb 13
  • 19. Structure: Team Orientation Teams work best when they: Are service- or product-oriented Are loosely coupled, autonomous Are highly aligned and trusted Own what they build My personal experience: Conway’s Law is true. @xaprb 14
  • 20. Process: First, Do No Harm Stabilize the patient, then transport Protect production (no holes below the waterline) @xaprb 15
  • 21. Process: Plan And Roadmap Work is work—maintain a single backlog Embrace DevOps in small chunks and build on success Lay out the progression in stages Emphasize what’s not changing, too @xaprb 16
  • 22. Process: Getting Started Pick a place to start. One team One app, service, or product First new, then established/legacy @xaprb 17
  • 23. Culture: Change Culture is emergent; you can’t operate on it directly Create a new path of least resistance Include everyone in consequences and bene ts @xaprb 18
  • 24. Culture: Leadership Support Exec mandate sometimes works; so does attraction Starve the old way Leadership isn’t just top of org chart @xaprb 19
  • 25. Culture: Communication and Trust Align around a North Star: a simple, compelling why Customer-centric culture, customer empathy Psychological safety enables risk-taking @xaprb 20
  • 26. People: You Need Experts You do need database expertise You do not need a database caretaker Engineers can learn database competency @xaprb 21
  • 28. Tooling FAIL Fragile or too-ambitious automation Lack of automation / accepting manual toil Two routes to production—code vs DB @xaprb 23
  • 29. Culture FAIL Any friction in the way of change Failure to create incentives to change Relying on a vendor to bring culture Insisting on adherence to One True Way Clinging to legacy DBA roles and duties @xaprb 24
  • 30. Aside: Legacy DBA Roles In a traditional sense, the job of the DBA means she is the only person with access to the servers that host the data, the go-to person to create new database cluster for new features, the person to design new schemas, and the only person to contact when anything database related breaks in a production environment. — Silvia Botros, SendGrid @xaprb 25
  • 31. Leadership FAIL Underinvesting in experience and skill Lack of management support Micromanagement, failure to manage up See also my Kafka Summit talk on advocating technical decisions. @xaprb 26
  • 32. Planning FAIL All-or-nothing Too much too fast Velocity over resilience @xaprb 27
  • 34. Challenge: Politics Selling DevOps to the DBA Selling DevOps to leadership Creating culture change @xaprb 29
  • 35. Challenge: Tooling Legacy databases aren’t cloud-native Data tier ops tooling isn’t as mature Schema changes Integration with e.g. canary deploys, feature ags @xaprb 30
  • 36. Challenge: HA, Scale, Performance Failover and recovery Locking/blocking Nonblocking schema changes at scale @xaprb 31
  • 37. Whither The DBA? Become a DBRE instead of a DBA Focus on data platform and architecture Be the subject matter expert supporting product teams @xaprb 32
  • 38. The Rewards The outcomes The process itself Individual bene ts @xaprb 33
  • 39. Acknowledgments Many people contributed to this talk. Thank you. (When I get their OK I will add their names.) @xaprb 34
  • 40. Slides and Contact Information Slides are at https://www.xaprb.com/talks/ or you can scan the QR code. Contact: baron@vividcortex.com, @xaprb @xaprb 35
  • 41. Appendix: Survey I asked my Twitter followers to respond to an informal, nonscienti c survey. The following two slides summarize some of the quantitative feedback. @xaprb 36
  • 42. Survey Results How important are each of the following in your view of “Database DevOps”? Automation Is it important to be DevOps-y with databases? Visibility and measurement CI, CD, etc Tight feedback loops Developers own DB performance of their code Avoiding specialized roles/privileges/human bottlenecks Autonomy Infrastructure as code Shared ownership, alignment 0 15 30 45 60 Very important Somewhat important Not important @xaprb 37
  • 43. Survey Results Cont’d How do you rate the importance of these factors in making progress? Cultural change was required Buy-in from engineers Tooling Individuals championing the cause Leadership from above Databases are hard to DevOps 0 15 30 45 60 Very important Somewhat important Not important @xaprb 38
  • 44. Resources Slides/Video: How to Monitor a Database including the Seven Golden Signals (CELT+USE) The Phoenix Project Database Reliability Engineering The 2018 DORA State of DevOps Report Three Steps To Psychological Safety My Kafka Summit talk on advocating technical decisions @xaprb 39
  • 45. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ devops-database