Self-Service Operations: Because Failure Still Happens (Developer Edition)
•
3 likes•1,518 views
Keynote presentation at DevNet Create 2017 by Damon Edwards, co-founder of Rundeck. Agile and DevOps have provided plenty of lessons for how to speed up the pace of application delivery and the frequency of application deployment. But delivery and deployment only covers one part of the day-to-day life of developers in large enterprises. What about what happens after deployment? In many enterprises, increasing the pace of delivery and frequency of deployment has just increased the operational support load, work interrupts, and context switching that were already cutting deeply into development teams' time. This talk will focus on the successful design patterns that high-performing, large scale organizations have applied to reduce the operational burden and support costs across their entire organization. Specifically, we’ll look at how they apply DevOps principles to improving the post-deployment lifecycle and how Developers play the key role in reducing the difficultly and cost of operations activity for everyone.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Self-Service Operations: Because Failure Still Happens (Developer Edition)
Similar to Self-Service Operations: Because Failure Still Happens (Developer Edition) (20)
More from Rundeck
More from Rundeck (20)
Recently uploaded
Recently uploaded (20)
Self-Service Operations: Because Failure Still Happens (Developer Edition)
- 1. Self-Service Operations: Because Failure Still Happens Damon Edwards @damonedwards
- 4. Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens!
- 5. Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens!
- 6. Deploy. Deploy. Deploy. Deploy. DevOps Dream Deploy. Deploy. Deploy. Depl… uh oh… we have a problem DevOps Reality Ops still happens! Escalations! Interruptions! Delays! Context Switching! Bridge Calls! Arguments!
- 7. Let’s start with a story…
- 11. 90% Reduction in MTTR 50% Reduction in escalations 55% Reduction of overall support costs
- 12. How did they do that?
- 13. But first…
- 14. Let’s look at the principles behind the improvement …
- 15. Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
- 16. Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service Development Team Operations Team Dev Ops Integrated Delivery Team
- 17. Two schools of thought about operations support Running Service “You build it. They run it.” “You build it. You run it.” Running Service “two-pizza team” Development Team Operations Team Dev Ops Integrated Delivery Team
- 18. “You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s the NOC… Talk them through: health checks, reviewing log files, and process of diagnosing and recovering the system. Same as you did for dev teams 2 months ago, QA teams last month, Ops during deploy last week, etc.
- 19. “You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm ….
- 20. “You build it. They run it.” (aka… the way it always was) It’s 2am …. It’s 2pm …. It’s Ops… “Will your applications be affected if we take down EU-West?” “How do we reconfigure your app if we change these firewall rules?” “We are getting customer complaints about performance. Help us figure out what changed”.
- 21. “You build it. They run it.” (aka… the way it always was) Running Service Development Team Operations Team
- 22. “You build it. They run it.” (aka… the way it always was) Running Service Development Team Operations Team
- 23. “You build it. You run it.” Dev Ops Integrated Delivery Team
- 24. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Dev Ops Integrated Delivery Team
- 25. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
- 26. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
- 27. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
- 28. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
- 29. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team
- 30. “You build it. You run it.” Running Service Running Service Running Service Running Service Running Service Running Service ? Incident!! Incident!! What would happen if… New feature!! New feature!! New API!! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Running Service Add this to your responsibilities! Dev Ops Integrated Delivery Team “two-pizza teams”? Just change how business is structured, funded, and operated.
- 31. How do we get the best of both models?
- 32. Have the labor scaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models?
- 33. Have the labor scaling benefits of “you build it, they run it” without the frequent escalations the bad handoffs How do we get the best of both models? Have the responsiveness/control of “you build it, you run it” without the scaling limitations
- 34. “Self-Service Operations” design pattern is the key enabler
- 35. “Self-Service Operations” design pattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor
- 36. “Self-Service Operations” design pattern is the key enabler Split definition, execution, and management control and moved to where most effective use of labor Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
- 37. What do Developers get out of Self-Service Operations?…
- 38. Reduced escalations and quicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
- 39. Reduced escalations and quicker resolutions Finish Deliverables Interrupt Interrupt ? ? ? ? Interrupt X "Too busy" "We're late!" Start Deliverables Fromcurrentproduction Finish Deliverables Interrupt ? ? ? ? Start Deliverables Fromcurrentproduction "This looks important"Interrupt ✔ Delivery Team (L2, L3) Delivery Team (L2, L3) NOC NOC NOC NOC NOC NOC NOC NOC Previously delivered Rundeck Jobs Old Model New Model
- 40. Tighter feedback loops Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev)
- 41. Ops managers can focus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
- 42. Ops managers can focus more on making your life easier! Old mindset: Protect capacity Say “no” New mindset: Scaling service Get more users Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Manager
- 43. How can Dev help Ops make this a reality…
- 44. First understand the pain of Operations today…
- 45. Operations is getting squeezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps Availability Auditing Security Compliance "Go faster!" "Open up!" "Be more secure!" "Be more reliable!"
- 46. Operations is getting squeezed OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!"
- 47. Ops is Unplanned Work and Planned Work… by design! +
- 48. Planned + Unplanned = Expensive Context Switching Gerald Weinberg via Jeff Atwood https://blog.codinghorror.com/the-multi-tasking-myth/
- 49. Crushing Technical Debt 900 Dev 500 QA / App Sup. 200 Ops
- 50. Crushing Technical Debt 900 Dev 500 QA / App Sup. 200 Ops
- 51. ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
- 52. ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Work Task Work Task Silo A Silo B Work Task ! Handoffs ! Feedback Silos are everywhere
- 53. Cross functional teams? ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
- 54. Cross functional teams? Never enough to go around. Context Work Task Work Task Work Task Work Task Queue ! Handoffs ! Handoffs Team C ContextContext Work Task Work Task Work Task Work Task Queue Work Task Work Task Work Task Work Task Queue Team A Team B
- 55. Silos + Tool Evolution = Islands of Automation Puppet Chef Shell Scripts Data ETL PowershellScripts Network Management Monitoring Ansible Legacy Datacenter Automation ContainerManagement SQL Tools NewTools New Tools
- 56. Squeezed between competing pressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under
- 57. Squeezed between competing pressures Unplanned and planned work Silos are everywhere Crushing technical debt Islands of automation Again: Understand the pressure that Ops is under = Ops never has enough time or people! +
- 58. How developers can help operations…
- 59. 1. Recognize that great Operations starts in Development
- 60. 2. Develop with an “Operable First” mindset
- 61. Develop with an “Operable First” mindset
- 62. Operations is the business (unless you literally sell packaged software) Develop with an “Operable First” mindset
- 63. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Develop with an “Operable First” mindset
- 64. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Develop with an “Operable First” mindset
- 65. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Develop with an “Operable First” mindset
- 66. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Develop with an “Operable First” mindset
- 67. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Develop with an “Operable First” mindset
- 68. Operations is the business (unless you literally sell packaged software) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification-driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests Develop with an “Operable First” mindset
- 69. 3. “Shift Left” as much Ops activity as possible
- 70. “Shift Left” as much Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools “Deploy.” Ops Ah-ha! Dev Ka-ching!
- 71. “Shift Left” as much Ops activity as possible Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!) “Deploy.” “Operate.” Ops Ah-ha! Dev Ka-ching!
- 72. 4. Develop Self-Service Operations capabilities
- 73. Step 1: Establish a Secure Ops Hub Custom Mix
- 74. Step 2: Establish a SDLC for Ops Procedures Custom Mix
- 75. Step 3: Connect with Enterprise Management Systems Custom Mix
- 76. Who created the procedure? Who reviewed it? Who? When? Where? Approval trail? Step 4: Make Compliance Really Happy Custom Mix
- 77. Again: How can Developers help Operations 1. Recognize that great Operations starts in Development 2. Develop with an “Operable First” mindset 3. “Shift Left” as much Ops activity as possible 4. Develop Self-Service Operations capabilities
- 78. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge”
- 79. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams
- 80. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy
- 81. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks
- 82. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations
- 83. Back to our Story… Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html “Support at the Edge” • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations • Combined with new incident response and escalation model
- 84. Recap Understand the pressures on Ops Explicit investment in process and tooling OpsBusiness Idea Shorter Time-to-Market Fast Feedback from Users Dev Ops Running Services Improved Quality Digital and DevOps "Go faster!" "Open up!" Availability Auditing Security Compliance "Be more secure!" "Be more reliable!" Costs "Spend less!" "Do more!" Understand your operating model Operations is the business) Deployability, configurability, monitoring are service features Build configurability into the service, don’t externalize it Demand “prod-like” environments everywhere Make any handoff between teams “verification- driven” Create immutable versioned artifacts and use standard packaging Integration tests over unit tests “Operable First” mindset Self-Service Operations Define actions (optional) Execute actions Define actions (or vet actions) Define policy Build and Scale "Producer" (a.k.a. Ops) "Consumer" (a.k.a. Dev) Apply Self-Service Operations design pattern “Shift left” operational concerns