This document discusses reducing operational toil through DevOps practices. It defines toil as interrupt-driven, manual work that prevents teams from focusing on innovation. Examples of toil include manually provisioning environments and troubleshooting failed deployments. The document recommends automating processes, enabling self-service, tracking toil levels, and focusing on reducing toil for overburdened teams first. Teams should aim to spend no more than 50% of their time on toil. Reducing toil allows teams to spend more time on strategic work that adds value.

1. Empower Devs, Simplify Ops,
and Accelerate your Digital Transformation

2. Your Hosts
Anders Wallgren
VP Technology Strategy,
CloudBees
Sam Fell
DevOps Enthusiast,
CloudBees
Damon Edwards
Chief Product Officer,
Rundeck

3. Today’s Topic: “TOIL”
From development to operations, interrupt-driven work and failed
deployments erode productivity and reduces time available to
innovate and add strategic value.
Join us as we discuss ideas designed to help you
● Deliver software services more efficiently, without the toil
● Use self-service to avoid “death by a thousand service desk
tickets”
● Eliminate variance and drift and while increasing security and
uptime

4. Toil: Giving a Name to a Problem We’ve All Felt

5. Toil vs Engineering Work

6. Excessive Toil Prevents Fixing the System

7. Excessive Toil Prevents Fixing the System

8. Excessive Toil Prevents Fixing the System

9. Toil: A Naturally Recurring Force

10. Toil: A Naturally Recurring Force
Launch
(ToDos & Unknowns)
Mature

11. Toil: Not Just an Operations Concern
Day 1 Day 2

12. Day 1
Development

13. Toil: Giving a Name to a Problem We’ve All Felt
(“DEV REMIX”)
delivering
application complexity
*as modified by Anders

15. Things that create TOIL in Dev
● Lack of shared visibility across Dev-QA-Ops
● Manually operated, non-integrated tool chains
● Different needs, processes and tools across teams and locations
● Brittle, error-prone, non-scalable non-share-able home-built systems
● No rollup traceability, auditability
● Not repeatable, predictable
● Wasted time, money dealing with errors
● Low infrastructure utilization
“What’s build # is
running in QA-DEN07”

16. ● Enable Self-Service Environment Provisioning, De-Provisioning
○ Spin up production-like environments anywhere in the cycle. Tear them down when finished.
● Enable Self-Service Deployments
○ Allow teams to predictably and efficiently deliver application versions, on-demand.
○ Eliminate snowflakes/singletons
● Version everything (applications, infrastructure and middleware)
● Break up monoliths - use loosely coupled architectures
● Take an engineering approach to pipeline improvement
○ Automate (and Smoke Test) everything
○ Governance and Auditability as a side-effect of automation
● Shift-left your Security
● Give each team visibility into all appropriate information
Start reducing toil in DEV today

17. All of these tips are backed up by
research: The DevOps Handbook
● 5+ years in the making
● 23 chapters
● 48 case studies
● 98,124 words
● 48 images
● 503 endnotes
● 192 footnotes
How do we know this?

18. Day 2
Operations

19. Start reducing toil today
1. Track toil levels for each team

20. Start reducing toil today
1. Track toil levels for each team
2. Set toil limit for each team (50% is conventional wisdom)

21. Start reducing toil today
1. Track toil levels for each team
2. Set toil limit for each team (50% is conventional wisdom)
3. Fund efforts to reduce toil (with emphasis on teams already over limit)

22. Start reducing toil today
1. Track toil levels for each team
2. Set toil limit for each team (50% is conventional wisdom)
3. Fund efforts to reduce toil (with emphasis on teams already over limit)
Michael Kehoe
Todd Palino
(LinkedIn)
At SREcon Americas 2019
Example
Process
“Code Yellow”

23. Where to focus?
Reduce
Technical Debt
Re-Engineer
Processes
Enable
Self-Service

25. Eliminate Interruptions
Eliminate Waiting
Automated
Runbooks
Do X.
… and a lot less toil

26. Empower teams to spot and fix the anti-patterns.

27. “Fix this for me, fix it again, then fix it again.”

28. “I could fix it, but I can’t get to it.”

29. “The dog-pile.”

30. “I’m an expert, I don’t read the wiki.”

31. “Known issue… doesn’t get permanent fix”

32. Q&A