Despite the best of intentions, we sometimes find ourselves working on a team of size one. Groups shrink for many reasons: attrition, mergers and acquisitions, transfers, and financial distress. It's never comfortable being a Single Point of Failure, but how can you survive this state of non-redundancy? Are there any benefits to being a team of "me, myself, and I", or is it all a pit of despair? What kind of red flags should you be on the lookout for? And, most importantly, what compelling leverage can you try to use to encourage team growth back to a reasonable size? At DevOps Days Silicon Valley 2015, I shared a series of unfortunate events that led to my current status: the Human SPoF; I also discussed some of the tactics I've used to survive. Automation, tools, and code-as-infrastructure are a force multiplier applied correctly, allowing one engineer to do the work of many. However, these wonders come with a price tag. I also covered some strategies to grow a team, and ways to maintain sanity while keeping the lights blinking and the disks spinning in a 24x7 real-time environment with over 2000 servers.

1. DevOps Army of One
Jonathan Disher

2. EHLO jon
• Principal DevOps Engineer, Oracle Data Cloud

3. EHLO jon
• Principal DevOps Engineer, Oracle Data Cloud
• Connoisseur of Fountain Pens

4. Redundant Array of Inexpensive-ish Styli
[RAIS]

5. EHLO jon
• Principal DevOps Engineer, Oracle Data Cloud
• Connoisseur of Fountain Pens
• Amateur Miata Mechanic

6. Which way to the datacenter?!
Image by Dito Milan, GotBlueMilk Photography

7. EHLO jon
• Principal DevOps Engineer, Oracle Data Cloud
• Connoisseur of Fountain Pens
• Amateur Miata Mechanic
• Briefly a load balancer

8. EHLO jon
• Principal DevOps Engineer, Oracle Data Cloud
• Connoisseur of Fountain Pens
• Amateur Miata Mechanic
• Briefly a load balancer
• More on that later

9. EHLO jon
• Principal DevOps Engineer, Oracle Data Cloud
• Connoisseur of Fountain Pens
• Amateur Miata Mechanic
• Briefly a load balancer
• More on that later
• Single Point of Failure

10. Poll: How many of you are now,
or have previously, worked in
DevOps groups of size==1?
(Specifically where the group was, at one time, size > 1,
and shrank to 1)

11. Once upon a time…
“Now, children, come on over here. I'm going to tell you a bedtime story.
Are you sitting comfortably? Then I'll begin.”
— John Wood, War Games (1983)

12. Once upon a time…
• 11/2012 - 3 engineers, 450 hosts

13. Once upon a time…
• 11/2012 - 3 engineers, 450 hosts
• 06/2013 - 2 engineers, 800 hosts

14. Once upon a time…
• 11/2012 - 3 engineers, 450 hosts
• 06/2013 - 2 engineers, 800 hosts
• 9/2013 - 1 eng, 1000 hosts

16. Once upon a time…
• 11/2012 - 3 engineers, 450 hosts
• 06/2013 - 2 engineers, 800 hosts
• 9/2013 - 1 eng, 1000 hosts
• 03/2014 - ORCL acquisition. 1 eng, 1300 hosts

17. Once upon a time…
• 11/2012 - 3 engineers, 450 hosts
• 06/2013 - 2 engineers, 800 hosts
• 9/2013 - 1 eng, 1000 hosts
• 03/2014 - ORCL acquisition. 1 eng, 1300 hosts
• 06/2015 - 1 eng, 2400 hosts

18. Once upon a time…
• 11/2012 - 3 engineers, 450 hosts
• 06/2013 - 2 engineers, 800 hosts
• 9/2013 - 1 eng, 1000 hosts
• 03/2014 - ORCL acquisition. 1 eng, 1300 hosts
• 06/2015 - 1 eng, 2400 hosts
• 11/2015 - 1 eng, 1500 hosts, 7 candidates!

20. Punctum Singulari Deficiendi

22. Punctum Singulari Deficiendi
• We’re always on the lookout for SPoF’s
• Hardware

23. "Single Point of Failure" by Charles Féval - http://commons.wikimedia.org/wiki/File:SPOF.png.
Licensed under CC BY-SA 2.5 via Commons - https://commons.wikimedia.org/wiki/File:Single_Point_of_Failure.png

24. Punctum Singulari Deficiendi
• We’re always on the lookout for SPoF’s
• Hardware
• Software

26. Punctum Singulari Deficiendi
• We’re always on the lookout for SPoF’s
• Hardware
• Software
• We often forget the wetware!
• Domain knowledge, environmental quirks

28. Handling The Workload

29. Handling The Workload
-or- Fifteen Pounds of Ops in a Five Pound Sack

30. Handling The Workload
• Automation and tools
• Provisioning

31. I need another monitor…

33. <3 csshX

34. Handling The Workload
• Automation and tools
• Provisioning
• Hostile hosting portals

35. Handling The Workload
• Automation and tools
• Provisioning
• Hostile hosting portals
• Access control

41. Downsides

42. Downsides
Wait, there’s a downside??!

43. Downsides
• Just because you are physically able to do it by
yourself, doesn’t mean you should.

44. Downsides
• Just because you are physically able to do it by
yourself, doesn’t mean you should.
• Tools and Automation have created breathing
room, not an excuse to not hire more people

45. Remediation & Survival

46. Remediation & Survival
• Learn to say ‘no’.

47. This is not you.

48. This is not you.
It took me a long time to
realize this.

49. Remediation & Survival
• Learn to say ‘no’.
• Leave work behind on Friday.

52. Remediation & Survival
• Learn to say ‘no’.
• Leave work behind on Friday.
• Community support.

53. Remediation & Survival
• Learn to say ‘no’.
• Leave work behind on Friday.
• Community support.
• Take your vacations!

54. Remediation & Survival
• Learn to say ‘no’.
• Leave work behind on Friday.
• Community support.
• Take your vacations!
• Hire more people!

55. Growing Your Team
It’s easy to hire more people, right?

56. Growing Your Team
• Ask for headcount. Come prepared with data.
• Ticket counts, emergencies, interrupts.
• Estimated vs actual time to complete projects

58. Growing Your Team
• Ask for headcount. Come prepared with data.
• Ticket counts, emergencies, interrupts.
• Estimated vs actual time to complete projects
• Delegation and Enablement
• MUST be people you trust

59. Growing Your Team
• Ask for headcount. Come prepared with data.
• Ticket counts, emergencies, interrupts.
• Estimated vs actual time to complete projects
• Delegation and Enablement
• MUST be people you trust
• Enlistment and Poaching

60. If all else fails…

64. So what was that about
a load balancer?

66. Thank you!