SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps

1. Mentoring Devs Into DevOps Justin Carmony Director of Development Deseret Digital Media

2. Lets Measure The Audience • Who here is a… • System Administrator? • Developer? • Manager / Management? • “DevOp?”

3. Confession: I’m a Developer

4. Self-Taught Ops Because There Was No One Else To Do It

5. About Me • Director of Development  for Deseret Digital Media • Utah PHP Usergroup  President • I Make (and Break)   Web Stuff (10 years) • Salt User in Production since 0.8 (I <3 Salt)

6. This Presentation • Lessons learned at DDM & previous jobs • Insight into our process of increasing “DevOps” • We’re still learning, but this what we’ve found. • Slides will be posted online, so don’t worry about copying slide content. • Feel free to ask on-topic questions during, and we’ll have questions at the end.

7. About DDM • Deseret Digital Media runs local website like KSL.com, DeseretNews.com • Running National and International Websites like OK.com, familia.com.br, etc. • ~10 million pageviews a day across sites. • ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.

8. Lets Start With a Story!

9. You Work for an Awesome Tech Company

10. Team Is Working Hard to Build New Things!

11. You launch your awesome product!

12. A Few More Features…

13. … and next thing you know…

14. Awesome Job Team, We Rock!

15. We Need ! Real-Time XYZ Feature! ASAP!

16. We Need ! Real-Time XYZ Feature! ASAP!

17. We Need ! Real-Time XYZ Feature! ASAP! &#$%!

18. “Huh, it works if you ! just turn off caching…”! - Dev @ 80th Hour This Week

19. “I’m sure this ! will work…”

20. “Our servers are melting!”

21. “We Need a Better Solution!”

22. So…

23. Where Do We Start?

24. We Have This Problem

25. Challenges We Faced • Giant mesh-up of technologies • Tightly-coupled & fragile infrastructure • Debugging production only bugs was difﬁcult • Bugs that were part code, part environment were a nightmare to track down.

26. So One Day… We Had A Genius Idea!

27. Lets Hire a DevOp!

28. I’m Not Joking We Actually Said This

29. Two Problems with this “Idea”

30. Problem #1 - We Didn’t Understand What We Really Wanted

31. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Proﬁt!

32. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Proﬁt! Everything Works ! Perfectly!

33. Problem #2 - People Who Are Great At Dev & Ops Are Hard To Find

34. Expectation:

35. Reality:

36. Honest Team Discussion: What is it we’re really looking for?

37. We Discovered a Few Things

38. What does DevOps Mean To Us? • DevOps: Dev & Ops, a Culture of Collaboration • Our Goal: “10 deploys a day without issues” • Everyone shares the goal of quick development of features AND a stable system that stays up.

39. Team Structure Devs: 30 Ops: 2

40. Team Structure Devs: 30 Ops: 2 DevOps: 1

41. Team Structure Devs: 30 Ops: 2 DevOps: 1

42. Team Structure Devs: 30 Ops: 2 DevOps: 1 Hiring one person won’t just solve all our problems!

43. Team Realizations • Hardest problem already solved: awesome team • No foreseeable rapid expansion, must operate at our current scale • Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.

44. Teams Already Had Some Ad-Hoc DevOps Tools - Real-time Logging - Capistrano Deploys - Nagios Alerts - Server Metrics - Puppet for File Mgmt - App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management - Homebrewed Metrics Sys. - Homebrewed Alert System

45. Teams Already Had Some Ad-Hoc DevOps Tools - Real-time Logging - Capistrano Deploys - Nagios Alerts - Server Metrics - Puppet for File Mgmt - App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management - Homebrewed Metrics Sys. - Homebrewed Alert System

48. We Formed A Strategy

49. Step #1: Promote Dev to DevOp Role

50. WAIT! Isn’t that the advice you just said was a bad idea?!

51. DevOp Engineer • Well Deﬁned Role: • Ownership over the TOOLS to improve DevOps efforts. • Resource for other teams to help use DevOps Tools. • Easy to work with, aptitude for systems & ops, likes to try new things.

52. Promoting From Within • A seasoned dev for your team already knows: • Your Pain Points • Your System’s Quirks • How the “Chaos Works” • Knows the people & personalities on your team

53. Step #2: Change Team Structure

56. Team Structure Goal: Spread Out Expertise By Increasing Ops Experience & Skills Among Devs Dev Ops

57. Team Structure Goal: Spread Out Expertise By Increasing Ops Experience & Skills Among Devs Dev Ops

58. Team Structure Dev Ops

60. Increasing Ops Among Devs • Identify Devs who liked “Ops” & wanted to Learn • Pair Dev with Op / Director • Learning Dev works on things, not Op /Director. • Pair program if needed.

61. Step #3: Increase Everyone’s Insight

62. Step #3: Increase Everyone’s Insight

63. Metrics • Everyone has access to Network, Server, and Application Metrics. • Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite • Each developer trained to add & track metrics in production. • We’re okay with 98% uptime of stats to avoid complexity.

64. Real-Time Logging

65. Real-Time Logging • Harder & more complicated at scale • Still trying to solve well, we have lots of logs. • Start with small window of data (i.e. 48 hours) and start to expand window. • We’re trying Logstash, ElasticSearch, and Kibana right now. • Generate Statistics off our Logs

66. Tracking Changes • Everything, everything, everything in git   (we use GitHub) • Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed * * - Your can merge w/o a review, but must be willing to defend your choice

67. Deploys

68. Everyone Can Deploy • Automated our deployment process to a single step. • Everyone can deploy, deployments are logged • Easy rollback is a requirement! • Implementing feature ﬂags to turn off single parts of our application.

69. Tests Unit Functional Integration Acceptance etc

70. Automated Tests • If you want to trust your Devs, you need tests • Legacy apps we wrote Integration Tests • New Apps & Refactored Legacy Parts have Unit Tests • Continuous Integration to make sure tests run

71. So, where’s the salt?

72. Step #4: Devs Use The Ops Tools

73. Devs can grok salt

74. Safe Environment For Devs to Learn

75. Safe Environment For Devs to Learn salt * cmd.run "rm -rf /tmp /*"

76. Safe Environment For Devs to Learn salt * cmd.run "rm -rf /tmp /*" Salt is awesome, but it can’t ! recover from that

77. Dev Salt Master Devs Can Look Into Every Server

78. Dev Salt Master • Every server has two minions: • Admin Salt (aka root) • Dev Salt (aka bob) • Each connect to different master server: • All Devs have access to Dev Salt Master • Trusted Devs get access to Admin Salt Master

79. Everything Salty in Git Reminder:

80. Dev Environment • Developers own the Dev Environment • Dev Teams manage the Salt States for their Env • Vagrant + Salt for their Env • Who makes changes? Developers • DevOp helps advise & offer support

82. Stage Environment • Stage & Production use same salt repos, different branches • Developers make all the changes for Application Servers • All Changes through Pull Requests • We’ll worry about env changes before code • Small changes we quickly release, large or long running branches are scary & dangerous

83. Production Environment • Merge change to Production Branch • salt * state.highstate • Reminder: Small quick changes over time, never a large change at once.

84. Environment Caveats • Ops & DevOps Manage VM Hosts, Physical Load Balancers, FireWalls, etc • Ops & DevOps manage servers that deal with data: • MySQL • MongoDB • etc

85. Mentoring Devs

86. Mentoring Devs • Not every Dev will become an amazing DevOp • Thats okay!

87. Level of “DevOps” Skills • Thinks about their impact on Ops: Everyone • Able to debug issues with production: Most • Able to make changes to environments: Many • “Awesome DevOp”: Some

88. Team’s “DevOps” Skills 0 25 50 75 100 Think Debug Change DevOp Current Goal

89. So Everything Is Awesome for us, right?

90. Honesty Slide: We Have Skeletons In Our Closets

91. Where We Are At • All Dev Environments using Vagrant + Salt • All New Stage & Prod Environments are Salty • Some Legacy Stage & Production Envs are Salty • Continuously working on getting out stuff salty.

92. Making This Work For Your Team

93. Honest Introspection • Determine for your team what are your… • Strengths • Weaknesses • Problems • Goals

94. Increase Team’s Insight • Make sure devs can see & understand how their code performs • Increase responsibility of team for those metrics. • If they break it, they ﬁx it.   Do not always bail them out. • Everyone can see everything.

95. Increase Team’s Insight • Make sure devs can see & understand how their code performs • Increase responsibility of team for those metrics. • If they break it, they ﬁx it.   Do not always bail them out. • Everyone can see everything.

96. Mentor Those With Desire / Aptitude • Give Developers Safe Environment to Learn • Let them submit code-reviewed changes for Stage & Production • When teaching / mentoring, let the learner drive, kindly offer advice and help. • It takes time, but worth the investment.

97. A Few Final Thoughts

98. Team Culture Matters

99. Positive Inﬂuence

100. Questions?

101. Thank You Justin Carmony Email: justin@justincarmony.com Twitter: @JustinCarmony IRC: carmony #salt #uphpu Website: justin@justincarmony.com

102. p.s. we’re hiring, email / pm / tweet me

SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps

SaltStack

SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps