MadridDevops September 2014 Meeting: "From chef09 to chef11, one approach to devops" Madrid, September 25, 2014 http://madrid.devops.es Just an excuse to talk about devops, chef (and puppet as comparison), its culture , its community and a migration project I have been involved with.

1. From chef09 to chef11
my approach to a real devops task
Antonio Peña @apenav for @MadridDevops
September 2014 http://about.me/apenav

2. http://madrid.devops.es

3. Motivation
● feedback
● sharing experience
● learning
● talking about devops

6. Impostor Syndrome

7. Three Parts
● Devops
● Chef
● Migration

8. About
DevOps

9. Software is broken or will be

10. Devops (I)

11. Devops (II)

12. Devops (III)

13. So little time, so much to talk about

14. Big Business. DevOps Myths (I)

15. Big Business. DevOps Myths (II)

16. My approach to a
configuration management project
Antonio Peña @apenav for @MadridDevops
September 2014 http://about.me/apenav

17. Best tool to make config mgmt?
● cfengine
● puppet
● chef
● salt
● ansible
-- Business decision-making. Make your choice
-- Strengths/weakness. Tend to converge.

18. About
Chef

21. What is Chef?
● a company
○ Opscode Inc.
○ getchef.com
○ @chef
● a product (with paid support)
○ packages, not recommended install from gemfiles
● an open source project
● a community (open source but not only)

22. Beginners, experts and middle class

23. Berkshelf.I

24. Berkshelf.II

25. “Spanish” Way
Master Chef RTVE

26. Howto devops in Spain? (I)
● e.g. Demo in a Big Company

27. Howto devops in Spain? (II)
● startup
● why not?
● sandbox

28. Howto devops in Spain? (III)
● consulting

29. Working in a company
as a Developer.
Daily tasks

30. Initial problem
Problem to be solved (in aprox. 2 weeks):
1. install chef-server 11 in a centos6 behind a
VPN
2. install 2 web frontends + 2 backends
a. (redis s/m + mysql m/s)

31. Real problem
1. refactor/migrate/write from scratch about 20
old-fashioned (2009) but very reliable (daily
used, very difficult to extend for an outsider)
ubuntu ruby18-webapp-related recipes to
newer ones
2. proof of concept to migrate chef09 company’
s repository to anything better

32. From scratch
http://devopsreactions.tumblr.com/post/97637875636/what-happens-when-i-am-allowed-to-rewrite-code-from

33. Methodology or the like
“Big bang” approach, but iterative:
● use chef11 community cookbooks
● almost no direct migration from chef09 ones

34. Criteria to use community cookbooks?
https://github.com/opscode-cookbooks/mysql
When in doubt use popular
opscode
community
cookbooks.
So many content to
follow!

35. About
Migration

36. To the kitchen

37. Let’s cook!
1. Hosting Environment
2. Cookbook Development
3. Monolithic cookbook approach
4. Everything is a cookbook
5. Production Cookbook Deployment
6. Production Cookbook Development
7. Opscode packages
8. Developers’ isolated ruby environment
9. Final Steps
10. Conclusions

38. Chef Server
Internals

39. 1.- Hosting Environment (I)
Open Source chef-server in a centos6.
Modify and test undocumented chef-server
config, with some “little” external problems:
- shared server behind a VPN
- unavailable ports: reassign 80/443 to 81/8443
- 8000 not opened in firewall (so no reports will
be available)

40. Asking for opening ports in firewall
http://devopsreactions.tumblr.com/post/41094252078/asking-the-security-team-for-a-firewall-exception

41. 2.- Cookbook Development (I)
Opscode free(*) hosting
● Great help because Multiproject!
● http://learn.getchef.com
● Berkshelf helped a lot with dependencies (**)
● Good practices freezing versions
(*) not open-source (limited to 10 nodes)
(**) not so easy when not opscode server

42. 2.- Cookbook Development (II)
Chef-DK:
● ruby binaries in specific /opt folders
● not needed rvm/rbenv/ruby source code/...
● own path & pre-installed useful gems and
tools

43. 2.- Cookbook Development (III)
● http://jtimberman.housepub.org/blog/2014/04/30/chefdk-and-
ruby/
● Berkshelf 3.0.
● The Test Kitchen integration testing framework.
● ChefSpec, for unit testing cookbooks.
● Foodcritic, static code analysis on cookbooks.
● All of the Chef tools you're already familiar with: Chef
Client, Knife, Ohai and Chef Zero.

44. 2.- Cookbook Development (IV)
Better practices? Any?
● No time for tests nor TDD (to learn about)
● Not enough RAM in laptop nor in AWS micro
instances for making testing/CI with Vagrant
● Foodcritic and more, but later
● I hope to use them ASAP: just because I like
it. The same with Puppet.

45. 2.- Cookbook Development (V)
First stages of development
● make risks, quick tests, quickwin, try and try
● think in advance, virtualhosts library
● ugly code you know it will be easy to change
● the community code is better than yours
● extend, not create from the ground
● parametrize cookbooks: redis

46. 3.- Monolithic cookbook approach
Thanks a lot to Mathias Lafeldt @mlafeldt
“monolithic cookbook” idea, easy to refactor
later (see Puppet Module Structure Redux).
● http://mlafeldt.github.io/practicing-ruby-cookbook
● https://github.com/elm-city-craftworks/
practicing-ruby-cookbook

47. 4.- Everything is a cookbook
● https://tomduffield.com/everything-as-a-cookbook-
chefconf2014/
● https://speakerdeck.com/tduffield/everything-as-
a-cookbook-1

48. Cool guys don’t look back to explosions?

49. 5.-ProductionCookbookDevelopment
● A new beginning. Almost from scratch.
Develop new recipes and cookbooks against a
new chef organization in opscode free hosting.
● install, fix, check-apply and repeat in
frontend staging instances
(Poor’s man plan–do–check–adjust)
https://en.wikipedia.org/wiki/PDCA

50. 6.-ProductionCookbookDeployment
● chef site cookbook install “cookbook” (new
git branch in ~/chef-repo/cookbook/)
● chef site cookbook install chef-client

51. 7.- Opscode packages (I)
● WARNING: chefdk.rpm installs binaries
under /usr/bin and gems in its own gemsdir
● REMOVE any puppet or ruby preexistent
package or binary, no interferences please!
● BEWARE: never use binaries without explicit
PATH e.g. /opt/chef/bin/chef-client

52. 7.- Opscode packages (II)
NO RVM nor system ruby allowed
● chef-server.rpm
● chef.rpm
● chefdk.rpm
every opscode rpm package installs one or
more ruby binaries in its own path

53. 8.- Developers isolated ruby environment (I)
● Developers need multiple rubies installed
(1.9.3 & 2.0.X)
● unprivileged users will install its own gems,
without interfering with chef ruby-binaries
SOLUTION: compile explicit versions from
source code, rewrite PATH in user environment
and install “bundler gem” as root.

54. 8.- Developers isolated ruby environment (II)
Opscode chef.rpm package provides 2.1 ruby
● install passenger-apache library+binary
compiled and installed as a gem under chef.
rpm gemlib path (a community recipe is in
charge of it)
● compatible to both 2.0.x and 1.9.x user
rubies

55. 8.- Developers isolated ruby environment(III)
Explicit ruby version (1.9.3 or 2.0.X) in
appropriate virtualhost apache files (from
templates)
NOTE: Foodcritic tool helped a lot when looking
for errors (e.g. specially with chef templates)

56. 9. Final Steps (I)
After just 2 weeks, it almost worked ok.
Traditional approach to development: logrotate
and monit forked recipes with “999” suffix
added to “semver” in metadata.rb
● not perfect
● Explicit installation of dependencies when in
Opensource Opscode server

57. 9. Final Steps (II)
Pareto principle! 80/20 Rule.
● Two more weeks with “fringes”
● create new organization in opscode free
account
● chef-solo-search for “local” databags
● BIG CRISIS: no chef-server available ->
chef-solo deployment

58. Angry chef?

59. 9. Final Steps (III)
Create git repo (first commit in 2 weeks)
● install explicit dependencies
● “chef cookbook site install”
● only needed “ancient” (fork and modify)
approach in two cookbooks: monit and
passenger

60. 9. Final Steps (IV)
“In extremis”: Refactor the “monolithic”
cookbook
Monolithic cookbook refactored as 3 cookbooks
Three layers of attribute+template files:
● general cookbook (i.e. language, servers,...)
○ company cookbook (i.e. final customer specific)
■ project cookbook (i.e. virtualhost config)

61. 9. Final Steps (V)
● only two “old fashioned” forked recipes:
monit and passenger
● new recipes forces a different way to use
databags

62. 10. Conclusions (I)
To FIX:
● move logrotate recipes in railssites2 to
another new bootstrapping-node-sysadmin
cookbook
● not exactly applied but inspired in
"everything is a cookbook"

63. 10. Conclusions (II)
● You need to have “Luck”
○ no way to configure staging chef-server (only
working through 127.0.0.1 and VPN)
○ no way to configure ubuntu from opscode deb
packages
● You need big motivation to success

64. 10. Conclusions (III)
To be improved:
● ubuntu/RHEL cookbooks
● tests
● mysql m/s cookbook
● monit recipe
● opensource chef-server hosting

66. THANKS FOR YOUR
PATIENCE