"Streamlining Your Puppet Development Workflow" given at Puppet Camp New York by Tom Doran, Yelp

1. Streamlining your puppet development
workflow
Tomas
Doran
@bobtfish
2014-­‐07-­‐07

2. Why invest in your workflow?
2

3. More time
3

4. To concentrate on
4

5. The important things in life
5

6. Like dwarf fortress!
6

7. Why invest in your workflow?
• Productivity!
• Work smarter, not harder
• You spend a lot of time writing/testing/debugging code
• Optimizing that is worthwhile
• Faster development cycle is more productive
• 5% faster cycle
• 5% more time for testing
• Less bugs
7

8. Revision control
• You must have your code in revision control
• git is preferred
• fast + cheap branches
• everyone else uses it
• github
• gitolite
!
• svn is also ‘workable’ 8

9. How do you run puppet?
• I like cron (daemon also fine)
• Two possible approaches
• —noop mode automatically + manual apply
• Automatic apply
9

10. How do you run puppet?
• I like cron (daemon also fine)
• Two possible approaches
• —noop mode automatically + manual apply
• Automatic apply
10

11. How do you run puppet?
• I like cron (daemon also fine)
• Two possible approaches
• —noop mode automatically + manual apply
• Automatic apply
• I recommend automatic apply
• Scary (don’t push to master unless you’re
confident!)
• puppet agent —disable (monitor this!)
• Testing workflow
• Eventual consistency
11

12. Move just fast enough to not break everything
• Test so that you’re confident
• Branch for every significant change
• Reduce batch size
• Small scary change easier to test
• Easier to roll back
• Otherwise - applying months of changes at once
• Really scary!
• Don’t even know desired effects!
• Communicate!
12

13. Standalone mode
puppet apply --modulepath
modules/:vendor/modules/ --show_diff --
hiera_config=./standalone-hiera.yaml $@
manifests/site-standalone.pp
!
13

14. —noop
• Use —noop mode for testing!
ssh -A "$HOST" -- "sh -c 'cd $DESTDIR/
$PUPPET_DIRNAME; ./tools/puppet-standalone
--verbose --show_diff —noop'"
• tools/what-would-happen-on
14

15. Dynamic environments
git branch => puppet environment
puppet agent -t —environment my_test_branch
!
• puppet >= 2.7 has environment support
• Use puppetupdate or r10k to push branches
(Links at the end!)
15

16. 16

17. Reporting
• Need to know what puppet did
• Puppet has logs + reporting functionality
• Push reports to:
• irc
• email (eww!)
• elasticsearch
• mysql
• puppetdb
• Saves compiled catalogs to disk
tools/what-just-happened-on
17

18. Foreman
18

19. Norman
19

20. Puppet enterprise
20

21. puppet-syntax
• Ruby gem
• Trivial to add to your project
• Checks .pp, .erb, .yaml
• Fast enough to run pre-commit
echo ‘bundle exec rake syntax’
>.git/hooks/pre-commit
chmod 755 .git/hooks/pre-commit
21

22. r10k/librarian - Puppetfile
• Awesome module deployment - with robots!
• Easy vendor/modules directory for modules from the forge
• Makes module = git repository pattern easier
• Not every module from the forge is useable immediately
• Fork on github (and make your changes open source)?
• Pull request and get them back upstream!
• Fork into internal git and modify.
• gitolite mirrors
• Improve performance
• No external dependencies
• Private forge (puppet-library)
22

23. Module template
• ‘puppet module generate’ uses a template
• Start from the GDS example one:
github.com/gds-operations/puppet-module-skeleton
• Modify to your taste!
23

24. ‘Real’ testing
• No hard rules.
• Invest to the level that’s right for you!
• Dev heavy teams
• Know about unit testing!
• Sysadmin heavy teams
• Less enthusiastic
• Do what provides value!
24

25. Feedback!
• Tighten your OODA loop!
• Don’t care how!
• N.B. Automated tests don’t work unless they’re
automated.
• I.E. MUST run on commit
• Whatever’s effective for your org
25

26. rspec-puppet
• Unit testing
• At least write a compile test for your code!
• Put it in your module template.
• Explicit dependencies FTW
!
• Use puppetlabs-spec-helper
• Inject mocks into spec/fixtures/manifests/site.pp
echo ‘define my::complex::dependency ($foo,
$bar) {}’ >> spec/fixtures/manifests/site.pp
26

27. serverspec
• Spin up Vagrant VM and apply your code
• Check properties of:
• files
• ports
• services
• Acceptance testing
• Slower and heavier weight than unit tests
• Can be highly valuable!
27

28. Jenkins - simple
• Put your tests together so that they can be run as one job
!
rake test
task :test =>
[:syntax, :spec, :integration]
• Get Jenkins to run it on commit to master
• git polling
• + add a post-receive hook to curl Jenkins
• Shout in email + irc!
28

29. Jenkins - less simple
• If branches are cheap (i.e. git!)
• Encourages people to push branches
• Code review++
• Adhoc
• Or pick your poison
• Run syntax checks and unit and/or integration tests on
every branch.
• Report back to committer
• irc notification
• write in code review
29

30. Jenkins integration branches
• Jenkins can merge branches!
• Push a branch
• Jenkins picks it up
• Merges with master
• Runs tests
• If they pass, pushes results
30

31. Jenkins integration branches
• Jenkins can merge branches!
• Push a branch
• Jenkins picks it up
• Merges with master
• Runs tests
• If they pass, pushes results
31

32. Do everything in irc
32

33. True believers
33

34. 34

35. Policy vs Automation
• Testing is awesome
• Code review is awesome
• ‘Process is the scar tissue from previous problems’
• Empower people to change the process!
• If you make the tools simple to use…
• People will use them!
• Make doing the right thing a no-brainer
35

36. We’re hiring! NY, SF, Dublin, London!
!
• https://slideshare.net/bobtfish/
• http://puppetlabs.com/blog/git-workflow-and-puppet-environments
• http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
• http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/
• https://github.com/youdevise/puppetupdate
• https://github.com/adrienthebo/r10k
• http://gitolite.com/gitolite/index.html
• http://puppetlabs.com/puppet/puppet-enterprise/
• http://docs.puppetlabs.com/guides/reporting.html
• http://theforeman.org/
• https://github.com/youdevise/norman/
• https://github.com/youdevise/puppet-logstash-reporter/
• https://github.com/gds-operations/puppet-module-skeleton
• https://github.com/drrb/puppet-library 36