Presenters: Fabio Lessa & Andrew Myers

1. Sock Puppets: Growing your
puppet codebase
Fabio Lessa & Andrew Myers

3. Part 1

4. Empty server room

5. Project teams
teams

7. OPS team for app
support

8. The
codebase
we've
started with
Something that represents a mess

9. Standard
Puppet Master setup

10. Code was split by environments

12. Workflow:
1. Push change
2. Wait for it to propagate to that
environment
3. Check it worked
If you need that applied to another
environment, go to 1

13. The real Workflow:
1. Push change
1.1. Stop the puppet daemon in all the critical nodes for that
environment
1.2. Run puppetd manually in one host
to ensure it works
1.3. Start the daemon again (on all the hosts you remember)
2. Wait for it to propagate to that environment
3. Check it worked (meh)
If you need that applied to another environment, go to 1 (if you
remember, that is)

15. Production
UAT
System Testing
CI

16. Making changes
was painful

17. Part 2

19. "don't duplicate your manifests in your tests" Focus
on the catalog problems he described earlier and
test your logic. Don't test if puppet is doing it's job,
test that your logic it's doing it's job.
- Nikolay Sturm

20. The goal...
Before After

21. So how do we test/refactor

22. Compile Puppet Catalogues

23. Compare Versions

25. Application specific tests

26. Tying it all together

27. Part 3

28. Continuous Delivery

29. We were moving application deploys, why
not do the same with infrastructure?

30. MCollective

31. The new setup

32. Picture of the pipelines

33. info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/hostinfo/lib/facter/hostinfo.rb
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/concat/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for
Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/puppetmaster/manifests/repos.pp:36

34. info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: You cannot specify more than one of
content, source, target at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/sbs/manifests/init.pp:73

35. notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/
Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message:
current_value absent, should be Setting java the following java as default:
1.6.0.31 sun 1.6.0 (noop)
notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh'
from 1 events
notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered
'refresh' from 1 events
notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value
absent, should be directory (noop)
err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not
evaluate: Could not retrieve information from environment noop source(s)
puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/
share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/
manifests/init.pp:30
notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value
directory, should be absent (noop)
notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure:
current_value file, should be absent (noop)

36. We were moving application deploys, why
not do the same with infrastructure?

38. Our MCollective agent

39. The new action

40. At every push, all nodes connect to the noop
environment and run in noop mode

42. Picture of a report

43. Change
management
process
?

44. What changed?

45. What did you test?

46. Who will do it?

47. Part 4

48. New data centre is ready!

49. Should be easy to migrate....

50. Okay so our manifests are incomplete,
but we need to be able to rebuild machines...
Virtualisation to the rescue...

51. Magnets?

52. What happens

53. Wrap Up

54. 120 pushes to
production in the
last 8 months: 2
failures

56. MTTR FTW!
5 minutes!

57. Thank you!