"Testing for Ops: Going Beyond the Manifest" by Christopher Webber, Infrastructure Engineer, Demand Media. Presentation Overview: This talk aims to show the value of rspec-puppet for those who come from a more Ops-centric background. The focus will be on using tests to go beyond just rewriting manifests in rspec. Instead the focus will be on scenarios like: - Are the baseline security measures in place? - Do the differences between dev and prod get reflected? - Are the config elements that are core to the application present? In addition, tests will help to be a place to help document the oddities of our configurations and ensuring that minor changes don't result in catastrophe. Speaker Bio: After beginning his career at UC Riverside supporting enterprise operations and bioinformatics research, Chris is now rocking being an infrastructure engineer at Demand Media in Santa Monica. He currently supports large high-traffic sites like eHow.com, LiveSTRONG.com, and Cracked.com. Chris enjoys attending local meetups, writing new Puppet modules, and creating small tools to make his team's lives a little easier. Find him on Twitter as @cwebber.

1. TESTING FOR OPS:
GOING BEYOND THE MANIFEST
Christopher Webber (@cwebber)
Infrastructure Engineer, Demand Media

2. ABOUT ME
• Infrastructure Engineer at Demand Media
• Always been in Support or Operations Roles
• Been a Puppet user since v 0.24.7 (2008)

3. ROLE MODULES
VS
LIBRARY MODULES
Post By Craig Dunn
(http://www.craigdunn.org/2012/05/239/)

4. OPS IMPLICITLY
GETS TESTING

5. THEN WE TRY TO DO IT
• And it is dumb
• Might as well write a script to transform the code from one
format to another
• Start with unit testing frameworks, looking for integration
testing

6. TYPES OF TESTING
• Syntax checking (puppet parser validate)
• Linting
• Unit Tests
• Integration Tests

7. SETTING UP OUR
ENVIRONMENT
• Ruby
• Use the version you use in prod
• Gems
• puppet (use the version you use in prod)
• rspec-puppet
• puppet-lint
• puppetlabs_spec_helper

8. THE TRIFECTA
1 # == Class: ssh
2 #
3 class ssh {
4
5 package {
6 'openssh-server':
7 ensure => installed
8 }
9
10 file {
11 '/etc/ssh/sshd_config':
12 ensure => present,
13 owner => 'root',
14 group => 'root',
15 mode => '0644',
16 content => template('ssh/sshd_config.erb'),
17 require => Package['openssh-server']
18 }
19
20 service {
21 'sshd':
22 ensure => running,
23 enable => true,
24 hasstatus => true,
25 hasrestart => true,
26 subscribe => File['/etc/ssh/sshd_config']
27 }
28
29 }

9. THE TESTS
1 require 'spec_helper'
2
3 describe 'ssh' do
4
5 it { should contain_package('openssh-server') }
6
7 it do
8 should contain_file('/etc/ssh/sshd_config')
9 .with_ensure('present')
10 .with_owner('root')
11 .with_group('root')
12 .with_mode('0644')
13 .with_require('Package[openssh-server]')
14 end
15
16 it do
17 should contain_service('sshd')
18 .with_ensure('running')
19 .with_enable(true)
20 .with_hasstatus(true)
21 .with_hasrestart(true)
22 .with_subscribe('File[/etc/ssh/sshd_config]')
23 end
24 end

10. IS THIS USEFUL?
At first,
I said NO

11. BUT WHY?
It really doesn’t capture the things we care
about.

12. ENTER INFOSEC
• SSH MUST HAVE THE FOLLOWING SETTINGS:
• PermitRootLogin no
• X11Forwarding no

13. ADD USEFUL TESTS
25 context "InfoSec Requirements" do
26
27 it do
28 should contain_file('/etc/ssh/sshd_config')
29 .with_content(%r{^PermitRootLogin no$})
30 end
31
32 it do
33 should contain_file('/etc/ssh/sshd_config')
34 .with_content(%r{^X11Forwarding no$})
35 end
36
37 end

14. AND FAILFailures:
1) ssh InfoSec Requirements
Failure/Error: .with_content(%r{^PermitRootLogin no$})
expected that the catalogue would contain File[/etc/ssh/sshd_config] with content
matching `/^PermitRootLogin no$/` but its value of `"<file contents>"` does not
# ./spec/classes/ssh_spec.rb:29:in `block (3 levels) in <top (required)>'
2) ssh InfoSec Requirements
Failure/Error: .with_content(%r{^X11Forwarding no$})
expected that the catalogue would contain File[/etc/ssh/sshd_config] with content
matching `/^X11Forwarding no$/` but its value of `"<file contents>"` does not
# ./spec/classes/ssh_spec.rb:34:in `block (3 levels) in <top (required)>'
Finished in 0.47736 seconds
5 examples, 2 failures
Failed examples:
rspec ./spec/classes/ssh_spec.rb:27 # ssh InfoSec Requirements
rspec ./spec/classes/ssh_spec.rb:32 # ssh InfoSec Requirements

15. WHO CARES?
• Can now fix and validate
• Becomes one more check and balance
• Safety in changes

16. OPERATIONAL EXAMPLES

17. LIBRARY MODULE EXAMPLES
https://github.com/cwebberOps/puppetconf-ssh

18. CONTINUING WITH SSH
• New Requirements
• We have a system that other systems ssh to and drop info
• We have determined that we need to increase the
MaxStartups to 40

19. NEW TEST CODE
25 it do
26 should contain_file('/etc/ssh/sshd_config')
27 .with_content(%r{^MaxStartups 10$})
28 end
29
30 context "maxstartups => 40" do
31
32 let (:params) {{ :maxstartups => 40 }}
33
34 it do
35 should contain_file('/etc/ssh/sshd_config')
36 .with_content(%r{^MaxStartups 40$})
37 end
38
39 end
https://github.com/cwebberOps/puppetconf-ssh

20. NEW PUPPET CODE
1 # == Class: ssh
2 #
3 class ssh (
4 $maxstartups = 10
5 ){
https://github.com/cwebberOps/puppetconf-ssh

22. FEEDBACK LOOP
• Did you remember to update the template with your new
variable?
• Much faster than vagrant destroy && vagrant up or even
vagrant provision

23. MOVING ON TO THE ROLE
• In the end, it is the module that matters the most
• Should test that the config has the right config for the app
https://github.com/cwebberOps/puppetconf-infra

24. TEST CODE
1 require 'spec_helper'
2
3 describe 'infra::collector' do
4
5 it do
6 should contain_file('/etc/ssh/sshd_config')
7 .with_content(%r{^MaxStartups 40$})
8 end
9
10 end
https://github.com/cwebberOps/puppetconf-infra

25. PUPPET CODE
1 # Class for setting up the infrastructure collector
2 class infra::collector {
3
4 class {
5 'ssh':
6 maxstartups => 40
7 }
8
9 }
https://github.com/cwebberOps/puppetconf-infra

26. BUT THE TESTS...
THEY DON’T WORK

27. MORE SETUP
• Rakefile
• puppet_rspec_helper
• .fixtures.yml

28. RAKEFILE & SPEC_HELPER
1 require 'rake'
2
3 require 'rspec/core/rake_task'
4 require 'puppetlabs_spec_helper/rake_tasks'
New
1 require 'rake'
2
3 require 'rspec/core/rake_task'
4
5 RSpec::Core::RakeTask.new(:spec) do |t|
6 t.pattern = 'spec/*/*_spec.rb'
7 end
Old

29. .FIXTURES.YML
1 fixtures:
2 repositories:
3 ssh: "https://github.com/cwebberOps/puppetconf-ssh.git"
4 symlinks:
5 infra: "#{ source_dir }"

30. FUN EXAMPLES

31. INFRA::JENKINS
1 require 'spec_helper'
2
3 describe 'infra::jenkins' do
4
5 let (:facts) {{
6 :osfamily => 'RedHat',
7 :operatingsystem => 'CentOS'
8 }}
9
10 it { should include_class('java') }
11 it { should include_class('jenkins') }
12
13 end
https://github.com/cwebberOps/puppetconf-infra

32. INFRA::JENKINS
1 # Class for standing up a jenkins box
2 class infra::jenkins {
3
4 class {
5 'java':
6 } ->
7
8 class {
9 'jenkins':
10 }
11
12 }
https://github.com/cwebberOps/puppetconf-infra

33. WHAT’S BROKE?
http://projects.puppetlabs.com/issues/2053
The Bug
The Tweet

34. INFRA::NODEJS &
INFRA::NODEJS::DEV1 require 'spec_helper'
2
3 describe 'infra::nodejs' do
4
5 let (:facts) {{
6 :osfamily => 'RedHat',
7 :operatingsystem => 'CentOS'
8 }}
9
10 it { should include_class('nodejs') }
11
12 it { should_not contain_package('nodejs-dev') }
13
14 it { should contain_file('/app_specific_stuff') }
15
16 end
https://github.com/cwebberOps/puppetconf-infra

35. INFRA::NODEJS &
INFRA::NODEJS::DEV
1 require 'spec_helper'
2
3 describe 'infra::nodejs::dev' do
4
5 let (:facts) {{
6 :osfamily => 'RedHat',
7 :operatingsystem => 'CentOS'
8 }}
9
10 it { should include_class('nodejs') }
11 it { should contain_package('nodejs-dev') }
12
13 end
https://github.com/cwebberOps/puppetconf-infra

36. INFRA::NODEJS &
INFRA::NODEJS::DEV
1 # Demo class that sets up nodejs
2 class infra::nodejs {
3
4 class {
5 '::nodejs':
6 dev_package => false
7 }
8
9 file {
10 '/app_specific_stuff':
11 ensure => directory
12 }
13
14 }
https://github.com/cwebberOps/puppetconf-infra

37. INFRA::NODEJS &
INFRA::NODEJS::DEV
1 # Override class
2 class infra::nodejs::dev inherits infra::nodejs {
3
4 Class['::nodejs'] {
5 dev_package => true
6 }
7
8 }
https://github.com/cwebberOps/puppetconf-infra

38. WHAT’S BROKE?
http://projects.puppetlabs.com/issues/5517
The Bug

39. WHY DO THESE EXAMPLES
MATTER?

40. WHAT’S NEXT
• Integration testing using rspec-system
• Continuous Integration

41. PREVIEW

42. RSPEC-SYSTEM1 require 'spec_helper_system'
2
3 describe 'ssh' do
4
5 it 'class should converge' do
6 pp = <<-EOS
7 class { 'ssh': }
8 EOS
9
10 puppet_apply(pp) do |r|
11 r.exit_code.should_not == 1
12 r.refresh
13 r.exit_code.should be_zero
14 end
15 end
16
17 describe service('sshd') do
18 it { should be_enabled }
19 it { should be_running }
20 end
21 end

43. JENKINS

44. QUESTIONS?