This document discusses testing Puppet code. It begins by explaining why testing Puppet is important due to the large amount of Puppet code and its use across many platforms. It then provides an overview of existing Puppet testing tools like puppet-lint, rspec-puppet, and beaker-rspec. It notes different types of Puppet users have varying needs and experiences with testing. Key challenges discussed are improving the getting started experience, adopting acceptance testing more widely, providing consistent interfaces, and addressing the Ruby dependency of many tools. The document concludes that the community has powerful tools but needs more consistent user experiences and workflows to help beginners test Puppet code.

1. (without introducing more risk)
The Future of Testing
Puppet Code
Puppet
Gareth Rushgrove
Who, What, Why, How

2. (without introducing more risk)
@garethr

3. (without introducing more risk)
Gareth Rushgrove

4. (without introducing more risk)
This Talk
What we’ll cover

5. - Why test Puppet anyway
- Existing Puppet testing tools
- Different users and usecases
- Future challenges
Gareth Rushgrove

6. Mainly a talk about problems,
rather than solutions
Gareth Rushgrove

7. Not a talk about how to test
Puppet code. Lot of existing
content on that subject
Gareth Rushgrove

8. (without introducing more risk)
Gareth Rushgrove
PuppetConf 2014

9. (without introducing more risk)
Gareth Rushgrove
Survey of 300 Puppet users

10. (without introducing more risk)
Gareth Rushgrove

11. (without introducing more risk)
Why Test Puppet
And a little history

12. Gareth Rushgrove

13. - You have lots of Puppet code
- You’re targeting a wide range of OS’s
- The Puppet language has more features
Gareth Rushgrove

14. (without introducing more risk)
Gareth Rushgrove
Lots of code
But how much is lots

15. Note that this just represents public
code on GitHub. Lots of private Puppet
code out there too
Gareth Rushgrove

16. (without introducing more risk)
Gareth Rushgrove
How many .pp files on GitHub?
SELECT COUNT(*)
FROM [puppet.puppet_files]
-- 489725

17. (without introducing more risk)
Gareth Rushgrove
How many lines of Puppet code?
SELECT
COUNT(line) total_lines
FROM (
SELECT SPLIT(content, 'n') AS line
FROM [puppet.puppet_content]
)
-- 6697099

18. (without introducing more risk)
Gareth Rushgrove
How big are .pp files?
SELECT QUANTILES(total_lines, 10) AS q
FROM (
SELECT
COUNT(line) total_lines
FROM (
SELECT SPLIT(content, 'n') AS line, id
FROM [puppet.puppet_content]
)
GROUP BY id
)

19. 50% of Puppet files are 20 lines or less.
90% of Puppet files are 100 lines or less
Gareth Rushgrove

20. The largest Puppet file public on GitHub
has 25,238 lines
Gareth Rushgrove

21. (without introducing more risk)
Gareth Rushgrove
Mixed environments
So many platforms

22. Gareth Rushgrove

23. A third of respondents run more than
one OS on there development machine
Gareth Rushgrove

24. Gareth Rushgrove

25. 60% of respondents run more than one
OS in production
Gareth Rushgrove

26. (without introducing more risk)
Gareth Rushgrove
The Puppet
language
Types, lambdas, iterators, functions

27. Gareth Rushgrove

28. (without introducing more risk)
Existing Tools
Providing some context

29. Gareth Rushgrove

30. (without introducing more risk)
Core tools
Common starting points
Gareth Rushgrove

31. (without introducing more risk)
Gareth Rushgrove
puppet-lint

32. (without introducing more risk)
Gareth Rushgrove
Style conformance
$ puppet-lint /etc/puppet/modules
foo/manifests/bar.pp - ERROR: trailing whitespace found on li
apache/manifests/server.pp - WARNING: variable not enclosed i
...

33. (without introducing more risk)
Gareth Rushgrove
rspec-puppet

34. (without introducing more risk)
Gareth Rushgrove
Unit tests for Puppet
it { is_expected.to compile }
it { is_expected.to contain_class('db') }
it do
is_expected.to contain_package('mysql-server')
.with_ensure('present')
end
it { is_expected.to have_exec_resource_count(0) }

35. (without introducing more risk)
Gareth Rushgrove
puppet-syntax

36. (without introducing more risk)
Gareth Rushgrove
Popular Puppet gems
SELECT package, COUNT(*) count
FROM (
SELECT REGEXP_EXTRACT(line, r'gem ['|"](.*)['|"]') packag
FROM (
SELECT SPLIT(content, 'n') line, id
FROM [puppet.gemfile_contents]
WHERE content CONTAINS 'gem' )
GROUP BY package, id
)
GROUP BY 1
ORDER BY count DESC
LIMIT 30;

37. (without introducing more risk)
Gareth Rushgrove
Popular gem results
puppetlabs_spec_helper 1157
puppet 1142
rake 1135
puppet-lint 768
beaker-rspec 610
rspec-puppet 576
metadata-json-lint 566
beaker 440
puppet-blacksmith 425
serverspec 359

38. (without introducing more risk)
Gareth Rushgrove
puppetlabs_spec_helper

39. (without introducing more risk)
Gareth Rushgrove
Advanced tools
Next steps

40. (without introducing more risk)
Gareth Rushgrove
beaker-rspec

41. (without introducing more risk)
Gareth Rushgrove
Serverspec

42. (without introducing more risk)
Gareth Rushgrove
rubocop

43. (without introducing more risk)
Gareth Rushgrove
Other tools
Alternative approaches

44. (without introducing more risk)
Gareth Rushgrove
puppet-catalog-test

45. (without introducing more risk)
Gareth Rushgrove
Test catalogue compilation
$ puppet-catalog-test -m modules -M site.pp
[INFO] Using puppet 3.0.2
[PASSED] foo (compile time: 0.168182 seconds)
[PASSED] default (compile time: 0.003451 seconds)
----------------------------------------
Compiled 2 catalogs in 0.1717 seconds (avg: 0.0858 seconds)

46. (without introducing more risk)
Gareth Rushgrove
Test Kitchen

47. (without introducing more risk)
Different Users
And different usecases

48. Gareth Rushgrove

49. Half of respondents regularly write
tests for Puppet code
Gareth Rushgrove

50. A quarter of respondents never write
tests for Puppet code
Gareth Rushgrove

51. The problem with a tool-centric
world view
Gareth Rushgrove

52. Understanding user needs means
starting with users
Gareth Rushgrove

53. Gareth Rushgrove
- The Vox Pupuli folks
- Lots of Puppet employees
- You in-house Puppet expert
- Custom functions
The module authors

54. The regular Puppet users
Gareth Rushgrove
- Roles and profiles
- Using Puppet weekly or more
- PuppetDB, Puppet Server
- Custom facts

55. The occasional Puppet user
Gareth Rushgrove
- Using Puppet as needed
- Basic Puppet code
- Hieradata

56. Today different types of Puppet users
have a very different experience
Gareth Rushgrove

57. Gareth Rushgrove

58. For those with no Ruby experience the
average was 2 out of 5
Gareth Rushgrove

59. For those with a little Ruby experience
the average was still only 2.5 out of 5
Gareth Rushgrove

60. Of the people who love the testing
experience 33% are self-described
power users of both Puppet and Ruby
Gareth Rushgrove

61. Gareth Rushgrove

62. But more than 50% of respondents have
little or no Ruby experience
Gareth Rushgrove

63. (without introducing more risk)
Challenges
What we need for the future

64. (without introducing more risk)
Gareth Rushgrove
Improved getting
started experience
Helping a new generation of users

65. - Getting started testing documentation
- Single entry point for users
- Opinionated workflow
Gareth Rushgrove

66. (without introducing more risk)
Gareth Rushgrove
Adopting
acceptance testing
Still too hard

67. 27 of the 67 Serverspec users are not
using either Beaker or Test Kitchen.
Gareth Rushgrove

68. Only 5 of the Beaker (without rspec)
users are not also using beaker-rspec
Gareth Rushgrove

69. 3 of the Test Kitchen users are also
using some flavour of Beaker
Gareth Rushgrove

70. So only 26% to 35% of respondents
are using an acceptance testing tool
with there Puppet code
Gareth Rushgrove

71. (without introducing more risk)
Gareth Rushgrove
Experience matters
Consistence interfaces

72. (without introducing more risk)
Gareth Rushgrove
Inconsistencies
$ puppet-lint vs rake lint vs rake validate
$ rake syntax vs puppet parser validate
$ rspec vs rake spec
$ BEAKER_destroy=no rake beaker vs beaker

73. We have a language style guide. What
about other Puppet user interfaces?
Gareth Rushgrove

74. (without introducing more risk)
Gareth Rushgrove
Ruby heritage
Ruby as a development environment

75. 60% of respondents described
themselves as Puppet power users
Gareth Rushgrove

76. Only 12% of respondents described
themselves as Ruby power users. A
third of those work for Puppet!
Gareth Rushgrove

77. - More help with using Ruby for testing?
- Support for testing in other languages?
- Native Puppet support for testing?
- Containerised testing abstraction?
Gareth Rushgrove

78. (without introducing more risk)
Gareth Rushgrove
jordan/spec

79. (without introducing more risk)
Gareth Rushgrove
Test Puppet with Puppet
file { '/tmp/test':
ensure => present,
content => 'The file content',
}
assertion { 'that the file has the correct contents':
subject => File['/tmp/test'],
attribute => 'content',
expectation => 'The file content',
}

80. (without introducing more risk)
Conclusions
Where to go from here

81. Thanks to the community we have a
collection of powerful testing tools in the
Puppet ecosystem
Gareth Rushgrove

82. But we don’t have consistent user
interfaces or the opinionated workflows
useful for beginners
Gareth Rushgrove

83. Testing shouldn’t just be a
power user feature
Gareth Rushgrove

84. As a community we should talk
about design more often
Gareth Rushgrove

85. (without introducing more risk)
Gareth Rushgrove
https://goo.gl/forms/nchOp5hYdLLgmJ8z1

86. Gareth Rushgrove

87. (without introducing more risk)
Questions?
And thanks for listening