Here are the slides from Rob Nelson's PuppetConf 2016 presentation called Enjoying the Journey from Puppet 3.x to 4.x. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

2. Enjoying the Journey from
Puppet 3.x to Puppet 4.x
Rob Nelson

3. Who Am I?
Puppet user since 2014 (3.3 era)
Vox Pupuli, puppet-lint contributor
@rnelson0, https://rnelson0.com/

4. Agenda
• Why upgrade?
• Refactor our codebase for Puppet 4
• Upgrade our Puppet master(s) and agents to Puppet 4.x
• Refactor our codebase to remove Puppet 2- and 3-isms
• Tips, tricks, and tools
• Enjoying ourselves

5. Why?
• Puppet 4 is old! First released March, 2015
• Puppet 3 is really old! End Of Support on December 31, 2016
• Puppet 4 only modules
• Puppet 4 language improvements
• Application Orchestration
• PE first, FOSS eventually; some free implementations (such as choria) appearing
• AIO Puppet and Puppetserver
• Better performance, security; same agent/puppetserver between FOSS and PE
• Puppet 5 is coming!

6. Who does this apply to?
• Puppet Enterprise users
• Puppet Opensource users
• Foreman (1.13+) users
• Master and Masterless

7. Blueprint
• Start with Puppet 3.x
• Read the release notes
• Plan the roadmap
• Validate / create tests
• Refactor until your current version passes all tests
• Upgrade/Replace the master(s)
• Upgrade the agents
• Repeat the Refactor / Upgrade steps until you get to 4.latest

8. Release Notes
• All of them - not just the latest version
• Identify potential issues, deprecated features, etc
• Determine the minimum version required to upgrade to target version
• Stay up to date

9. Define the Roadmap
• Determine the current version
• In-place upgrades or new infrastructure?
• Identify intermediate version steps
• Enable Future Parser [and Strict Variables] before you hit 4.x
• PE: Requires intermediate upgrades or fresh installs (check KB)
• FOSS: Go straight to 4.latest
• Determine how upgrades and interruptions affect ecosystem products – PE
Console/puppetboard, SEIMs, monitoring, etc.

10. FOSS Example Roadmap
• 3.6.0 -> 3.8.7
• 3.8.7 w/Future Parser [and Strict Variables]
• 3.8.7 -> 4.7.0
• Today’s example roadmap

11. PE Example Roadmap
• 3.7.2 -> 3.8.6
• 3.8.6 w/Future Parser [and Strict Variables]
• 3.8.6 -> 2015.3.3
• 2015.3.3 -> 2016.2.1

12. Validate/Create Tests
• Tests assure (mostly) predictable behavior
• Determine what kinds of tests you need - unit, acceptance, integration, other?
• Good testing setup in puppet-module-skeleton
• Never written an rspec-puppet test? puppet-retrospec generates naive tests
that need tuned
• Existing tests must pass before modifying code
• Turn on Future Parser [and Strict Variables] only at 3.8.x
• Beyond tests: catalog diffs, personalized tests

13. Rspec Tests
$ cat spec/classes/apache_spec.rb
require 'spec_helper'
describe 'profile::apache', :type => :class do
let :facts do
{
facts_hash
}
end
context 'with defaults for all parameters' do
it { is_expected.to create_class('profile::apache') }
it { is_expected.to contain_package('httpd') }
it { is_expected.to contain_user("apache") }
end
end

14. Rspec Run
[rnelson0@build03 profile:production]$ bundle exec rspec spec/classes/
apache_spec.rb
profile::apache
with defaults for all parameters
should contain Class[profile::apache]
should contain Package[httpd]
should contain User[apache]
Finished in 7.82 seconds (files took 2.49 seconds to load)
3 examples, 0 failures

15. Refactor
• Create a new branch for the target version, e.g. 3.8.7
• Test against current and target versions, e.g. ~>3.6.0 and ~>3.8.0, with and
without Future Parser/Strict Variables
• Identify failing tests, refactor to fix
• Upgrade modules as early as possible. Be aware of the required Puppet
version for a module version, and look out for defunct or migrated modules,
such as those transferred to Vox Pupuli
• Move forward when tests are green for current AND next version

16. Testing with particular Puppet versions
$ grep PUPPET Gemfile
gem "puppet", ENV['PUPPET_GEM_VERSION'] || '~> 4.0'
[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='~>3.8.0'
[rnelson0@build controlrepo]$ bundle update
Installing puppet 3.8.7 (was 4.6.0)
[rnelson0@build controlrepo]$ bundle exec puppet --version
3.8.7
[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='3.8.1'
[rnelson0@build controlrepo]$ bundle update
Installing puppet 3.8.1 (was 3.8.7)
[rnelson0@build controlrepo]$ bundle exec puppet --version
3.8.1

17. High level Master(s) upgrade process
• Prep for new master/in-place upgrade
• Deploy new/upgrade in testing
• Revert
• Deploy new/upgrade in production
• Start with Master of Masters or other “parent” nodes first
• Update separate PuppetDB node, puppetdb-termini on masters

18. Replace the Master
• Prepare a new operational environment
• Do not serve bad/incorrect catalogs to existing nodes
• Deploy a new master on the target puppet version
• Bootstrap configuration/code
• Test the master against itself, puppet agent -t
• Deploy and test canary nodes in the same operational environment

19. In-place Master upgrade
• Snapshot (or equivalent) the master(s) and canary nodes
• Restrict access to the master:
• Control access with firewall/load balancer
• Disable puppet agent on nodes with orchestration
• Revoke certificates for non-canary nodes
• Revoke the CA, generate a new CA and new agent certs for canary nodes only
• Upgrade the master
• Test the master then canary nodes with puppet agent -t

20. Troubleshooting
• Collect logs from the master and canaries
• Look for changed resources, number of resources in catalog
• Revert production environment
• Analyze cause(s)
• Refactor code and data to address issues
• Try again
• Learn from failures, prevent them in the future

21. Upgrade the Agents
• Can often skip on PATCH versions and some MINOR versions (see rel notes)
• puppetlabs/puppet_agent (requirements) updates agents on next check-in
• Orchestration
• Replace nodes with new instances running the new agent
• By hand

22. Repeat
• Relax, enjoy the success of an upgrade!
• Start working on the next version/feature flags
• Repeat the Refactor / Upgrade steps

23. Keeping Up

24. Keeping Up
Refactor to take advantage of Puppet 4 language improvements, new tools (ex:
r10k -> PE Code Manager), new file locations, etc.
• PE has quarterly upgrades, FOSS more frequent
• The less frequently you do something, the more painful it is. “Upgrade early
and upgrade often!”
• Try not to get more than 2 MINORs behind
• Test against puppet version ~>4.0 (latest v4) and run bundle update before
manual tests

25. Puppet 4 Language Improvements
• Replace create_resources() with iteration
• Replace validate_*() with data types (including a Sensitive type)
• There is a validate_legacy() helper function available in puppetlabs/stdlib to
assist with replacing validate_*() functions (blog)
• Simplified resource wrappers with * and + operators
• Improved default attributes are per-expression
• New template type EPP is available
• Puppet Lookup, Data In Modules, and other hiera improvements
• Use $facts[] instead of global variables to tidy up the namespace and remove
ambiguity

26. Tips & Tricks – Puppet Enterprise
• PE includes support, use it for planning/errors
• Puppet Enterprise Upgrade Service to engage Pro Services
• PE Classifier changes over time. Review
Preconfigured Node Groups documentation
• pe_puppetserver_gem is out, puppetserver_gem is in
• Do not use PE’s bundled Ruby for other Ruby tasks, conflicts between
bundled/downloaded gems. Recommend rbenv/rvm or SCL-equiv instead
• Do not ever do this on your master. EVER!

27. Tips & Tricks - Strings
Understand how string conversion works in puppet, hiera, rspec-puppet, and how it
has changed:
• rspec-puppet: 'undef' represents an undefined value
• Puppet DSL: it is the string undef! Try :undef, without quotes, instead
• If you have a file resource with a title or path of ${undefvar}/${populatedvar},
rspec will start failing because file { 'undef/etc/app.conf' :} is not valid
• Similar issue with 'true' vs true and 'false' vs false
• Other common issues: input from hiera/ENC, quoted numbers as strings, stringify
vs structured facts, unquoted strings in case selectors, etc
• May require acceptance tests/canary nodes to become apparent

28. Tips & Tricks - Hiera
• Hiera eyaml gem is lost during the upgrade to the 4.x puppetserver
• Enable the yaml backend and ensure that the master does not rely on eyaml data
• Run the agent on the master to redeploy the gem (with puppet/hiera or similar)
before agents check in
• %{}: used to prevent variable interpolation, as in %%{}{environment} to generate
the string %{environment}. In 3.x and in 4.5 resolves to an empty string, in 4.0-4.4
it returned the scope, giving strings like %<#Hiera:7329A802#>{environment}. Use
%{::} instead, as in %%{::}{environment}. Affects PE < 2016.2.0
• datadir: some versions expect :: prepends to variables and others do not.
Change %{environment} to %{::environment}. Likely PE < 2016.2.0 as well

29. Tips & Tricks - Other
• Review modules and their supported versions. May be incorrect or weak
assumptions (>= 3 but should also include < 4 – check tests)
• Upgrades across major versions mean additional troubleshooting
• Upgrade early – but with caveats
• Many tools to assist with automating version upgrades in your Puppetfile
• ERB scope: prepend most variables with @ (<%= var %> to <%= @var %>)
• Script to detect usage of hardcoded /etc/puppet paths, no longer correct in v4
• External fact weighting bug: FACT-1413
• Minimize coupled/entangled changes
• Ask for help! Colleagues, social media, etc.

30. Tools
• Puppet Community Slack / IRC and Mailing Lists
• puppet-ghostbuster helps you find "dead code" that you may want to prune
before you start on your refactoring journey.
• rspec-puppet, puppetlabs_spec_helper, and puppet-lint are improving their
Puppet 4 support
• A number of catalog diff tools exist (diff generators and a viewer) to inspect
the actual catalog differences from active nodes across different versions of
Puppet.

31. Links
Additional information on Puppet 4 and Migrations
• Official Puppet Upgrade Docs
• Whirlwind Tour of Puppet 4 by R.I. Pienaar
• The Power of Puppet 4 by Martin Alfke
• Puppet - our journey from Puppet 3.8 to Puppet 4 by Jonas Genannt

32. Summary
• Plan the upgrade blueprint
• Have working tests before upgrading
• Step through the new versions / feature flags
• Refactor code to take advantage of the language and tool improvements
• Keep mowing
• Enjoy the journey!