Here are the slides from Rob Nelson's PuppetConf 2016 presentation called Enjoying the Journey from Puppet 3.x to 4.x. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
3. Who Am I?
Puppet user since 2014 (3.3 era)
Vox Pupuli, puppet-lint contributor
@rnelson0, https://rnelson0.com/
4. Agenda
• Why upgrade?
• Refactor our codebase for Puppet 4
• Upgrade our Puppet master(s) and agents to Puppet 4.x
• Refactor our codebase to remove Puppet 2- and 3-isms
• Tips, tricks, and tools
• Enjoying ourselves
5. Why?
• Puppet 4 is old! First released March, 2015
• Puppet 3 is really old! End Of Support on December 31, 2016
• Puppet 4 only modules
• Puppet 4 language improvements
• Application Orchestration
• PE first, FOSS eventually; some free implementations (such as choria) appearing
• AIO Puppet and Puppetserver
• Better performance, security; same agent/puppetserver between FOSS and PE
• Puppet 5 is coming!
6. Who does this apply to?
• Puppet Enterprise users
• Puppet Opensource users
• Foreman (1.13+) users
• Master and Masterless
7. Blueprint
• Start with Puppet 3.x
• Read the release notes
• Plan the roadmap
• Validate / create tests
• Refactor until your current version passes all tests
• Upgrade/Replace the master(s)
• Upgrade the agents
• Repeat the Refactor / Upgrade steps until you get to 4.latest
8. Release Notes
• All of them - not just the latest version
• Identify potential issues, deprecated features, etc
• Determine the minimum version required to upgrade to target version
• Stay up to date
9. Define the Roadmap
• Determine the current version
• In-place upgrades or new infrastructure?
• Identify intermediate version steps
• Enable Future Parser [and Strict Variables] before you hit 4.x
• PE: Requires intermediate upgrades or fresh installs (check KB)
• FOSS: Go straight to 4.latest
• Determine how upgrades and interruptions affect ecosystem products – PE
Console/puppetboard, SEIMs, monitoring, etc.
12. Validate/Create Tests
• Tests assure (mostly) predictable behavior
• Determine what kinds of tests you need - unit, acceptance, integration, other?
• Good testing setup in puppet-module-skeleton
• Never written an rspec-puppet test? puppet-retrospec generates naive tests
that need tuned
• Existing tests must pass before modifying code
• Turn on Future Parser [and Strict Variables] only at 3.8.x
• Beyond tests: catalog diffs, personalized tests
13. Rspec Tests
$ cat spec/classes/apache_spec.rb
require 'spec_helper'
describe 'profile::apache', :type => :class do
let :facts do
{
facts_hash
}
end
context 'with defaults for all parameters' do
it { is_expected.to create_class('profile::apache') }
it { is_expected.to contain_package('httpd') }
it { is_expected.to contain_user("apache") }
end
end
14. Rspec Run
[rnelson0@build03 profile:production]$ bundle exec rspec spec/classes/
apache_spec.rb
profile::apache
with defaults for all parameters
should contain Class[profile::apache]
should contain Package[httpd]
should contain User[apache]
Finished in 7.82 seconds (files took 2.49 seconds to load)
3 examples, 0 failures
15. Refactor
• Create a new branch for the target version, e.g. 3.8.7
• Test against current and target versions, e.g. ~>3.6.0 and ~>3.8.0, with and
without Future Parser/Strict Variables
• Identify failing tests, refactor to fix
• Upgrade modules as early as possible. Be aware of the required Puppet
version for a module version, and look out for defunct or migrated modules,
such as those transferred to Vox Pupuli
• Move forward when tests are green for current AND next version
17. High level Master(s) upgrade process
• Prep for new master/in-place upgrade
• Deploy new/upgrade in testing
• Revert
• Deploy new/upgrade in production
• Start with Master of Masters or other “parent” nodes first
• Update separate PuppetDB node, puppetdb-termini on masters
18. Replace the Master
• Prepare a new operational environment
• Do not serve bad/incorrect catalogs to existing nodes
• Deploy a new master on the target puppet version
• Bootstrap configuration/code
• Test the master against itself, puppet agent -t
• Deploy and test canary nodes in the same operational environment
19. In-place Master upgrade
• Snapshot (or equivalent) the master(s) and canary nodes
• Restrict access to the master:
• Control access with firewall/load balancer
• Disable puppet agent on nodes with orchestration
• Revoke certificates for non-canary nodes
• Revoke the CA, generate a new CA and new agent certs for canary nodes only
• Upgrade the master
• Test the master then canary nodes with puppet agent -t
20. Troubleshooting
• Collect logs from the master and canaries
• Look for changed resources, number of resources in catalog
• Revert production environment
• Analyze cause(s)
• Refactor code and data to address issues
• Try again
• Learn from failures, prevent them in the future
21. Upgrade the Agents
• Can often skip on PATCH versions and some MINOR versions (see rel notes)
• puppetlabs/puppet_agent (requirements) updates agents on next check-in
• Orchestration
• Replace nodes with new instances running the new agent
• By hand
22. Repeat
• Relax, enjoy the success of an upgrade!
• Start working on the next version/feature flags
• Repeat the Refactor / Upgrade steps
24. Keeping Up
Refactor to take advantage of Puppet 4 language improvements, new tools (ex:
r10k -> PE Code Manager), new file locations, etc.
• PE has quarterly upgrades, FOSS more frequent
• The less frequently you do something, the more painful it is. “Upgrade early
and upgrade often!”
• Try not to get more than 2 MINORs behind
• Test against puppet version ~>4.0 (latest v4) and run bundle update before
manual tests
25. Puppet 4 Language Improvements
• Replace create_resources() with iteration
• Replace validate_*() with data types (including a Sensitive type)
• There is a validate_legacy() helper function available in puppetlabs/stdlib to
assist with replacing validate_*() functions (blog)
• Simplified resource wrappers with * and + operators
• Improved default attributes are per-expression
• New template type EPP is available
• Puppet Lookup, Data In Modules, and other hiera improvements
• Use $facts[] instead of global variables to tidy up the namespace and remove
ambiguity
26. Tips & Tricks – Puppet Enterprise
• PE includes support, use it for planning/errors
• Puppet Enterprise Upgrade Service to engage Pro Services
• PE Classifier changes over time. Review
Preconfigured Node Groups documentation
• pe_puppetserver_gem is out, puppetserver_gem is in
• Do not use PE’s bundled Ruby for other Ruby tasks, conflicts between
bundled/downloaded gems. Recommend rbenv/rvm or SCL-equiv instead
• Do not ever do this on your master. EVER!
27. Tips & Tricks - Strings
Understand how string conversion works in puppet, hiera, rspec-puppet, and how it
has changed:
• rspec-puppet: 'undef' represents an undefined value
• Puppet DSL: it is the string undef! Try :undef, without quotes, instead
• If you have a file resource with a title or path of ${undefvar}/${populatedvar},
rspec will start failing because file { 'undef/etc/app.conf' :} is not valid
• Similar issue with 'true' vs true and 'false' vs false
• Other common issues: input from hiera/ENC, quoted numbers as strings, stringify
vs structured facts, unquoted strings in case selectors, etc
• May require acceptance tests/canary nodes to become apparent
28. Tips & Tricks - Hiera
• Hiera eyaml gem is lost during the upgrade to the 4.x puppetserver
• Enable the yaml backend and ensure that the master does not rely on eyaml data
• Run the agent on the master to redeploy the gem (with puppet/hiera or similar)
before agents check in
• %{}: used to prevent variable interpolation, as in %%{}{environment} to generate
the string %{environment}. In 3.x and in 4.5 resolves to an empty string, in 4.0-4.4
it returned the scope, giving strings like %<#Hiera:7329A802#>{environment}. Use
%{::} instead, as in %%{::}{environment}. Affects PE < 2016.2.0
• datadir: some versions expect :: prepends to variables and others do not.
Change %{environment} to %{::environment}. Likely PE < 2016.2.0 as well
29. Tips & Tricks - Other
• Review modules and their supported versions. May be incorrect or weak
assumptions (>= 3 but should also include < 4 – check tests)
• Upgrades across major versions mean additional troubleshooting
• Upgrade early – but with caveats
• Many tools to assist with automating version upgrades in your Puppetfile
• ERB scope: prepend most variables with @ (<%= var %> to <%= @var %>)
• Script to detect usage of hardcoded /etc/puppet paths, no longer correct in v4
• External fact weighting bug: FACT-1413
• Minimize coupled/entangled changes
• Ask for help! Colleagues, social media, etc.
30. Tools
• Puppet Community Slack / IRC and Mailing Lists
• puppet-ghostbuster helps you find "dead code" that you may want to prune
before you start on your refactoring journey.
• rspec-puppet, puppetlabs_spec_helper, and puppet-lint are improving their
Puppet 4 support
• A number of catalog diff tools exist (diff generators and a viewer) to inspect
the actual catalog differences from active nodes across different versions of
Puppet.
31. Links
Additional information on Puppet 4 and Migrations
• Official Puppet Upgrade Docs
• Whirlwind Tour of Puppet 4 by R.I. Pienaar
• The Power of Puppet 4 by Martin Alfke
• Puppet - our journey from Puppet 3.8 to Puppet 4 by Jonas Genannt
32. Summary
• Plan the upgrade blueprint
• Have working tests before upgrading
• Step through the new versions / feature flags
• Refactor code to take advantage of the language and tool improvements
• Keep mowing
• Enjoy the journey!