PUPPET MODULES
FOR FUN AND PROFIT
Puppet Conf San Francisco 2012
Alessandro Franceschi
Lab42 / GrandSla

PUPPET @ Lab 42
2007 - Meet Puppet. Managed the Bank of Italy webfarm
2008 - First generation of Lab42 Puppet Modules
2009 - Multi OS support and standardization
2010 - A redesigned and coherent Example42 Module set
Puppet Modules Standards and Interoperability (PuppetCamp 2010 - Belgium)
Re-Use your Modules! (PuppetCamp 2010 - San Francisco)
2011 - Introducing Puppi
Puppi: Puppet strings to the shell (PuppetCamp Europe 2011 - Amsterdam)
2012 - Example42 Next Gen modules
- GrandSla: Puppet driven Infrastructure and Support
Developing IT Infrastructures with Puppet (CodeMotion 2012 - Rome)
A Holistic approach to Puppet modules (PuppetCamp Dublin and Geneva 2012)
“Job Driven” modules development

MODULES PATTERNS...
Data Separation
Configuration data is defined outside the module (or even Puppet manifests)
Module’s behavior is managed via APIs
Allow module’s extension and override via external data
Reusability
Customize behavior without changing module code
Do not force author’s idea on how configurations should be provided
Support different OS. Easily allow new additions
Standardization
Follow PuppetLabs style guidelines (puppet-lint)
Have coherent, predictable and intuitive interfaces
Provide contextual documentation (puppet-doc)
Interoperability
Limit cross-module dependencies
Allow easy modules’ cherry picking
Be self contained, do not interfere with other modules’ resources

... AND ANTI-PATTERNS
Data Mixed with Logic
Configuration data mixed inside the module’s logic
Module’s behavior defined in many different places
Module’s logic is rigid and can’t be defined externally
Works for me (here and now)
Module just works for the current setup
Can’t easily be re-used in other places for other projects
Works just for the currently used OSes
Code chaos
No layout rules, no standard style
Not standardized and predictable parameters
Who said “documentation”?
Interoperability
Who cares.
Basically whatever is quick and dirty... but is this really an anti-pattern?*
*(IMHO, yes)

Example42 modules: 10 design rules
Rule 1 - Provide alternatives for Data Separation
Rule 2 - Provide choice on Configuration Files supply
Rule 3 - Configure everything but provide OS defaults.
Rule 4 - Allow management of general module’s behavior
Rule 5 - Allow Custom Options for endless parameters
Rule 6 - Permit easy extension with custom classes
Rule 7 - Offer easy removal of the module’s resources
Rule 8 - Limit cross-dependencies. Prerequisites as options.
Rule 9 - Automatically monitor and firewall resources
Rule 10 - Puppi integration: Puppet knowledge to the CLI

Example42
DATA SEPARATION ALTERNATIVES
Rule #1
Set (Top Scope/External Node Classifier) variables and include classes:
$::openssh_template = 'site/openssh/openssh.conf.erb'
include openssh
Use Hiera:
hiera('openssh_template')
include openssh
Use Parametrized Classes:
class { 'openssh':
template => 'site/openssh/openssh.conf.erb',
}
Happily mix different patterns:
$::monitor = true
$::monitor_tool = [ 'nagios' , 'munin' , 'puppi' ]
class { 'openssh':
template => 'site/openssh/openssh.conf.erb',
}

Example42
PARAMS_LOOKUP EVERYWHERE
Rule #1
Each parameter is processed by the params_lookup function
class openssh (
[...] # openssh module specific parameters ...
$my_class = params_lookup( 'my_class' ),
$source = params_lookup( 'source' ),
$source_dir = params_lookup( 'source_dir' ),
$source_dir_purge = params_lookup( 'source_dir_purge' ),
$template = params_lookup( 'template' ),
$service_autorestart = params_lookup( 'service_autorestart' , 'global' ),
$options = params_lookup( 'options' ),
$version = params_lookup( 'version' ),
$absent = params_lookup( 'absent' ),
$disable = params_lookup( 'disable' ),
$disableboot = params_lookup( 'disableboot' ),
$monitor = params_lookup( 'monitor' , 'global' ),
$monitor_tool = params_lookup( 'monitor_tool' , 'global' ),
$monitor_target = params_lookup( 'monitor_target' , 'global' ),
[...] # Other common parameters
) inherits openssh::params {
[...]
}
Flexibility on booleans: they are sanitized by the any2bool function
You set:
$absent => “yes” # (or “1”, ‘Y’, “true”, true ...)
The module internally uses:
$bool_absent = any2bool($absent)

Example42
PARAMS LOOKUP ORDER
Rule #1
The function params_lookup is provided by the Puppi module
It allows data to be defined in different ways:
Via Hiera, if available
As Top Scope variable (as provided by External Node Classifiers)
Via defaults set in the module’s params class
The “global” argument is used to define site_wide behavior
# If there’s a direct param that’s the value
class { ‘openssh’:
monitor => true
}
# Otherwise, If Hiera is available:
hiera(“monitor”) # If global lookup is set
hiera(“openssh_monitor”) # A specific value overrides the global one
# If variable is still not evaluated, Top Scope is looked up:
$::monitor # If global lookup is set
$::openssh_monitor # If present, overrides $::monitor
# Module’s params are used as last option defaults:
$openssh::params::monitor

Example42
CUSTOMIZE: CONFIGURATION FILE
Rule #2
Provide Main Configuration as a static file ...
class { 'openssh':
source => 'puppet:///modules/site/ssh/sshd.conf'
}
... an array of files looked up on a first match logic ...
class { 'openssh':
source => [ "puppet:///modules/site/ssh/sshd.conf-${fqdn}",
"puppet:///modules/site/ssh/openssh.conf"],
}
... or an erb template:
class { 'openssh':
template => 'site/ssh/sshd.conf.erb',
}
Config File Path is defined in params.pp (can be overriden):
config_file => '/etc/ssh/sshd_config',

Example42
CUSTOMIZE: CONFIGURATION DIR
Rule #2
You can manage the whole Configuration Directory:
class { 'openssh':
source_dir => 'puppet:///modules/site/ssh/sshd/',
}
This copies all the files in lab42/files/ssh/sshd/* to local config_dir
You can purge any existing file on the destination config_dir which are
not present on the source_dir path:
class { 'openssh':
source_dir => 'puppet:///modules/site/ssh/sshd/',
source_dir_purge => true, # default is false
}
WARNING: Use with care
Config Dir Path is defined in params.pp (can be overriden):
config_dir => '/etc/ssh',

Example42
CUSTOMIZE: PATHS AND NAMES
Rule #3
Customize Application Parameters. An example:
Use the puppet module to manage pe-puppet!
class { 'puppet':
template => 'lab42/pe-puppet/puppet.conf.erb',
package => 'pe-puppet',
service => 'pe-puppet',
service_status => true,
config_file => '/etc/puppetlabs/puppet/puppet.conf',
config_file_owner => 'root',
config_file_group => 'root',
config_file_init => '/etc/sysconfig/pe-puppet',
process => 'ruby',
process_args => 'puppet',
process_user => 'root',
config_dir => '/etc/puppetlabs/puppet/',
pid_file => '/var/run/pe-puppet/agent.pid',
log_file => '/var/log/pe-puppet/puppet.log',
log_dir => '/var/log/pe-puppet',
}

Example42
DEFAULTS IN PARAMS.PP
Rule #3
Each module has a params class with defaults for different OS
class openssh::params {
### Application related parameters
$package = $::operatingsystem ? {
default => 'openssh-server',
}
$service = $::operatingsystem ? {
/(?i:Debian|Ubuntu|Mint)/ => 'ssh',
default => 'sshd',
}
$process = $::operatingsystem ? {
default => 'sshd',
}
[...]
$port = '22'
$protocol = 'tcp'
# General Settings
$my_class = ''
$source = ''
$source_dir = ''
$source_dir_purge = ''
[...]
### General module variables that can have a site or per module default
$monitor = false
$monitor_tool = ''
$monitor_target = $::ipaddress
$firewall = false
$firewall_tool = ''
$firewall_src = '0.0.0.0/0'
[...]

Example42
MANAGE BEHAVIOR
Rule #4
Enable Auditing:
class { 'openssh':
audit_only => true, # Default: false
}
No changes to configuration files are actually made and potential changes are
audited
Manage Service Autorestart:
class { 'openssh':
service_autorestart => false, # Default: true
}
No automatic service restart when a configuration file / dir changes
Manage Software Version:
class { 'foo':
version => '1.2.0', # Default: unset
}
Specify the package version you want to be installed.
Set => ‘latest’ to force installation of latest version

Example42
CUSTOM OPTIONS
Rule #5
With templates you can provide an hash of custom options:
class { 'openssh':
template => 'site/ssh/sshd.conf.erb',
options => {
'LogLevel' => 'INFO',
'UsePAM' => 'yes',
},
}
The Hash values can be used in your custom templates:
- Allow management of any kind of configuration parameter
- Provide endless configuration values without adding new parameters
- Works only for parameters used in templates on in custom classes

Example42
CUSTOM OPTIONS IN TEMPLATES
Rule #5
Alternative ways to use the options hash in an erb template:
Direct but not safe (you must always provide all the used options)
UsePAM <%= options['UsePAM'] %>
Failsafe with defaults (verbose but safe)
<% if scope.lookupvar("openssh::options['UsePAM']") then -%>
UsePAM <%= options['UsePAM'] %>
<% else -%>UsePAM no<% end -%>
Show what you have (useful for config files has defaults for every option)
<% scope.lookupvar("openssh::options").sort_by {|key, value|
key}.each do |key, value| -%>
<%= key %> <%= value %>
<% end -%>
The smart way: options_lookup (Use the option value or set a default)
UsePAM <%= scope.function_options_lookup(['UsePAM',‘no’]) %>

Example42
CUSTOMIZE: CUSTOM CLASS
Rule #6
Provide added resources in a Custom Class:
class { 'openssh':
my_class => 'site/my_openssh',
}
This autoloads: site/manifests/my_openssh.pp
Custom class can stay in your site module:
class site::my_openssh {
file { 'motd':
path => '/etc/motd',
content => template('site/openssh/motd.erb'),
}
}
You hardly need to inherit openssh: there are parameters for everything
Do not call your class site::openssh, naming collisions could happen.

Example42
EASY DECOMMISSIONING
Rule #7
Disable openssh service:
class { 'openssh':
disable => true
}
Deactivate openssh service only at boot time:
class { 'openssh':
disableboot => true
}
Useful when a service is managed by another tool (ie: a cluster suite)
Remove openssh (package and files):
class { 'openssh':
absent => true
}
Monitoring and firewalling resources removal is automatically managed

Example42
CROSS-MODULE INTEGRATIONS
Rule #8
Integration with other modules sets and conflicts management is not easy.
Strategy 1: Provide the option to use the module’s prerequisite resources:
class { 'logstash':
install_prerequisites => false, # Default true
}
The prerequisites resources for this module are installed automatically BUT can be
managed by third-party modules
Strategy 2: Use if ! defined when defining common resources
if ! defined(Package['git']) {
package { 'git': ensure => installed }
}
Not a definitive solution, but better than nothing.
Strategy 3: Always define in Modulefile the module’s dependencies
dependency 'example42/puppi', '>= 2.0.0'
Strategy 4: Never assume your resource defaults are set for others
Exec { path => "/bin:/sbin:/usr/bin:/usr/sbin" }

Example42
EXTEND: MONITOR
Rule #9
Manage Abstract Automatic Monitoring:
class { 'openssh':
monitor => true,
monitor_tool => [ 'nagios','puppi','monit' ],
monitor_target => $::ip_address # Default
}
Monitoring is based on these parameters defined in params.pp:
port => '22',
protocol => 'tcp',
service => 'ssh[d]', # According to OS
process => 'sshd',
process_args => '',
process_user => 'root',
pid_file => '/var/run/sshd.pid',
Abstraction is managed in the Example42 monitor module
Here “connectors” for different monitoring tools are defined and can be added (also
using 3rd party modules).

Example42
EXTEND: FIREWALL
Rule #9
Manage Automatic Firewalling (host based):
class { 'openssh':
firewall => true,
firewall_tool => 'iptables',
firewall_src => '10.0.0.0/8',
firewall_dst => $::ipaddress_eth1, # Default is $::ipaddress
}
Firewalling is based on these parameters defined in params.pp:
port => '22',
protocol => 'tcp',
Abstraction is managed in the Example42 firewall module
Currently only the “iptables” firewall_tool is defined, it uses Example42 iptables module
to manage local iptables rules

Example42
EXTEND: PUPPI
Rule #10
Manage Puppi Integration:
class { 'openssh':
puppi => true, # Default: false
puppi_helper => 'standard', # Default
}
The Puppi module is a prerequisite for all Example42 modules
Is required because it provides common libs, widely used in the modules
BUT the actual puppi integration is optional (and disabled by default)
Puppi integration allows CLI enrichment commands like:
puppi info openssh
puppi log openssh
puppi check openssh
Note: puppi support for info/log commands for NextGen modules is under
development
Puppi helpers allow you to customize Puppi behavior

DOWNLOAD
Example42 Puppet Modules Site:
http://www.example42.com
GitHub repositories:
http://github.com/example42
Git Download:
git clone -r http://github.com/
example42/puppet-modules-nextgen
Note on GitHub repos:
puppet-modules-nextgen contains only NextGen
modules (as git submodules)
puppet-modules contains both NextGen and older
modules

One more thing...
How to make a NextGen module
git clone -r http://github.com/example42/puppet-modules-nextgen
cd puppet-modules-nextgen
Example42-tools/module_clone.sh
This script creates a skeleton for a new module based on different Example42 foo module
templates. Run it from the directory that contains the foo module (moduledir).
By default it uses the "foo" module as template.
Specify -t <source_module> to use a different template.
Example:
Example42-tools/module_clone.sh -t foo_webapp
Source module template is foo
Enter the name of the new module based on foo: mynewmodule
E di t my n ewm o dul e / m an i f e st s/ param s.pp t o m an age di f f e re n t OS
A new, basic, NextGen module based on the foo template is done.
Add features and application specific resources to enrich it

Graphics:
www.tatlin.net
ad maiora
Questions?
@alvagante