Rafael Brito at PuppetCamp Europe 2010 "Deploying datacenters with Puppet." Follow along with the Video at: https://www.youtube.com/watch?v=3DaWrKQ82j4 Puppet Camp Europe 2010: Ghent, Belgium May 27-28, 2010

1. Deploying Data Centers with Puppet –
A user’s perspective
May 27, 2010
© NYSE Euronext. All Rights Reserved.

2. Objective of this Presentation
Share to the Puppet community the solution arrived by NYSE Euronext’s
System Architecture and Engineering (SAE) team to address the technical
challenges imposed by the Business Strategy of the company:
• Consolidate various Data Centers across the globe in two brand-new
state-of-the-art facilities, one in the US and another in Europe;
• Adapt a Global standard process to build and configure servers and
applications across multiple countries and teams;
• Quickly deploy changes in large scale in an automated way.

3. Main Tools Used in the Solution
• RHEL Kickstart
– A single, very frugal kickstart profile for each RHEL release for the
entire company.
• RH Network Satellite
– Manage packages of the OS, Third Parties and Home-grown RPMs.
– Manage patches and upgrades.
• Puppet
– Work horse of this model
– Common framework to apply and configure the Global standard server
configuration and particularities of each environment and applications.

4. The Three Layer Approach
We have organized our configuration needs in three levels:
Base: All NYX global customizations on top of the RHEL default settings.
These must be propagated across the enterprise.
Example: global standard size for /usr file system, kernel parameter kernel.core_pattern
= /var/crash/core.%e.%p.%h
Zone (or environment or network): Configuration items common to a specific
network or environment. Zone layer inherits the Base layer with the ability to
override anything if necessary.
Example: /etc/resolv.conf file across one given production network.
Application: Any specific configuration required by any given application.
This layer inherits the Zone layer and has the ability to change anything set
on the previous two layers.
Example: UTP application requires a 50GB /appl file system and a “qt” package installed.

5. The Puppet Modules behind of this approach

6. An example of the modules-base manifests
• NYX Global Build customizes 250+ items on top of RHEL default OS
• All servers include the entire modules-base via the module name ‘base’
• Example:
• class base::setup {
include ssh::setup
include banners::setup
include resolver::setup
include ntp::setup
include sysctl::setup
include profile::setup
include services::setup
include locale::setup
include email::setup
include yum::setup
include crashdump::setup
include bootloader::setup
include rclocal::setup
include sudo::setup
include hardware::setup
include tcpwrappers::setup
include firewall::setup
include etc_services::setup
}
6

7. An example of the modules-base manifests
class sysctl::setup {
# Configure kernel core dumps
sysctl { "kernel.core_pattern":
val => “/var/crash/core.%e.%p.%h",
ensure => present,
}
}
class profile::setup {
# Deploy custom /etc/bashrc
file { "/etc/profile":
owner => root, group => root, mode => 0644,
source => "puppet:///profile/etc_profile",
}
}
7

8. An example of the modules-zones manifests
The organization has 20+ zones (and growing) across the globe, including
production, QA, development networks, which one with own settings.
Example of a zone setting:
class trading::setup {
$country = 'us'
$searchpath = 'nyx.com'
$domain = 'nyx.com'
$nameservers = [ "10.0.X.X", "10.0.X.X" ]
$timeservers = [ "10.0.X.X version 3 prefer", "10.0.X.X" ]
$snmp_ro_communities = [ "dfasdfasdfasdf" ]
$snmp_rw_communities = [ "sdfasdfasdfasd 127.0.0.1" ]
$snmp_dlmods = [ "cmaX /usr/lib64/libcmaX64.so" ]
$trapsink = [ "sdfasdfasd" ]
$syslogserver = "10.0.X.X"
$postfix_mailhost = "mailrelay.nyx.com"
$ilo_licensed = "true"
$ilo_password = "my_password"
# Including the base classes
include base::setup
}

9. An example of the modules-app manifests
• 60+ applications requiring their own settings
• We try to configure the application puppet manifest to be pluggable on any zone
Example of one application manifest:
class sg::setup {
package { ["libxslt", "libmng"]:
ensure => present,
}
lvm::config_lvm { "tcpdump":
mount_point => "/tcpdump",
lvname => "ltcpdump",
vgname => vg00,
lvsize => "5G",
fstype => "ext3",
owner => “myuser",
group => "users",
mode => "0755",
}
}

10. How the three layers work together
•The zone class usually sets variables that will be used by modules-base for
configuration.
•The zone class necessarily includes the modules-base.
•All servers must belong to one zone.
•The application module is coded to be pluggable to any zone (i.e. match
engine application module can be on a QA environment or production
environment)
•The ability to build anything, anywhere and mimic Production config to QA
environment and vice-versa.
•The modules-app can override settings inherit on modules-zones. The
modules-zones can override settings inherit on modules-base.
• Using puppet, we synchronize the build server process with the latest
requirements of the applications.

11. The Last Piece: the Node Definition File
The node definition is the entity that matches the hostname to the zone and
application modules. In addition, it has the networking configuration of the server.
node "buildtest.nyx.com" {
# Zone Class
include corplan::setup
# Application Class
include sg::setup
# Networking
network::interface::setup {"bond0":
ensure => present,
config_type => bonded,
ip => "10.0.0.100",
netmask => "255.255.255.0",
slave0 => "eth0",
slave1 => "eth1",
gateway => "10.0.0.254",
arp_target => "10.0.0.254",
arp_interval => "3000",
restart_network => false,
}
}

12. NYX Build Server Components
• DHCP/DNS Server (optional)
• TFTP / PXE Server
• Satellite Server or Proxy
• Apache Server
• Puppet Server
• MySQL Server
• The server resides on a separate network called build network since we
do not build any server on productions network.

13. Automated Installation Process - Overview
PreBoot Phase
Default kickstart On the Build Server, the
label that register script generates a
PXE Boot for 1st time
executes a perl customized PXE file after
script via “wget” looking up Serial Number in
to register the mysql database. Custom
Serial Number PXE has hostname and OS
and the MAC version (RHEL4 or RHEL5).
address and
Server out of the reboots server.
box
Phase 0
Start the RHEL The kickstart post install script
installation with register the new server on the
PXE Boot for 2nd time
the minimum satellite server and assign it to
packages and the appropriate channels.
Global Standard Puppet client is installed just
File Systems before server is rebooted.
setup and sizes.
Server come back
from the first reboot
Continue

14. Automated Installation Process – Overview Cont.
Phase 1
Load lean OS from disk + init script
An init.d script A node definition is
runs puppet for mandatory at this point and
the first time and puppet client configures
downloads its own server to the NYX Global
configuration from base (default) configuration
the puppet server. and networking.
Server reboots for the third
time
Phase 2
On Phase2 is The phase 2 execution can be
Phase2
where Puppet is triggered directly from Phase1
init script
executed for the init script (in this case is
last time to executed on the build network)
configure the or can be executed once the
server to their final server is on the production
zone network.
(environment) and
application
Server might reboot or settings.
execute Phase2 init.d
script directly

15. PreBoot Phase
• Very simple kickstart profile launched by default PXE label
• It does nothing than executing a script via wget
• It registers the server with two parameters: Serial Number and the MAC
• The script will create a custom PXE boot file with given MAC that contains
the RHEL release and the hostname of the server.
• Example of the kickstart pre-script:
%pre
SERVER=`grep 'url address' /tmp/anaconda.log | awk '{print $6}'`
cd /tmp && wget -q http://${SERVER}/pub/dmidecode
chmod +x /tmp/dmidecode
SN=`dmidecode | grep "Serial Number" | head -1 | awk '{print $3}'`
MAC=`ifconfig | grep HWaddr | awk '{print $5}'`
wget -q http://${SERVER}/cgi-bin/register.pl?mac=$MAC&sn=$SN
reboot

16. Phase 0
• A single kickstart profile for each RHEL release to the entire organization
• Very minimum number of packages installed
• Larger OS File system sizes than the RHEL Default
• Server is registered to the Satellite
• Puppet client is installed (puppetd)
• NYX-Firstscript is set to run on the next reboot via init.d

17. Phase 0 – Fragment of the Kickstart profile
%post --interpreter /bin/sh
echo "Installing puppet..."
yum -y install puppet facter ruby-rdoc ruby-shadow
# Puppet is install via the activation key but by default will run as
a daemon
echo "Chkconfig'ing puppet off..."
chkconfig puppet off
# Grab nyx-firstboot script from the satellite or proxy
RHN_SERVER=`grep '^serverURL=' /etc/sysconfig/rhn/up2date | cut -d'/'
-f3`
echo "Detected Puppetmaster/RHN as: $RHN_SERVER"
echo "Downloading nyx-firstboot script..."
wget -O /etc/init.d/nyx-firstboot http://$RHN_SERVER/pub/glb/nyx-
firstboot
# Make it executable and make it start on firstboot
echo "Installing nyx-firstboot..."
chmod 755 /etc/init.d/nyx-firstboot
chkconfig --add nyx-firstboot

18. Phase 1
• It is run by the nyx-firstboot init script
• nyx-firstboot executes puppetd twice: one to configure itself and another to
set the server to the base config.
• nyx-firstboot script command lines that executes puppetd
# Get puppet server address, the same as the RHN Proxy/RHN Satellite
PUPPET_SERVER=`grep '^serverURL=' /etc/sysconfig/rhn/up2date | cut
-d'/' -f3`
# Run puppet once to configure itself and get SSL certs auto-signed
puppetd -tov --server=$PUPPET_SERVER --tags
utilities::puppetclientbootstrap --no-noop
# Run puppet again to apply all base build manifests
FACTER_sys_building=true puppetd -tov --server=$PUPPET_SERVER --tags
sys_building --no-noop

19. Phase 2 (Last phase)
The nyx-phase2 script runs puppetd again, this time puppetd fully executes
the puppet manifests which configures the server entirely to the zone and
application. The nyx-phase2 is set to execute automatically from init.d OR
triggered automatically by nyx-firstboot.
Fragment of the script:
# Run puppet to apply application specific manifests
puppetd -tov --no-noop | /usr/bin/tee /tmp/puppet2b
if egrep -q "err:|warning:" /tmp/puppet*; then
logger -t nyx-phase2 "Found excessive errors on puppet logs!"
else
# Shutdown the server - to be shipped to the rack
logger -t nyx-phase2 "Shutting down the system! No errors found on
the puppet logs!"
/usr/bin/poweroff
fi

20. Data Center Deployment: Numbers and Lessons
• One server gets completely installed in 20 minutes in unattended mode
• We usually install up to 15 servers in parallel, but we had to upgrade first the
puppet master to run on Apache Web Server. The default web server has shown not
to be very powerful.
• Usually, one individual in one day can install up to 50 servers by himself
• The most difficult part was getting the requirements of each application

21. Ancillary Tools for This Project
• Subversion (extremely important):
– we had to disseminate the culture of version control among many individuals and teams.
– Each layer of the modules and node definitions have its own subversion repositoriy
• Web Subversion
• OpenGrok (search the manifests)
• Redmine (ticket and changes control)
• Tidal Scheduler – To execute puppet client once the servers are in the production
network

22. Next Steps
• Run puppetd to deploy changes across the board on the day-by-day basis. This will
require more a cultural change and coordination across more departments.
• Run puppetd in report mode daily (no-intrusive) to detect discrepancies between
the reality and the puppet manifests
• Use puppet dashboard as a central database of Node definitions (called External
Node Classifier)
• Create more custom facts and types in ruby (diminish the number of execs inside
the manifests).
• Use of external resources to automate other configuration such as Veritas Cluster.

23. Thank you!
Q&A