Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density. Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed. Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances. www.serverdensity.com/puppetcamp/

1. From zero to Puppet
http://www.rankpop.com/you-need-to-start-structuring-your-blog-posts-asap/

3. +15TB / mth
+1bn docs /mth
2-5k inserts/s @ 3ms
10K RPM @ 140ms

4. Servers
HTTP Load Balancer – 5x
Apache - 14x
Build - 2x
MongoDB - 19x data,
13x routing,
6x configuration,
6x arbiter,

5. HTTP Load Balancer
$globalIPs_array = split($globalIPs, ',')
<% globalIPs_array.each do |globalIP| ­%>
ListenHTTP
# primary public IP address
Address <%= globalIP %>
Port 80

6. Apache

7. Apache
file
{
'vhost':
path => "/etc/apache2/sites­enabled/${::siteDomain}",
ensure => file,
content => template('apache­php/vhost.erb'),
notify => Service['apache2'],
}

8. Apache
<VirtualHost *:80>
ServerName <%= siteDomain %>
DocumentRoot <%= documentRoot %>
...
ErrorLog /var/log/apache2/error­<%= siteDomain %>.log
<% if @requestLogging and requestLogging == "yes" %>
CustomLog /var/log/apache2/access­<%= siteDomain %>.log vhost_combined
<% end %>
<% if enableSSL == "yes" %>
<VirtualHost *:443>
SSLEngine On
SSLCertificateFile /var/www/ssl/<%= siteDomain %>.crt
SSLCertificateKeyFile /var/www/ssl/<%= siteDomain %>.key
...

9. $excess_bagage = [
"ppp",
"bind9­host",
"laptop­detect",
"open­iscsi",
"libnss3:i386"
]
package
{
$excess_bagage:
ensure => purged,
}

10. exec
{
'ufw allow is­1':
command => '/usr/sbin/ufw allow from 184.173.178.67',
unless => '/usr/sbin/ufw status verbose | grep "184.173.178.67"',
}
exec
{
'ufw enable':
command => '/usr/sbin/ufw enable',
unless => '/usr/sbin/ufw status | grep "Status: active"',
Require => [Exec['ufw allow is­1']],
notify => Exec['ufw reload'],
}
exec
{
'ufw reload':
command => '/usr/sbin/ufw reload',
require => Exec['ufw enable'],
refreshonly => true,
}

11. SSH
file
{
'sshd_config':
path => '/etc/ssh/sshd_config',
ensure => file,
content => template('sshd/sshd_config.erb'),
}
Port 22
Protocol 2
AllowUsers david pessoa chris rob wes
Compression yes
...
<% if @duoSecKey and @duoIntKey and @duoHost %>
ForceCommand /usr/sbin/login_duo
PermitTunnel no
AllowTcpForwarding no
<% end %>

12. System updates
CANARIES="
aws­prod­apac­singapore­exm­a1
a2.wdc.sl
exm1.wdc.sl
mtx­web1.wdc.sl
sdapp­web1.wdc.sl
sdapi­web1.wdc.sl
"
function canaries {
echo "REMOVE *­web1.wdc.sl FROM ROTATION!"
echo "press ENTER"
read
echo "have you REALLY removed *­web1.wdc.sl from rotation?"
echo "press ENTER"
read
for i in $CANARIES
do
echo $i
mco rpc puppetral ­I $i create type=exec
title="/bin/bash ­c 'apt­get dist­upgrade ­y'"
done
}

13. System updates
function reboot_canaries {
echo "REMOVE *­web1.wdc.sl FROM ROTATION!"
echo "press ENTER"
read
echo "have you REALLY removed *­web1.wdc.sl from rotation?"
echo "press ENTER"
read
for i in $CANARIES
do
echo $i
mco rpc puppetral ­I $i create type=exec
title="/bin/bash ­c 'reboot'"
echo "­­­­­­­­­­­­­­"
done
}

14. What to Reboot
mco rpc puppetral create type=exec
title="/bin/bash ­c 'ls /var/run/reboot­required'"
| grep ­B 1 "Resource was created"
| grep ".sl"
exm1.wdc.sl
Status: change from notrun to 0 failed:
/bin/bash ­c 'ls /var/run/reboot­required'
returned 2 instead of one of [0]
exm­md1a.wdc.sl
Status: Resource was created
Resource: {"tags"=>["exec"],
"title"=>"/bin/bash ­c 'ls /var/run/reboot­required'",
"type"=>"Exec",
"parameters"=>{:returns=>:notrun},
"exported"=>false}

15. Live Management
- Emergency
# Metrics
<%= metricsIP %> metrics­svc
# Infrastructure services
184.173.178.66 puppet
...

16. Live Management
- Emergency

17. Live Management
- Emergency

18. Live Management
- Emergency

19. Live Management
- Emergency

20. Pedro Pessoa
pessoa@serverdensity.com
www.serverdensity.com