How we built a distributed Multi-master environment.

1. MultiMaster
scaling for multiple regions
Greg Cockburn
@gergnz

2. problem:
How do we provide a
Puppet Service Globally
When WAN pipes suck

3. what's in our tool box?
VMware ESX
LDAP
F5 Load Balancers
Puppet Enterprise Edition

4. Items that need to be addressed
• Puppet Certificate management
• Node Classification and ENC replication
• Master Replication
• Master Availability
• Master Scalability
• Reporting and notifications
• Change Control

5. One Solution that Worked

6. Build a Puppeteer:
• This is a Puppet Master Master
• No Client Access
• Acts as a PuppetCA
• Central Point of Entry for Code Updates
• Ensures that the Puppet Masters are in sync

7. LDAP as an ENC:
• Existing highly available UNIX/Linux backbone service
• Already replicated to every region
• Masters are configured to speak with their nearest LDAP
replica
• Provides an effective audit trail
• Node definitions are abstracted away from the Puppet
manifests

8. Replicating Puppet Configuration:
• The Puppet Master is effective at syncing files
• Use the Puppet Fileserver to replicate the masters
o manifests
o modules
o files
o templates
• The Puppeteer can 'kick' the other masters to force a run
• Create a puppet::master class to ensure, masters are
fully controlled

9. F5 Global Traffic Management (GTM)
& DNS:
• Local Puppet Master addresses are returned to
clients based on the DNS server the request
came from
• If a Master is down then next nearest is returned
• Any Puppet Master globally can answer the
client

10. F5 Local Traffic Management (LTM):
• On sites with heavy loads this can be used to
rapidly scale the local Puppet Master service
• If a local Master is taken out of service F5 will
automatically send you to the nearest local
Master

11. All Tied Together:

12. Workflow – Adding a New Server
• Define the client characteristics in the LDAP ENC (eg.
Datacentre, Environment, Server Flavour)
• Configure the build tools
• PXE boot then server, OS is installed and puppet
bootstraps
• Once the client certificate is signed the server is
configured

13. Workflow (adding a master):
• Build a 'standard' client
• Redefine in ENC (LDAP) as a puppetmaster
• Destroy local certificates
• generate special certificates on puppetmaster using --
dns_alt_names
• rerun puppet and Master configurations will sync down

14. So What’s New:
Since this configuration was deployed Puppet Labs have
been busy:
• Puppet Sites - Will soon be released and addresses a lot
of the issues here
• PuppetDB – The new standard for stored configs

15. Special thanks to Jon Spinks @ Sourced
Group
Sourced Group are a Puppet Labs partner
providing integration services for Puppet
Enterprise Edition

16. Q&A
Please go and bother Jon Spinks to find out what Sourced
have been doing with Puppet to automate Amazon Web
Services