1. Story of Puppet @eBay Classifieds Group
or how we came to the setup we use

2. Some bio or “Who is Vladimir”
Sysadmin for eighteen years
Currently in Solutions team of eCG’s marktplaats.nl
All-round nerd
Hobbyist programmer

3. Puppet at eBay Classifieds:
9 countries
13 platforms
1 codebase
1 change management process

4. Puppet at eBay Classifieds:
We started adopting puppet in 2008
Since then we had about 4 different iterations of “this is bloated and
doesn’t work, let’s think more and re-do”
Currently we believe we achieved a workable scenario where we:
Enable teams to share and re-use puppet code
Enforce the same code style and standards
Allow for peer-reviews of the code, also enforce them
“If it’s not in Puppet – it does not exist.”

5. What we are going to talk about:
• Different iterations of puppet implementation at eCG
• Go in-depth on the structure of the current one
• Process around the codebase
• Tools

6. “Bucket of recipes”
Puppet at eBay Classifieds (1st implementation):
A bunch of modules without visible structure (server-role based)
Relations between modules on any levels (includes, instantiations)
One Subversion repository, where everyone works on trunk
What were the lessons learned?
 As puppet adoption grew, modules for different platforms started
clashing (same, yet a bit different functionality)
 As number of modules and their dependencies grew people were
afraid of making changes - dependency tree was very hard to follow
 Resulted in tons of duplicate code
 Change management was very risky – you commit something and
it’s live

7. “Buckets of recipes”
Puppet at eBay Classifieds (2nd implementation):
What did we improve?
Release management with Subversion branching
Better module grouping
What were the lessons learned?
 Waiting on releases is killing. If you need a change - you need it
now, not in 2 or 3 days
 Provokes people to lock machines to their own puppet environment
 Surprises at times of the releases. For some people it was daytime,
for others it was middle of the night

8. Puppet at eBay Classifieds (3rd implementation):
What did we improve?
Switched to Git
Completely redesigned modules structure (more on this later)
Introduced extlookup for data separation vs configuration logic
Release more often
What were the lessons learned?
 When release process is human-dependent it will be a bottleneck
 Extlookup is nice, but in many places unreadable

9. Puppet at eBay Classifieds (4th implementation):
What did we improve?
Introduced Gerrit for code reviews
Implemented automatic puppet-lint, template syntax and PEP8 checkers
Introduced automated quorum votes for peer reviews
Release as soon as your code passed checks and YOU need it live
Further improved module tree structure
Replaced extlookup with hiera
Introduced puppetdb

10. So how does it look now?
Four module levels
One include direction Config
No includes on the same level
Platform
eCG
Shared

11. Shared modules
Shared modules are very basic, deploy one
piece of software in its standard configuration (no Config
customization)
Platform
Standalone. Includes no other modules
Installs needed packages, basically
eCG
Parameterized to allow configuration override
Examples: apache, ntpd, activemq, mysql, postgres Shared

12. eCG modules
eCG modules add our specific eCG-wide
approved customizations. Rule: it must play well Config
with majority of teams to make it
Platform
Mostly include shared modules
Never include each other
eCG
Override default shared module configuration
with eCG specific one via parameters Shared
May be parameterized to allow for configuration
tweaks or overrides
Examples: puppetmaster, nagios checks, collectd
plugins

13. Platform modules
Platform level introduces company-specific
modules and configuration customizations
Config
May include eCG modules
May include shared modules Platform
Never include each other
eCG
Shared

14. Config modules
Config level introduces server-role specific
modules
Config
Are the starting point for our servers
configuration
Platform
May include platform, eCG or shared modules
eCG
Never include each other
Node definition must include config module Shared

15. Why did we do it like this?
Easy starting point. Look up a node, see its
class, open the class – you see on one page
how your server is configured on a global level Config
If you need to know more, dive deeper
Platform
We enable code sharing, we do not enforce it
If you see a module in shared you can use – use eCG
it. If you can’t use it, extend it in a backwards-
compatible way or write other extension on an Shared
appropriate level

16. Hiera
Great way to separate code from data
Company-based structure
Lookup path based on facts to allow company data separation
while keeping the same source tree
:datadir: /var/lib/hiera/%{environment}/%{companyprefix}
Inside company dirs a variety of fact-based lookups to allow for
fine-grained data structures

17. PuppetDB
Much faster catalog compilations for huge catalogs (nagios
server with 200k resources)
REST API to talk to for inventory and searches
Many brilliant ways to (ab)use it
Stores all facts (can be used to feed CMDBs)
Stores most recent catalog (can be used for inspections)
POC we are running - internal Puppet naginator module is very slow,
collection of exported resources takes about 15 minutes. We use
PuppetDB to generate the same nagios configs in just under 8
seconds
“If there is data, it can be used”. PuppetDB is a central data store.
Love.

18. So all good, but how does this work in practice?
Local Test the change
Push to Gerrit Code review Merge
environment with --environment
Per-user environments on puppetmasters
Make the change in your own env
Test it from your own env
Push to Gerrit
Ask team members to review
Merge

19. Gerrit
Great code review tool (https://code.google.com/p/gerrit/)
Acts as a Git server with fine-grained access control
Each commit is a separate change, that requires votes for
approvals
Votes can be +1 for Code Review, +2 for change approval
No +2 – no merge
We use voting system with auto +2
Depending on platform size or number of platforms affected by the
change more votes are needed to get a change through
Each change can be “pushed through” in case of an emergency

20. More Gerrit goodies
Allows for post-push hooks
Puppet-lint
Ruby templates parsing
PEP8 (we love Python)
Each change is separately checked out by hooks and tested
for syntax or other obvious errors
Concept of “one commit – one change” forces people to create
atomic changes that are easy to revert in case of errors
Keeping it all in the same system, even if you decide to split Git
repositories still makes work done by colleagues VISIBLE

21. Gerrit and puppetmasters
Master is automatically pulled by crons on puppetmasters
Gerrit disallows pushing directly to master
Even if you force-merge a change, it makes your change
VISIBLE and easily REVERTIBLE
Takes about half a minute extra time
Ensures you didn’t slack out on syntax checkers

22. To sum it all up
There is no single structure that fits every company
Puppet code is like any other code
Write
Use
Refactor if needed
Layered modules structure can work well in many cases
Complementary tooling around Puppet is a life- (and time-)
saver
Lint(s), Gerrit, PuppetDB

23. Work to be done in our house
Approval process improvements
Better quorum schemas
Easier ways of code style and syntax checking
See that people don’t leave changes “hanging”
Integrate Gerrit with corporate GitHub
Work more with community
Open source some generic shared modules
Offer changes to PuppetForge modules
POC with Vagrant for puppet code testing

24. Questions?

25. Thank you!
Special thanks to:
• PuppetLabs and eCG for the opportunity to speak here
• Teams of eCG for bearing with me all these years
• You all for attending and listening
You can reach me by:
• Mailing to favoretti@gmail.com
• Following me on GitHub
(https://github.com/favoretti)
We are hiring!
Visit http://ebayclassifiedsgroup.com/careers
for more info.