Julien Rottenberg, CityGrid. Presentation of the tools and workflow for our puppet setup. How puppet helps us managing 500+ servers in a hybrid environment O&O Datacenter and EC2, hands free. Watch the video at http://youtu.be/FPwga7HwomM PuppetCamp LA, May '12.

1. Puppet @ Citygrid
PuppetCamp LA 2012

2. Who am I ?
My name is Julien Rottenberg
Recovering System Administrator
Puppet user since 2009 (at GN)
I like to write code and answer questions

3. Puppet promises
No need to ssh to a
server to understand
how it works
Look at the puppet code

4. "Those who cannot remember the
past are condemned to repeat it."
George Santayana
literally... go to a server, run `history` as root.

5. Puppet promises
Guarantee that a server is setup exactly the
same way on all environments

6. Puppet promises

7. Puppet promises
All the building steps
are auditable - SDLC
Drift management
Open knowledge, no
blackbox

8. Puppet promises
and finally...

9. Toolset
Open source tools :
● No need to reinvent the wheel
● You are not alone with 'that problem',
get help

10. Toolset

11. Toolset
puppet doc
● validate code (puppet parser validate)
● produce html doc
Running on commit through Jenkins ~ 10
sec

12. Toolset

13. Toolset
puppet-lint == coding standard
Takes ~ 2 min
Part of the 'Ci-Game'

14. Toolset
trunk = work in progress - could break but fix
ASAP
production = tested code - do not break

15. Toolset Trunk --> Production (CherryPicking)
We use svnmerge.py
Code in progress = technical debt
It needs to be merged and deployed or
reverted
Fail the build if pending Cherry Picks > 30
Can't CherryPick yourself ~ code review

16. Toolset
Svn even with svnmerge has limitations
Git ?

17. Toolset
Vagrant
Sits on top of Virtualbox
Allows to 'go crazy'
Test, experiment, share configured box
$ vagrant provision == puppet apply

18. Toolset
nVentory
As an External Node Classifier
Classes:
- default_install
- gcd

19. Toolset
nVentory
Helper to build configuration files, etc.
ie replacement for Stored Config

20. Toolset
Foreman
Api for puppet status, visibility of puppetrun
(Failed attempt on ENC - probably pebkac)

21. Toolset
Man ! that's a lot of tools !1!!
And it's fine, design by contract.
Each application does one thing
and was designed to do that
thing only (so does it well)
Pipe them
Dennis Ritchie

22. Toolset
Boto for
● ec2
● route53
● elb
● sqs
● sns
River dolphin native to the amazon WP

23. Culture cloud
EC2 glitches, expect them
It used to `just work` :
● gmond (split view dns side effect, using
gmetric)
● jmx (need to specify public name at runtime)
● ELB pseudo load balancer
● Bad neighbors

24. Culture cloud

25. Culture cloud
# Build 10 nodes
$ create_ec2_instance.py -e dev -s m1.medium -c thd -n 10
# assign them to the hadoop::datanode class (specific install on top of our vanilla)
$ nv --name aws1devthd --addtonodegroup hadoop::datanode
# send traffic & enable monitoring (configs to be updated on the next puppet run)
$ nv --name aws1devthd --set status=inservice
One AMI to rule them all (map set per region)

26. Culture Collaborate

27. Culture Collaborate
"If the company's doing well and people
don't hate each other, you're probably
doing ok."
John Allspaw, speaking at Devopsdays 2011 MV.

28. You guys !
Make it better :
- Integrated function for 'secrets' (no get_var or
hiera + gpg)
Something like an encrypted data luggage
- puppet agent stability, puppetd vs cron ?

29. Guys !

30. Yesterday...

31. Two last things...
On call rotation side effect
Don't be a human keyboard

32. Shameless plug
We aggregate advertising & content from local
businesses and distribute across our network of
web & mobile sites
● Ingest : Hadoop & Cassandra
● Serve : Solr & MongoDb + Tomcat
● Code : Java mostly

33. Q/A This page : goo.gl/soxEs
puppet doc : http://docs.puppetlabs.com/guides/style_guide.html#puppet-doc
puppet-lint : http://puppet-lint.com/
Jenkins : http://jenkins-ci.org/
svn strategy : http://weblogs.java.net/blog/johnsmart/archive/2009/05/ci_branching_an.html
svnmerge.py : http://www.orcaware.com/svn/wiki/Svnmerge.py
vagrant : http://vagrantup.com/
nventory : http://sourceforge.net/apps/trac/nventory/wiki
foreman : http://theforeman.org/
boto : https://github.com/boto/boto
Don't be an Ahole : http://www.hollenback.net/DevOpsMeansDontBeAnAhole
Developers on call - Mitchell Hashimoto : http://vimeo.com/31367609
Don't be a human keyboard : http://sysadvent.blogspot.com/2010/12/day-13-dont-be-human-keyboard.html
Citygrid is hiring : http://careers.citygrid.com/
Puppet Master LA User Group : http://www.meetup.com/LAPMUG/