1. SNEAK PEEK
The Puppet Environment at
Fairfax Media / Domain Group
Dejan Golja // Gonzalo Servat

2. Introduction
Dejan Golja
Senior Systems Engineer // Fairfax Media
Twitter @dejangolja

3. Introduction
Gonzalo Servat
Senior DevOps Engineer // Domain Group
Twitter @gservat

4. Who are we?

5. Our Puppet History
☁ Early Puppet adopters … since version 0.25.X
☁ Large scale environment with distributed team
☁ We made every possible mistake
We’re on our 3rd major Puppet implementation!

6. Lessons Learned
☁ Do NOT make changes directly on the Puppet Master
☁ DO your testing - lint, code validation, etc.
☁ Do NOT pollute your downloaded official modules
☁ DO define and document the standard workflow
☁ Do NOT store sensitive data into modules

7. Example of ugliness (old site.pp)

8. Standardised
process for
deploying
changes
Easily repeatable
for multiple
environments
Puppet “V3” Requirements
High
Availability
Bring order to
our modules

9. Ingredients
☁ Packer
☁ Puppetlabs R10k
☁ Custom Ruby Service - “Captain Hook”
☁ Notification Systems - Slack/Flowdock
EC2 ASG ELB
SNS SQS RDS
CloudFormation

10. ☁ Module preferences
☁ Puppetlabs supported
☁ Vendor supported
☁ Puppet approved
☁ Custom modules (role/profile)
☁ Create module guidelines else it gets very messy!
☁ Each module lives in a separate Git repository
Puppet Modules

11. Git Flow
Feature
Branch
Workflow

12. Architecture

13. Puppet Client

14. Puppet CA

15. Puppet Masters

16. Puppet DB

17. Architecture

18. $ aws cloudformation create-stack
--stack-name puppetmaster-staging
--templatebody file://infrastructure-puppetmaster.template
--parameters
ParameterKey=Environment,ParameterValue=staging
ParameterKey=InstanceType,ParameterValue=m3.large
ParameterKey=MinScalingGroupSize,ParameterValue=2
ParameterKey=MaxScalingGroupSize,ParameterValue=4
CloudFormation Example

19. Awesome, so we have a
neat-looking HA/FT set-up…
…but how do we manage pushing
changes to Puppet Masters?!

20. ☁ Puppet masters can “come and go” randomly
☁ Keep the modules up to date per environment
☁ K.I.S.S. - Tame the learning curve for the team
☁ Avoid reinventing the wheel
Challenges

21. OUR SOLUTION

22. P3 Workflow

23. 1. Make changes and commit/push to git server
2. Git server triggers post-commit hook (POST) to Captain Hook server
3. Captain Hook server validates the payload & creates new message:
☁ Full refresh ➩ r10k deploy environment -p
☁ Light refresh ➩ r10k deploy environment
☁ Module refresh ➩ r10k deploy module <name>
4. Captain hook server pushes new message to SNS
5. Captain hook client polls & reads messages ( ➩ r10k)
6. … and we get notified in slack/flowdock/dashboard
Typical Puppet Workflow

24. DEMO

25. ☁ Fairfax is a large scale complex environment
☁ Many systems engineers are constantly pushing changes
☁ Puppet architecture designed for HA and fault tolerance
☁ Puppet workflow helps us deploy changes to multiple
masters safely and easily
Summary

27. Our Blogs
http://engineering.fairfaxmedia.com.au/
http://tech.domain.com.au/
Our Code
http://github.com/fairfaxmedia
Your Feedback or Questions
You can contact us on twitter: @dejangolja @gservat
Before you go ...