"3 Years of Puppet at Cisco: The Secrets to Our Success" by Keith Chambers and Ryan Uber, Cisco WebEx. Presentation Overview: WebEx Social (WxS) is an enterprise social networking product by Cisco. Customers can subscribe to WxS as a cloud service or they can purchase a license and run WxS on-premise. Puppet is at the core of our technology stack that includes MongoDB, Solr, RabbitMQ, memcached, ZooKeeper, Nagios, and Graphite. We love Puppet. We pioneered embedding Puppet in Cisco products 3 years ago and have been tireless advocates ever since. We have learned valuable technology/process/culture lessons and developed reusable patterns for: - Scaling to thousands hosts with Puppet - Ensuring host upgrades are 100% reliable - Avoiding Puppet codebase spaghetti - Detecting and eliminating system drift at scale - Creating a culture of shared responsibility through architecture design. Speaker Bio: Keith Chambers,Technical Leader, Cisco Keith Chambers is the platform architect for the WebEx Social product at Cisco. He joined Cisco in 2000 and is based out of Seattle Washington. In his current role, Keith leads the engineering team responsible for deploying and operating WebEx Social. Prior to joining the WebEx Social team, Keith worked in Cisco's Unified Communication Business Unit and Technical Assistance Center. Keith was instrumental in the successful virtualization of Cisco's entire Unified Communications portfolio and spoke at VMworld 2009 on the topic of "virtualizing real-time applications". VMware presented Keith with their prestigious vExpert award in 2009. Outside of Cisco, Keith is a house deejay and producer. He enjoys traveling the world and spending time with his family, friends, and beautiful wife Jackie.

1. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1
3 Years of Puppet at Cisco:
The Secrets to Our Success
PuppetConf 2013
Keith Chambers & Ryan Uber

2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
• Worked at Cisco for 13 years
- 7 years in Technical Assistance Center (support)
- 2 years in Product Marketing
- 4 years as WebEx Social Platform Architect
• Passionate about technology
• Passionate about people
• Always looking to raise the bar

3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
• Worked at Cisco for 3 years
- Software Engineer
- Has played a critical role in our success
• Work at a managed host provider for 5 years
- Designed and implemented most of the infrastructure
- Supported everything he built
• Puppet user for 6 years

4. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
• Overview
• Experience
• Plumbing
• History & Evolution
• Secrets to our success

5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5

6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
• Enterprise Social Media
- Create content
- Communities
- Follow
- Like
- Activity feed
- Recommendations
• On-premise deployment
model
• Later added cloud hosted

7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Persistence
• Traditional tiered web service
• Large number of services
- App Server
- Worker
- OpenFire
- RabbitMQ
- Memcached
- MongoDB
- Oracle
- Solr
Caching
Application
Web Server
Worker

8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
• Virtual appliance delivery model
• Roles based architecture
- 1 service per VM
- 25 VM deployments are
common
• Director provides system wide:
- Configuration
- Software maintenance
- Diagnostics and logging
- Health monitoring
• Everything works out of the box
Oracle
OS
MongoDB
OS
Solr
OS
Worker
OS
Memcached
OS
App
OS
RabbitMQ
OS
OpenFire
OS
Virtualized Compute & Storage
Director
OS

9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9

10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
1. Customer downloads a single OVA from Cisco

11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
1. Customer downloads a single OVA from Cisco
2. Customer deploys the Director VM from the OVA by selecting the
Director configuration option

12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12

13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
1. Customer downloads a single OVA from Cisco
2. Customer deploys the Director VM from the OVA by selecting the
Director configuration option
3. Customer configures Director VM network settings

14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
1. Customer downloads a single OVA from Cisco
2. Customer deploys the Director VM from the OVA by selecting the
Director configuration option
3. Customer configures Director VM network settings
4. Customer browses to the Director UI and configures system settings
and defines the topology

15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15

16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16

17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
1. Customer downloads a single OVA from Cisco
2. Customer deploys the Director VM from the OVA by selecting the
Director configuration option
3. Customer configures Director VM network settings
4. Customer browses to the Director UI and configures system settings
and defines the topology
5. Customer deploy all VMs defined in the topology from the OVA by
selecting the appropriate configuration

18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
1. Customer downloads a single OVA from Cisco
2. Customer deploys the Director VM from the OVA by selecting the
Director configuration option
3. Customer configures Director VM network settings
4. Customer browses to the Director UI and configures system settings
and defines the topology
5. Customer deploy all VMs defined in the topology from the OVA by
selecting the appropriate configuration
• System handles all other setup and configuration

19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
1. Customer download a single release image from file from Cisco

20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
1. Customer download a single release image from file from Cisco
2. Customer uploads the release image to the Director

21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21

22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
1. Customer download a single release image from file from Cisco
2. Customer uploads the release image to the Director
3. Customer clicks upgrade

23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
1. Customer download a single release image from file from Cisco
2. Customer uploads the release image to the Director
3. Customer clicks upgrade
• System handles the upgrade

24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24

25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25

26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26

27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
OVF
OS
Bootstrap
Config
Network
Settings
Director FQDN
Glue Code

28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28

29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
• Package everything as an RPM
• RPMs distributed from Director via
Yum
• Controller is a Puppet Master
Director
OS
Analytics
Store
OS
Puppet
Master
Yum

30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
• Versioned build artifact containing:
- All RPMs for all roles
- All Puppet modules
• Upgrade mechanics:
- New image uploaded to Director
- Old image unmounted and new
image mounted
- Director calls Puppet run for all
hosts
- On success the updated version
is stored in on VM file system
- Salt used to fetch version
number from all VMs

31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
Director
OS
Analytics
Store
QUAD OS
JSON Store
QUAD OS
RDBMS
Store
QUAD OS
Graph
Store
QUAD OS
Rsyslog
• Rsyslog
- High performance
- Reliable
- Part of CentOS 6
Rsyslogx Rsyslogx Rsyslogx Rsyslogx

32. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Director
OS
Analytics
Store
OS
BSON
Store
OS
RDBMS
Store
OS
Social
Graph
OS
monit monit monit monit
Nagios
• Nagios:
- Reliable
- Extremely flexible
- Available in EPEL
• Monit
- Reliable
- Flexible and extensible
- Enables self healing
- HTTP API
- Available in EPEL

33. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
Director
OS
Analytics
Store
OS
BSON
Store
OS
RDBMS
Store
OS
Social
Graph
OS
Graphite
collectd collectd collectd collectd
• Graphite:
- Reliable
- Extremely flexible
- Available in EPEL
• Collectd
- Reliable
- 100+ plugins
- Extensible
- In EPEL

34. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34

35. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
Phase 1
Mid 2010
• 20GB VM
• CentOS 5
• deployer.zip
• Monitoring 1.0
• 100% proprietary
• Time to market focused
• No DevOps tooling
experience in the team

36. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Phase 1
Mid 2010
• 20GB VM
• CentOS 5
• deployer.zip
• Monitoring 1.0
• 100% proprietary
• Time to market focused
• No DevOps tooling
experience in the team
Phase 2
Early 2011
• RPM + Yum
• Puppet
• Release Image
• Reduce VM footprint
• Major pushback on Puppet
• Your team matters

37. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Phase 1
Mid 2010
• 20GB VM
• CentOS 5
• deployer.zip
• Monitoring 1.0
• 100% proprietary
• Time to market focused
• No DevOps tooling
experience in the team
Phase 2
Early 2011
• RPM + Yum
• Puppet
• Release Image
• Reduce VM footprint
• Major pushback on Puppet
• Your team matters
Phase 3
Mid 2011
• Director REST API
• OVF Properties
• Unified Logging w/
Rsyslog + Scribe
• Rsyslog was resisted due
to name association with
syslog

38. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Phase 1
Mid 2010
• 20GB VM
• CentOS 5
• deployer.zip
• Monitoring 1.0
• 100% proprietary
• Time to market focused
• No DevOps tooling
experience in the team
Phase 2
Early 2011
• RPM + Yum
• Puppet
• Release Image
• Reduce VM footprint
• Major pushback on Puppet
• Your team matters
Phase 3
Mid 2011
• Director REST API
• OVF Properties
• Unified Logging w/
Rsyslog + Scribe
• Rsyslog was resisted due
to name association with
syslog
Phase 4
Early 2012
• CentOS 6 upgrade
• Monit
• Salt
• Remove Scribe
• Maintain fewer libraries
• Further reduce VM footprint
• Started replacing Monitoring 1.0
• Salt proved buggy

39. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Phase 1
Mid 2010
• 20GB VM
• CentOS 5
• deployer.zip
• Monitoring 1.0
• 100% proprietary
• Time to market focused
• No DevOps tooling
experience in the team
Phase 2
Early 2011
• RPM + Yum
• Puppet
• Release Image
• Reduce VM footprint
• Major pushback on Puppet
• Your team matters
Phase 3
Mid 2011
• Director REST API
• OVF Properties
• Unified Logging w/
Rsyslog + Scribe
• Rsyslog was resisted due
to name association with
syslog
Phase 4
Early 2012
• CentOS 6 upgrade
• Monit
• Salt
• Remove Scribe
• Maintain fewer libraries
• Further reduce VM footprint
• Started replacing Monitoring 1.0
• Salt proved buggy
Phase 5
Late 2012
• Replaced Monitoring 1.0 with
Nagios, Collectd, Graphite
• Last minute decision to go with
Graphite
• Puppet code spaghetti
• VM tampering undercutting upgrade
reliability
• Other Cisco groups want to reuse
our code

40. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
Phase 1
Mid 2010
• 20GB VM
• CentOS 5
• deployer.zip
• Monitoring 1.0
• 100% proprietary
• Time to market focused
• No DevOps tooling
experience in the team
Phase 2
Early 2011
• RPM + Yum
• Puppet
• Release Image
• Reduce VM footprint
• Major pushback on Puppet
• Your team matters
Phase 3
Mid 2011
• Director REST API
• OVF Properties
• Unified Logging w/
Rsyslog + Scribe
• Rsyslog was resisted due
to name association with
syslog
Phase 4
Early 2012
• CentOS 6 upgrade
• Monit
• Salt
• Remove Scribe
• Maintain fewer libraries
• Further reduce VM footprint
• Started replacing Monitoring 1.0
• Salt proved buggy
Phase 5
Late 2012
• Replaced Monitoring 1.0 with
Nagios, Collectd, Graphite
• Last minute decision to go with
Graphite
• Puppet code spaghetti
• VM tampering undercutting upgrade
reliability
• Other Cisco groups want to reuse
our code
Phase 6
Started 2012
• Generic programmable
Controller
• Deploy VMs via IaaS API
• Orchestrate deploy and upgrade
with Fabric
• Baked VM images
• Hiera for module parameters

41. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41

42. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
Keep it simple
• 80/20 rule
• Build only what you need
• What works is boring
Learn and adapt
• Don’t fall in love
with what you’ve
built
Unix tool chain
pattern
• Loosely couple with
best of bread tools
• End-to-end
solutions lock you in

43. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
• Excellent at configuration management
• Bulletproof reliability
• Long term investment protection
• Vibrant community and ecosystem
• Commitment to open source

44. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
• Alex Honor of DTO Solutions and Simplify Ops
- Insightful program leadership
- Walking encyclopedia of proven patterns
- Always there when I’m in a jam

45. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45