SlideShare una empresa de Scribd logo
1 de 27
Descargar para leer sin conexión
Puppet Deployment at OnApp


        Wai Keen Woon
        CTO, CDN Division
        waikeen.woon@onapp.com
WARNING	




<ObligatoryPlug>
About OnApp
       A leading provider of software for hosts



The leading cloud                     The instant global CDN for hosts
management software for
hosts

                   OnApp launched July 1st 2010
                     Deep industry knowledge
                          Backed by LDC
                 100+ employees in US, EU, APAC
Vital Statistics


                   1 in 3
                   public clouds


                 800+
              cloud deployments


                   300+
                   global clients
Customer Stories
Instant CDN that gives you…



75+       PoPs



low       cost, high margin



get       paid for idle capacity
OK.
       	




</ObligatoryPlug>
Systems Overview

l    Core & Development
      l    ~20 physical servers
      l    ~200 VMs
      l    Homogeneous environment – 64-bit Debian everywhere
      l    Mainly use OpenVZ and KVM for virtualization
l    CDN Delivery Edge Servers
      l    100+ servers in 60+ cities
      l    Running on the OnApp platform – either Xen or KVM
l    Puppet integral to our setup – since day 1
Why Puppet?

l    More reliable configuration of servers. Less need to
      “run ssh in a for loop” and miss out something.
l    Self-documenting – our manifests are almost able to
      bootstrap an empty server.
      l    Our manifests can't bootstrap an empty environment yet.
      l    Limitation – manifests describe what/where/how something
            is setup, but doesn't describe *why*.
l    Nice syntax – easy on the eyes. Comprehensive builtin
      resource types. Able to fallback to dumb ways of doing
      things if required (use file, exec et al).
Core Infra Environments

l    Systems manifest describes everything.
l    Three environments:




                         β
What Would OnApp Setup...

l    Essential utilities (tcpdump, less, vim, etc).
l    Users & their SSH keys, sudoers.
      l    Developer's shell => /bin/false if production
l    Base firewall rules.
l    Nagios agent.
l    Set uniform locality settings: UTC timezone,
      en_US.UTF-8 locale.
l    SMTP that smarthosts to our central relay.
l    Syslogd for remote logs to central logging server.
l    Finally, the services.
Core Infra Manifest Excerpt
$portal_domain   = "portal.alpha.onappcdn.com"        node
                                                        "monitoring.alpha.onappcdn.com" {
$portal_db_host = "portal.alpha.onappcdn.com"
                                                           include base
$portal_db_user = "aflexi_webportal"
                                                           include s_db_monitoring
                                                           include s_monitoring_server
$auth_nameservers = { "ns1" => "175.143.72.214",
                                                           include collectd::rrdcached
                         "ns2" => "175.143.72.214",
                                                           include s_munin
                         "ns3" => "175.143.72.214",
                                                           include s_monitoring_alerts
                         "ns4" => "175.143.72.214",
                                                           include s_monitoring_graph
                     }
                                                      }

$monitoring_host_server =                                 class collectd::rrdcached {
                                                           package { "rrdcached":
      [ "monitoring.alpha.onappcdn.com",
        "dns.alpha.onappcdn.com" ]                             ensure     => latest,
                                                           }
                                                           service { "rrdcached":
 BLUE       – env config definitions                           ensure     => running,
 RED            – node definitions                         }
 GREEN – class definitions                            }
Package Repo Integration

l    Jenkins builds debs of our code and stores it into an apt
      repository for the environment it is built for.
l    Puppet keeps packages up-to-date (ensure => latest)
      and restarts services on package upgrades.
      Puppet-agent[25431]:
      (/Stage[main]/Debian/Exec[apt-get-update]/returns) executed
      successfully

      puppet-agent[25431]:
      (/Stage[main]/Python::Aflexi::Mq/Package[python-aflexi-mqcore]/
      ensure)
      ensure changed '7065.20120530.113915-1' to '7066.20120604.090916-1'

      puppet-agent[25431]:
      (/Stage[main]/S_mq/Service[worker-rabbitmq])
      Triggered 'refresh' from 1 events

      puppet-agent[25431]: Finished catalog run in 16.08 seconds
Nagios Integration

l    Plugs into nagios – uses “exported resources”
Nagios Integration

Server manifest                           Nagios service manifest
                                           *collects the resources to check
 *exports the service that is checked


 @@nagios_service { "check_load_$fqdn":    Nagios_service <<| tag == "onappcdn.cm" |>>
                                           {
check_command =>                             target => "/etc/n3/conf.d/services.cfg",
         "check_nrpe_1arg!check_load",       require => Package["nagios3"],
   use           => "generic-service",       notify => Exec["reload-nagios"],
   host_name     => $fqdn,                 }
   service_description => "check_load",
   tag                 => $domain,
 }
Nagios Integration

l    What's logged on the nagios server when puppet runs?
      puppet-agent[15293]: (/Stage[main]/Nagios::Monitor_private/
      Nagios_host[hrm.onappcdn.com]/ensure) created

      puppet-agent[15293]: (/Stage[main]/Nagios::Monitor_private/
      Nagios_service[check_load_hrm.onappcdn.com]/ensure) created

      nagios3: Nagios 3.2.1 starting... (PID=5601)

      puppet-agent[15293]: (/Stage[main]/Nagios::Base/Exec[reload-
      nagios]) Triggered 'refresh' from 8 events
Monitoring Puppet Itself

l    Lots of tools/dashboards out there to achieve this.
l    For us: “grep -i err */syslog”. Dumb, but works until we
      need to Really Address it.
l    Common issues:
      l  Puppet gets “stuck”. And only one puppet instance

          can run at any one time.
      l  Manifest errors – syntax, merge issues.


      l  Badly-written manifests (vague dependencies,

          conditions/commands not robust enough).
      l  An important dependent resource failing (e.g. apt-get

          install fails due to dpkg-configure error).
File/Dir Organization

l    We use git to revision control our                                   l    Common branch
                                                                                  Manifests/
      puppet manifests.                                                                alpha.pp
                                                                                       beta.pp
l    Style we adopted mainly comes                                               Modules/
                                                                                       Base/
      from Hunter Haugen*                                                              Users/
l    A branch for each environment,                                       l    Alpha env branch
                                                                                   Modules/
      plus a “common” branch.                                                           Python/
                                                                                   Services/
l    Each branch checked out as a                                                      Nameserver/
      separate directory in /etc/puppet/                                   l    Beta env branch
      environments/$env                                                            Modules/
                                                                                        Python/
l    And puppetmaster's includedir                                                Services/
                                                                                        Nameserver/
      configured to that directory.
       * - http://hunnur.com/blog/2010/10/dynamic-git-branch-puppet-environments/
File/Dir Organization

l    Common goes into its own branch – for convenience;
      less merging needed for manifests that we are Really
      Sure won't differ between environments.
l    System manifest into common/manifests/$env.pp
      l    Initially tried putting manifest into alpha/beta/omega
            branches as site.pp – merge hell.
l    Introduced extra variable - $effective_env
      l    Abstracts the puppet environment name, from the
            environment that the manifest runs in.
File/Dir Organization

l    Hotfixes branch off omega and merged to alpha/beta/
      omega.
l    Development branches off alpha
      l    This branch can be trialed as a separate environment (use
            --environment to specify custom env on puppet client).
      l    Merge to alpha → beta → omega.
      l    Or merge as feature branch to any other environment.
l    “git diff branchA branchB” - differences are shown
      clearly between environments.
Edge Servers

l    Our edge servers are hosted on OnApp cloud (only).
l    When creating an edge server, the cloud control panel
      l    Instantiates a VM from a lightly-customized Debian image.
      l    Configures the package repositories.
      l    Issues a puppet run to set up.
l    Advantage of setting it up through puppet instead of a
      “gold image” - our system can be installed on bare
      metal if needed, can be reproducibly installed on
      $future_debian_release
Edge Servers

l    Our edge servers are hosted on OnApp cloud (only).
l    When creating an edge server, the control panel
      instantiates a VM from a lightly-customized Debian
      image, and issues a puppet run to set it up.
Edge Servers – External Node Classifier

l    No text manifest – all code, using “external node
      classifier”.
l    Assign variables and classes specific to the edge
      server through node classifier. E.g. its password, the
      services it runs.
l    In python,

          output = {}
          output[“classes”] = [ “class1”, “class2” ]
          output[“parameters”] = { “param1”: “value1” }
          print yaml.dump(output)
Edge Servers – External Node Classifier

l    This YAML-encoded structure...
      $ puppet-nodeclassifier 85206671.onappcdn.com

      classes: [base, nginx ]
      parameters: { edge_secret_key: 86zFsrM7Ma, monitoring_domain:
      monitoring.alpha.onappcdn.com }


l    … is equivalent to this textual manifest:
      node 85206671.onappcdn.com {
        $edge_secret_key = “86zFsrM7Ma”
        $monitoring_domain = “monitoring.alpha.onappcdn.com”
        include base
        include nginx
      }
Edge Servers Storedconfigs

l    Puppet stores facts about the edge servers into
      MySQL.
l    We make minimal use of this – for example sizing
      nginx's in-memory cache depending on the amount of
      memory it has.
l    Could probably use more e.g. set # threads based on
      cpu core count.
l    The data's always there if we ever want to query it...
Q&A

l    Questions? Comments?


l    P/S – final plug – we're hiring sysadmins!
PuppetCamp SEA 1 - Puppet Deployment  at OnApp

Más contenido relacionado

La actualidad más candente

Puppet for dummies - ZendCon 2011 Edition
Puppet for dummies - ZendCon 2011 EditionPuppet for dummies - ZendCon 2011 Edition
Puppet for dummies - ZendCon 2011 EditionJoshua Thijssen
 
How we use and deploy Varnish at Opera
How we use and deploy Varnish at OperaHow we use and deploy Varnish at Opera
How we use and deploy Varnish at OperaCosimo Streppone
 
Preparation study of_docker - (MOSG)
Preparation study of_docker  - (MOSG)Preparation study of_docker  - (MOSG)
Preparation study of_docker - (MOSG)Soshi Nemoto
 
PuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetPuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetWalter Heck
 
From Dev to DevOps - FOSDEM 2012
From Dev to DevOps - FOSDEM 2012From Dev to DevOps - FOSDEM 2012
From Dev to DevOps - FOSDEM 2012Carlos Sanchez
 
From Dev to DevOps - ApacheCON NA 2011
From Dev to DevOps - ApacheCON NA 2011From Dev to DevOps - ApacheCON NA 2011
From Dev to DevOps - ApacheCON NA 2011Carlos Sanchez
 
Puppet for Java developers - JavaZone NO 2012
Puppet for Java developers - JavaZone NO 2012Puppet for Java developers - JavaZone NO 2012
Puppet for Java developers - JavaZone NO 2012Carlos Sanchez
 
How to Develop Puppet Modules: From Source to the Forge With Zero Clicks
How to Develop Puppet Modules: From Source to the Forge With Zero ClicksHow to Develop Puppet Modules: From Source to the Forge With Zero Clicks
How to Develop Puppet Modules: From Source to the Forge With Zero ClicksCarlos Sanchez
 
From Dev to DevOps - Apache Barcamp Spain 2011
From Dev to DevOps - Apache Barcamp Spain 2011From Dev to DevOps - Apache Barcamp Spain 2011
From Dev to DevOps - Apache Barcamp Spain 2011Carlos Sanchez
 
Getting started with Ansible
Getting started with AnsibleGetting started with Ansible
Getting started with AnsibleIvan Serdyuk
 
Augeas, swiss knife resources for your puppet tree
Augeas, swiss knife resources for your puppet treeAugeas, swiss knife resources for your puppet tree
Augeas, swiss knife resources for your puppet treeJulien Pivotto
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteBram Vogelaar
 
More tips n tricks
More tips n tricksMore tips n tricks
More tips n tricksbcoca
 
Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))Michele Orselli
 
Automation with Ansible and Containers
Automation with Ansible and ContainersAutomation with Ansible and Containers
Automation with Ansible and ContainersRodolfo Carvalho
 
rake puppetexpert:create - Puppet Camp Silicon Valley 2014
rake puppetexpert:create - Puppet Camp Silicon Valley 2014rake puppetexpert:create - Puppet Camp Silicon Valley 2014
rake puppetexpert:create - Puppet Camp Silicon Valley 2014nvpuppet
 
Puppet: Eclipsecon ALM 2013
Puppet: Eclipsecon ALM 2013Puppet: Eclipsecon ALM 2013
Puppet: Eclipsecon ALM 2013grim_radical
 

La actualidad más candente (19)

Puppet for dummies - ZendCon 2011 Edition
Puppet for dummies - ZendCon 2011 EditionPuppet for dummies - ZendCon 2011 Edition
Puppet for dummies - ZendCon 2011 Edition
 
How we use and deploy Varnish at Opera
How we use and deploy Varnish at OperaHow we use and deploy Varnish at Opera
How we use and deploy Varnish at Opera
 
Preparation study of_docker - (MOSG)
Preparation study of_docker  - (MOSG)Preparation study of_docker  - (MOSG)
Preparation study of_docker - (MOSG)
 
PuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetPuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of Puppet
 
From Dev to DevOps - FOSDEM 2012
From Dev to DevOps - FOSDEM 2012From Dev to DevOps - FOSDEM 2012
From Dev to DevOps - FOSDEM 2012
 
From Dev to DevOps - ApacheCON NA 2011
From Dev to DevOps - ApacheCON NA 2011From Dev to DevOps - ApacheCON NA 2011
From Dev to DevOps - ApacheCON NA 2011
 
Puppet for Java developers - JavaZone NO 2012
Puppet for Java developers - JavaZone NO 2012Puppet for Java developers - JavaZone NO 2012
Puppet for Java developers - JavaZone NO 2012
 
How to Develop Puppet Modules: From Source to the Forge With Zero Clicks
How to Develop Puppet Modules: From Source to the Forge With Zero ClicksHow to Develop Puppet Modules: From Source to the Forge With Zero Clicks
How to Develop Puppet Modules: From Source to the Forge With Zero Clicks
 
From Dev to DevOps - Apache Barcamp Spain 2011
From Dev to DevOps - Apache Barcamp Spain 2011From Dev to DevOps - Apache Barcamp Spain 2011
From Dev to DevOps - Apache Barcamp Spain 2011
 
Getting started with Ansible
Getting started with AnsibleGetting started with Ansible
Getting started with Ansible
 
Augeas, swiss knife resources for your puppet tree
Augeas, swiss knife resources for your puppet treeAugeas, swiss knife resources for your puppet tree
Augeas, swiss knife resources for your puppet tree
 
Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
 
More tips n tricks
More tips n tricksMore tips n tricks
More tips n tricks
 
Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))Vagrant for real codemotion (moar tips! ;-))
Vagrant for real codemotion (moar tips! ;-))
 
Automation with Ansible and Containers
Automation with Ansible and ContainersAutomation with Ansible and Containers
Automation with Ansible and Containers
 
Intro to-puppet
Intro to-puppetIntro to-puppet
Intro to-puppet
 
rake puppetexpert:create - Puppet Camp Silicon Valley 2014
rake puppetexpert:create - Puppet Camp Silicon Valley 2014rake puppetexpert:create - Puppet Camp Silicon Valley 2014
rake puppetexpert:create - Puppet Camp Silicon Valley 2014
 
Puppet: Eclipsecon ALM 2013
Puppet: Eclipsecon ALM 2013Puppet: Eclipsecon ALM 2013
Puppet: Eclipsecon ALM 2013
 
Anatomy of a reusable module
Anatomy of a reusable moduleAnatomy of a reusable module
Anatomy of a reusable module
 

Destacado

PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & HadoopPuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & HadoopOlinData
 
PuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetPuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetOlinData
 
PuppetCamp SEA 1 - The State of Puppet
PuppetCamp SEA 1 - The State of PuppetPuppetCamp SEA 1 - The State of Puppet
PuppetCamp SEA 1 - The State of PuppetOlinData
 
PuppetCamp SEA 1 - Version Control with Puppet
PuppetCamp SEA 1 - Version Control with PuppetPuppetCamp SEA 1 - Version Control with Puppet
PuppetCamp SEA 1 - Version Control with PuppetOlinData
 
PuppetCamp SEA 1 - Puppet & FreeBSD
PuppetCamp SEA 1 - Puppet & FreeBSDPuppetCamp SEA 1 - Puppet & FreeBSD
PuppetCamp SEA 1 - Puppet & FreeBSDOlinData
 
Cloud Hosting: Lessons from the trenches
Cloud Hosting: Lessons from the trenchesCloud Hosting: Lessons from the trenches
Cloud Hosting: Lessons from the trenchesnullmind
 
OnApp Customer Webinar - April 14 2011
OnApp Customer Webinar - April 14 2011OnApp Customer Webinar - April 14 2011
OnApp Customer Webinar - April 14 2011nullmind
 

Destacado (8)

On app Carlos Rego
On app Carlos RegoOn app Carlos Rego
On app Carlos Rego
 
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & HadoopPuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
PuppetCamp SEA 1 - Using Vagrant, Puppet, Testing & Hadoop
 
PuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetPuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of Puppet
 
PuppetCamp SEA 1 - The State of Puppet
PuppetCamp SEA 1 - The State of PuppetPuppetCamp SEA 1 - The State of Puppet
PuppetCamp SEA 1 - The State of Puppet
 
PuppetCamp SEA 1 - Version Control with Puppet
PuppetCamp SEA 1 - Version Control with PuppetPuppetCamp SEA 1 - Version Control with Puppet
PuppetCamp SEA 1 - Version Control with Puppet
 
PuppetCamp SEA 1 - Puppet & FreeBSD
PuppetCamp SEA 1 - Puppet & FreeBSDPuppetCamp SEA 1 - Puppet & FreeBSD
PuppetCamp SEA 1 - Puppet & FreeBSD
 
Cloud Hosting: Lessons from the trenches
Cloud Hosting: Lessons from the trenchesCloud Hosting: Lessons from the trenches
Cloud Hosting: Lessons from the trenches
 
OnApp Customer Webinar - April 14 2011
OnApp Customer Webinar - April 14 2011OnApp Customer Webinar - April 14 2011
OnApp Customer Webinar - April 14 2011
 

Similar a PuppetCamp SEA 1 - Puppet Deployment at OnApp

Using puppet
Using puppetUsing puppet
Using puppetAlex Su
 
Installaling Puppet Master and Agent
Installaling Puppet Master and AgentInstallaling Puppet Master and Agent
Installaling Puppet Master and AgentRanjit Avasarala
 
One click deployment
One click deploymentOne click deployment
One click deploymentAlex Su
 
#OktoCampus - Workshop : An introduction to Ansible
#OktoCampus - Workshop : An introduction to Ansible#OktoCampus - Workshop : An introduction to Ansible
#OktoCampus - Workshop : An introduction to AnsibleCédric Delgehier
 
From Dev to DevOps - Codemotion ES 2012
From Dev to DevOps - Codemotion ES 2012From Dev to DevOps - Codemotion ES 2012
From Dev to DevOps - Codemotion ES 2012Carlos Sanchez
 
Developing IT infrastructures with Puppet
Developing IT infrastructures with PuppetDeveloping IT infrastructures with Puppet
Developing IT infrastructures with PuppetAlessandro Franceschi
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrapeSharad Aggarwal
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby TeamArto Artnik
 
From Dev to DevOps
From Dev to DevOpsFrom Dev to DevOps
From Dev to DevOpsAgile Spain
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to KubernetesPaul Czarkowski
 
Virtualization and automation of library software/machines + Puppet
Virtualization and automation of library software/machines + PuppetVirtualization and automation of library software/machines + Puppet
Virtualization and automation of library software/machines + PuppetOmar Reygaert
 
How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...
How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...
How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...Puppet
 
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Puppet
 
Building Docker images with Puppet
Building Docker images with PuppetBuilding Docker images with Puppet
Building Docker images with PuppetNick Jones
 
Node.js basics
Node.js basicsNode.js basics
Node.js basicsBen Lin
 
Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)
Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)
Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)Simon Boulet
 
Lean Php Presentation
Lean Php PresentationLean Php Presentation
Lean Php PresentationAlan Pinstein
 

Similar a PuppetCamp SEA 1 - Puppet Deployment at OnApp (20)

Using puppet
Using puppetUsing puppet
Using puppet
 
Installaling Puppet Master and Agent
Installaling Puppet Master and AgentInstallaling Puppet Master and Agent
Installaling Puppet Master and Agent
 
One click deployment
One click deploymentOne click deployment
One click deployment
 
#OktoCampus - Workshop : An introduction to Ansible
#OktoCampus - Workshop : An introduction to Ansible#OktoCampus - Workshop : An introduction to Ansible
#OktoCampus - Workshop : An introduction to Ansible
 
From Dev to DevOps - Codemotion ES 2012
From Dev to DevOps - Codemotion ES 2012From Dev to DevOps - Codemotion ES 2012
From Dev to DevOps - Codemotion ES 2012
 
Developing IT infrastructures with Puppet
Developing IT infrastructures with PuppetDeveloping IT infrastructures with Puppet
Developing IT infrastructures with Puppet
 
Puppet
PuppetPuppet
Puppet
 
Puppet quick start guide
Puppet quick start guidePuppet quick start guide
Puppet quick start guide
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrape
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby Team
 
DevOps Braga #6
DevOps Braga #6DevOps Braga #6
DevOps Braga #6
 
From Dev to DevOps
From Dev to DevOpsFrom Dev to DevOps
From Dev to DevOps
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to Kubernetes
 
Virtualization and automation of library software/machines + Puppet
Virtualization and automation of library software/machines + PuppetVirtualization and automation of library software/machines + Puppet
Virtualization and automation of library software/machines + Puppet
 
How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...
How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...
How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf...
 
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
 
Building Docker images with Puppet
Building Docker images with PuppetBuilding Docker images with Puppet
Building Docker images with Puppet
 
Node.js basics
Node.js basicsNode.js basics
Node.js basics
 
Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)
Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)
Deploying with Super Cow Powers (Hosting your own APT repository with reprepro)
 
Lean Php Presentation
Lean Php PresentationLean Php Presentation
Lean Php Presentation
 

Más de OlinData

AWS Cost Control: Cloud Custodian
AWS Cost Control: Cloud CustodianAWS Cost Control: Cloud Custodian
AWS Cost Control: Cloud CustodianOlinData
 
Introduction to 2FA on AWS
Introduction to 2FA on AWSIntroduction to 2FA on AWS
Introduction to 2FA on AWSOlinData
 
AWS Data Migration case study: from tapes to Glacier
AWS Data Migration case study: from tapes to GlacierAWS Data Migration case study: from tapes to Glacier
AWS Data Migration case study: from tapes to GlacierOlinData
 
Issuing temporary credentials for my sql using hashicorp vault
Issuing temporary credentials for my sql using hashicorp vaultIssuing temporary credentials for my sql using hashicorp vault
Issuing temporary credentials for my sql using hashicorp vaultOlinData
 
Log monitoring with Logstash and Icinga
Log monitoring with Logstash and IcingaLog monitoring with Logstash and Icinga
Log monitoring with Logstash and IcingaOlinData
 
FOSDEM 2017: GitLab CI
FOSDEM 2017:  GitLab CIFOSDEM 2017:  GitLab CI
FOSDEM 2017: GitLab CIOlinData
 
Cfgmgmtcamp 2017 docker is the new tarball
Cfgmgmtcamp 2017  docker is the new tarballCfgmgmtcamp 2017  docker is the new tarball
Cfgmgmtcamp 2017 docker is the new tarballOlinData
 
Icinga 2 and Puppet - Automate Monitoring
Icinga 2 and Puppet - Automate MonitoringIcinga 2 and Puppet - Automate Monitoring
Icinga 2 and Puppet - Automate MonitoringOlinData
 
Webinar - Auto-deploy Puppet Enterprise: Vagrant and Oscar
Webinar - Auto-deploy Puppet Enterprise: Vagrant and OscarWebinar - Auto-deploy Puppet Enterprise: Vagrant and Oscar
Webinar - Auto-deploy Puppet Enterprise: Vagrant and OscarOlinData
 
Webinar - High Availability and Distributed Monitoring with Icinga2
Webinar - High Availability and Distributed Monitoring with Icinga2Webinar - High Availability and Distributed Monitoring with Icinga2
Webinar - High Availability and Distributed Monitoring with Icinga2OlinData
 
Webinar - Windows Application Management with Puppet
Webinar - Windows Application Management with PuppetWebinar - Windows Application Management with Puppet
Webinar - Windows Application Management with PuppetOlinData
 
Webinar - Continuous Integration with GitLab
Webinar - Continuous Integration with GitLabWebinar - Continuous Integration with GitLab
Webinar - Continuous Integration with GitLabOlinData
 
Webinar - Centralising syslogs with the new beats, logstash and elasticsearch
Webinar - Centralising syslogs with the new beats, logstash and elasticsearchWebinar - Centralising syslogs with the new beats, logstash and elasticsearch
Webinar - Centralising syslogs with the new beats, logstash and elasticsearchOlinData
 
Icinga 2 and puppet: automate monitoring
Icinga 2 and puppet: automate monitoringIcinga 2 and puppet: automate monitoring
Icinga 2 and puppet: automate monitoringOlinData
 
Webinar - Project Management for DevOps
Webinar - Project Management for DevOpsWebinar - Project Management for DevOps
Webinar - Project Management for DevOpsOlinData
 
Using puppet in a traditional enterprise
Using puppet in a traditional enterpriseUsing puppet in a traditional enterprise
Using puppet in a traditional enterpriseOlinData
 
Webinar - PuppetDB
Webinar - PuppetDBWebinar - PuppetDB
Webinar - PuppetDBOlinData
 
Webinar - Scaling your Puppet infrastructure
Webinar - Scaling your Puppet infrastructureWebinar - Scaling your Puppet infrastructure
Webinar - Scaling your Puppet infrastructureOlinData
 
Webinar - Managing your Docker containers and AWS cloud with Puppet
Webinar - Managing your Docker containers and AWS cloud with PuppetWebinar - Managing your Docker containers and AWS cloud with Puppet
Webinar - Managing your Docker containers and AWS cloud with PuppetOlinData
 
Webinar - Manage user, groups, packages in windows using puppet
Webinar - Manage user, groups, packages in windows using puppetWebinar - Manage user, groups, packages in windows using puppet
Webinar - Manage user, groups, packages in windows using puppetOlinData
 

Más de OlinData (20)

AWS Cost Control: Cloud Custodian
AWS Cost Control: Cloud CustodianAWS Cost Control: Cloud Custodian
AWS Cost Control: Cloud Custodian
 
Introduction to 2FA on AWS
Introduction to 2FA on AWSIntroduction to 2FA on AWS
Introduction to 2FA on AWS
 
AWS Data Migration case study: from tapes to Glacier
AWS Data Migration case study: from tapes to GlacierAWS Data Migration case study: from tapes to Glacier
AWS Data Migration case study: from tapes to Glacier
 
Issuing temporary credentials for my sql using hashicorp vault
Issuing temporary credentials for my sql using hashicorp vaultIssuing temporary credentials for my sql using hashicorp vault
Issuing temporary credentials for my sql using hashicorp vault
 
Log monitoring with Logstash and Icinga
Log monitoring with Logstash and IcingaLog monitoring with Logstash and Icinga
Log monitoring with Logstash and Icinga
 
FOSDEM 2017: GitLab CI
FOSDEM 2017:  GitLab CIFOSDEM 2017:  GitLab CI
FOSDEM 2017: GitLab CI
 
Cfgmgmtcamp 2017 docker is the new tarball
Cfgmgmtcamp 2017  docker is the new tarballCfgmgmtcamp 2017  docker is the new tarball
Cfgmgmtcamp 2017 docker is the new tarball
 
Icinga 2 and Puppet - Automate Monitoring
Icinga 2 and Puppet - Automate MonitoringIcinga 2 and Puppet - Automate Monitoring
Icinga 2 and Puppet - Automate Monitoring
 
Webinar - Auto-deploy Puppet Enterprise: Vagrant and Oscar
Webinar - Auto-deploy Puppet Enterprise: Vagrant and OscarWebinar - Auto-deploy Puppet Enterprise: Vagrant and Oscar
Webinar - Auto-deploy Puppet Enterprise: Vagrant and Oscar
 
Webinar - High Availability and Distributed Monitoring with Icinga2
Webinar - High Availability and Distributed Monitoring with Icinga2Webinar - High Availability and Distributed Monitoring with Icinga2
Webinar - High Availability and Distributed Monitoring with Icinga2
 
Webinar - Windows Application Management with Puppet
Webinar - Windows Application Management with PuppetWebinar - Windows Application Management with Puppet
Webinar - Windows Application Management with Puppet
 
Webinar - Continuous Integration with GitLab
Webinar - Continuous Integration with GitLabWebinar - Continuous Integration with GitLab
Webinar - Continuous Integration with GitLab
 
Webinar - Centralising syslogs with the new beats, logstash and elasticsearch
Webinar - Centralising syslogs with the new beats, logstash and elasticsearchWebinar - Centralising syslogs with the new beats, logstash and elasticsearch
Webinar - Centralising syslogs with the new beats, logstash and elasticsearch
 
Icinga 2 and puppet: automate monitoring
Icinga 2 and puppet: automate monitoringIcinga 2 and puppet: automate monitoring
Icinga 2 and puppet: automate monitoring
 
Webinar - Project Management for DevOps
Webinar - Project Management for DevOpsWebinar - Project Management for DevOps
Webinar - Project Management for DevOps
 
Using puppet in a traditional enterprise
Using puppet in a traditional enterpriseUsing puppet in a traditional enterprise
Using puppet in a traditional enterprise
 
Webinar - PuppetDB
Webinar - PuppetDBWebinar - PuppetDB
Webinar - PuppetDB
 
Webinar - Scaling your Puppet infrastructure
Webinar - Scaling your Puppet infrastructureWebinar - Scaling your Puppet infrastructure
Webinar - Scaling your Puppet infrastructure
 
Webinar - Managing your Docker containers and AWS cloud with Puppet
Webinar - Managing your Docker containers and AWS cloud with PuppetWebinar - Managing your Docker containers and AWS cloud with Puppet
Webinar - Managing your Docker containers and AWS cloud with Puppet
 
Webinar - Manage user, groups, packages in windows using puppet
Webinar - Manage user, groups, packages in windows using puppetWebinar - Manage user, groups, packages in windows using puppet
Webinar - Manage user, groups, packages in windows using puppet
 

Último

Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 

Último (20)

Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 

PuppetCamp SEA 1 - Puppet Deployment at OnApp

  • 1. Puppet Deployment at OnApp Wai Keen Woon CTO, CDN Division waikeen.woon@onapp.com
  • 3. About OnApp A leading provider of software for hosts The leading cloud The instant global CDN for hosts management software for hosts OnApp launched July 1st 2010 Deep industry knowledge Backed by LDC 100+ employees in US, EU, APAC
  • 4. Vital Statistics 1 in 3 public clouds 800+ cloud deployments 300+ global clients
  • 6. Instant CDN that gives you… 75+ PoPs low cost, high margin get paid for idle capacity
  • 7. OK. </ObligatoryPlug>
  • 8. Systems Overview l  Core & Development l  ~20 physical servers l  ~200 VMs l  Homogeneous environment – 64-bit Debian everywhere l  Mainly use OpenVZ and KVM for virtualization l  CDN Delivery Edge Servers l  100+ servers in 60+ cities l  Running on the OnApp platform – either Xen or KVM l  Puppet integral to our setup – since day 1
  • 9. Why Puppet? l  More reliable configuration of servers. Less need to “run ssh in a for loop” and miss out something. l  Self-documenting – our manifests are almost able to bootstrap an empty server. l  Our manifests can't bootstrap an empty environment yet. l  Limitation – manifests describe what/where/how something is setup, but doesn't describe *why*. l  Nice syntax – easy on the eyes. Comprehensive builtin resource types. Able to fallback to dumb ways of doing things if required (use file, exec et al).
  • 10. Core Infra Environments l  Systems manifest describes everything. l  Three environments: β
  • 11. What Would OnApp Setup... l  Essential utilities (tcpdump, less, vim, etc). l  Users & their SSH keys, sudoers. l  Developer's shell => /bin/false if production l  Base firewall rules. l  Nagios agent. l  Set uniform locality settings: UTC timezone, en_US.UTF-8 locale. l  SMTP that smarthosts to our central relay. l  Syslogd for remote logs to central logging server. l  Finally, the services.
  • 12. Core Infra Manifest Excerpt $portal_domain = "portal.alpha.onappcdn.com" node "monitoring.alpha.onappcdn.com" { $portal_db_host = "portal.alpha.onappcdn.com" include base $portal_db_user = "aflexi_webportal" include s_db_monitoring include s_monitoring_server $auth_nameservers = { "ns1" => "175.143.72.214", include collectd::rrdcached "ns2" => "175.143.72.214", include s_munin "ns3" => "175.143.72.214", include s_monitoring_alerts "ns4" => "175.143.72.214", include s_monitoring_graph } } $monitoring_host_server = class collectd::rrdcached { package { "rrdcached": [ "monitoring.alpha.onappcdn.com", "dns.alpha.onappcdn.com" ] ensure => latest, } service { "rrdcached": BLUE – env config definitions ensure => running, RED – node definitions } GREEN – class definitions }
  • 13. Package Repo Integration l  Jenkins builds debs of our code and stores it into an apt repository for the environment it is built for. l  Puppet keeps packages up-to-date (ensure => latest) and restarts services on package upgrades. Puppet-agent[25431]: (/Stage[main]/Debian/Exec[apt-get-update]/returns) executed successfully puppet-agent[25431]: (/Stage[main]/Python::Aflexi::Mq/Package[python-aflexi-mqcore]/ ensure) ensure changed '7065.20120530.113915-1' to '7066.20120604.090916-1' puppet-agent[25431]: (/Stage[main]/S_mq/Service[worker-rabbitmq]) Triggered 'refresh' from 1 events puppet-agent[25431]: Finished catalog run in 16.08 seconds
  • 14. Nagios Integration l  Plugs into nagios – uses “exported resources”
  • 15. Nagios Integration Server manifest Nagios service manifest *collects the resources to check *exports the service that is checked @@nagios_service { "check_load_$fqdn": Nagios_service <<| tag == "onappcdn.cm" |>> { check_command => target => "/etc/n3/conf.d/services.cfg", "check_nrpe_1arg!check_load", require => Package["nagios3"], use => "generic-service", notify => Exec["reload-nagios"], host_name => $fqdn, } service_description => "check_load", tag => $domain, }
  • 16. Nagios Integration l  What's logged on the nagios server when puppet runs? puppet-agent[15293]: (/Stage[main]/Nagios::Monitor_private/ Nagios_host[hrm.onappcdn.com]/ensure) created puppet-agent[15293]: (/Stage[main]/Nagios::Monitor_private/ Nagios_service[check_load_hrm.onappcdn.com]/ensure) created nagios3: Nagios 3.2.1 starting... (PID=5601) puppet-agent[15293]: (/Stage[main]/Nagios::Base/Exec[reload- nagios]) Triggered 'refresh' from 8 events
  • 17. Monitoring Puppet Itself l  Lots of tools/dashboards out there to achieve this. l  For us: “grep -i err */syslog”. Dumb, but works until we need to Really Address it. l  Common issues: l  Puppet gets “stuck”. And only one puppet instance can run at any one time. l  Manifest errors – syntax, merge issues. l  Badly-written manifests (vague dependencies, conditions/commands not robust enough). l  An important dependent resource failing (e.g. apt-get install fails due to dpkg-configure error).
  • 18. File/Dir Organization l  We use git to revision control our l  Common branch Manifests/ puppet manifests. alpha.pp beta.pp l  Style we adopted mainly comes Modules/ Base/ from Hunter Haugen* Users/ l  A branch for each environment, l  Alpha env branch Modules/ plus a “common” branch. Python/ Services/ l  Each branch checked out as a Nameserver/ separate directory in /etc/puppet/ l  Beta env branch environments/$env Modules/ Python/ l  And puppetmaster's includedir Services/ Nameserver/ configured to that directory. * - http://hunnur.com/blog/2010/10/dynamic-git-branch-puppet-environments/
  • 19. File/Dir Organization l  Common goes into its own branch – for convenience; less merging needed for manifests that we are Really Sure won't differ between environments. l  System manifest into common/manifests/$env.pp l  Initially tried putting manifest into alpha/beta/omega branches as site.pp – merge hell. l  Introduced extra variable - $effective_env l  Abstracts the puppet environment name, from the environment that the manifest runs in.
  • 20. File/Dir Organization l  Hotfixes branch off omega and merged to alpha/beta/ omega. l  Development branches off alpha l  This branch can be trialed as a separate environment (use --environment to specify custom env on puppet client). l  Merge to alpha → beta → omega. l  Or merge as feature branch to any other environment. l  “git diff branchA branchB” - differences are shown clearly between environments.
  • 21. Edge Servers l  Our edge servers are hosted on OnApp cloud (only). l  When creating an edge server, the cloud control panel l  Instantiates a VM from a lightly-customized Debian image. l  Configures the package repositories. l  Issues a puppet run to set up. l  Advantage of setting it up through puppet instead of a “gold image” - our system can be installed on bare metal if needed, can be reproducibly installed on $future_debian_release
  • 22. Edge Servers l  Our edge servers are hosted on OnApp cloud (only). l  When creating an edge server, the control panel instantiates a VM from a lightly-customized Debian image, and issues a puppet run to set it up.
  • 23. Edge Servers – External Node Classifier l  No text manifest – all code, using “external node classifier”. l  Assign variables and classes specific to the edge server through node classifier. E.g. its password, the services it runs. l  In python, output = {} output[“classes”] = [ “class1”, “class2” ] output[“parameters”] = { “param1”: “value1” } print yaml.dump(output)
  • 24. Edge Servers – External Node Classifier l  This YAML-encoded structure... $ puppet-nodeclassifier 85206671.onappcdn.com classes: [base, nginx ] parameters: { edge_secret_key: 86zFsrM7Ma, monitoring_domain: monitoring.alpha.onappcdn.com } l  … is equivalent to this textual manifest: node 85206671.onappcdn.com { $edge_secret_key = “86zFsrM7Ma” $monitoring_domain = “monitoring.alpha.onappcdn.com” include base include nginx }
  • 25. Edge Servers Storedconfigs l  Puppet stores facts about the edge servers into MySQL. l  We make minimal use of this – for example sizing nginx's in-memory cache depending on the amount of memory it has. l  Could probably use more e.g. set # threads based on cpu core count. l  The data's always there if we ever want to query it...
  • 26. Q&A l  Questions? Comments? l  P/S – final plug – we're hiring sysadmins!