Open Source Toolchains to Manage Cloud Infrastructure presentation for Cloud Computing Expo East - June 6, 2011.
Added APIs (jclouds, fog, libcloud, deltacloud)
Open Source Toolchains to Manage Cloud Infrastructure
1. Open Source Toolchains
to Manage Cloud
Infrastructure
Mark R. Hinkle
VP of Community
Cloud.com
Twitter:
@clouddotcom
@mrhinkle
Email:
mrhinkle@cloud.com
mrhinkle@gmail.com
2. 2
Introduction
• Responsible for Driving Adoption of
CloudStack Open Source Cloud Computing
Software
• Former manager of Zenoss Open Source
project 100,000 users, 1.5 million downloads
• Former Linux Desktop Advocate (Zealot?)
• Former LinuxWorld Magazine Editor-in-Chief
• Open Management Consortium Conspirator
• Open Desktop Consortium Instigator
• Author - “Windows to Linux Business Desktop
Migration” - Thomson
• NetDirector Project - Open Source
Con guration Management Project
• Sometimes Author and Blogger at
SocializedSoftware.com/NetworkWorld
• Start-up junkie, Glutton for punishment
3. 3
Agenda
• Introductions
• Cloud & Systems Management Overview
• Open Source Tools for Cloud
Management
• Creating Open Source Toolchains
• Questions (and Hopefully Answers)
5. 5
Jevon’s Paradox and the Cloud
…the efficiency with which a resource is used tends
to increase (rather than decrease) the rate of
consumption of that resource.
…in cloud computing increased
consumption can introduces signi cant
management overhead despite higher
hardware utilization and other bene ts
1865, the English economist William Stanley Jevons
8. Automation Balances the 8
Equation
• MeatCloud, Can’t Keep up with
Cloud Computing
• Devops & Agile IT Philosophy
• Script Repetitive Tasks
• Integrate Tools
• Automate, Automate, Automate
9. DevOps – Agile Management for 9
the Cloud
• Agile, fast moving
• Smaller incremental change
• Increased coordination across
groups (development/operations)
• Eye towards Automation
10. 10
Management Toolchains
Toolchain (n): Monitoring
A set of tools where Patching and
Provisioning
the output of one
tool becomes the Con guration
input of another tool
11. 11
The Myth of the Nines
Availability % Downtime per Downtime per Downtime per
Year Month Week
99.9% (three nines) 8.76 hours 43.2 minutes 10.1 minutes
99.95% 4.38 hours 21.56 minutes 5.04 minutes
99.99% (four nines) 52.6 minutes 4.32 minutes 1.01 minutes
99.999% ( ve nines) 5.26 minutes 25.9 seconds 6.05 seconds
99.9999% (six nines) 31.5 seconds 2.59 seconds .0605 seconds
Average polling interval for monitoring? 5 minutes?
Even superhuman operations people can’t be alerted and take action in under 5 minutes.
One outage per year could drop service level to three nines or worse.
14. 14
4 Types of Management Tools
Provisioning
Installation of operating systems and other software
Configuration Management
Sets the parameters for servers, can specify
installation parameters
Orchestration/Automation
Automate tasks across systems
Monitoring
Records errors and health of IT infrastructure
15. 15
Cloudy Tools
• Network Capable
• Cloud “Aware”
• Easy-to-Integrate
• Adhere to Open Standards
• Lend Themselves to Automation
16. Open Source 16
Provisioning Tools
Year Started Language License Installation
Targets
Cobbler 2007 Python GPL Red Hat,
OpenSUSE
Fedora, Debian,
Ubuntu
Fully Automatic 2000 Perl GPL Debian
Installation (FAI)
Kickstart ? Python GPL Most .dep and
RPM based Linux
distros
Spacewalk 2008 Perl, Python, Java GPL Fedora, Centos
Crowbar/Chef 2011 Ruby Apache ?
17. Open Source 17
Configuration Management Tools
Year Started Language License Client/Server
Bcfg2 2003 Python BSD Yes
Cfengine 1993 C Apache Yes
Chef 2009 Ruby Apache Chef Solo – No
Chef Server - Yes
Puppet 2004 Ruby GPL yes
18. Open Source 18
Monitoring Tools
Year Started License Language Type of Collection
Monitoring Methods
Cacti (RRDTool) 2001 GPL PHP Performance SNMP, syslog
Nagios 1999 GPL C/PHP Availability SNMP,TCP, ICMP,
IPMI, syslog
OpenNMS 2000 GPL Java Availability/ SNMP,
Performance
Zabbix 2001 GPL C/PHP Availability/ SNMP, TCP/ICMP,
Performance and IPMI, Synthetic
more Transactions
Zenoss 2005 GPL Python Availability, SNMP, ICMP, SSH,
Performance, Event syslog, WMI
Management
19. Open Source 19
Automation/Orchestration Tools
Year Language License Client/ Support
Started Server Organization
Capistrano 2006 Ruby MIT Yes None
RunDeck 2010 Java Apache Yes DTO Solutions
Func 2007 Python GPL Yes Fedora Project
MCollective 2009 Ruby Apache Yes PuppetLabs
20. 20
Cloud APIs Aren’t Created Equal
Open Source Abstractions
API Language
jclouds java
deltacloud python
Lib cloud python
Fog Ruby
23. 23
Automated Toolchain
Capistrano
Command and Application Service RunDeck
Fabric
Control Orchestration Func
Provisioning Activity
BCFG2
Con guration Cfengine
System Con guration Chef
Puppet
Cloud: OS Install:
Eucalyptus Kickstart
Cloud Image OS OpenStack Cobbler
Bootstrapping
Launch Install CloudStack Spacewalk
Abiquo
24. 24
Example: GeekNet
• Servers are automatically built using
con guration management software (Puppet)
• Discovery tool (Zenoss) nds infrastructure
and populates a CMDB then spits out Hundreds of servers,
information to scripts that translate serving web, databases,
information to BIND con gurations for DNS and other infrastructure
• Monitoring tool adds hosts to polling tool for some of the world’s
(Nagios) to check servers for availability most highly trafficked
• As infrastructure changes systems are updated websites – over 40 million
automatically visitors per month.
• Servers can be spun up and managed in
minutes, not hours automatically with little or
no human interaction
27. 27
Contact Me
Professional: mrhinkle@cloud.com
Personal: mrhinkle@gmail.com
Professional: mrhinkle@cloud.com
Personal: mrhinkle@gmail.com
Professional: http://open.cloud.com
Personal: http://socializedsoftware.com Mark R. Hinkle
VP of Community
Twitter: Open Source Enthusiast
@clouddotcom
@mrhinkle
28. Open Source Toolchains to Manage Cloud
Computing
http://www.slideshare.net/socializedsoftware
by Mark R. Hinkle is licensed under a Creative
Commons Attribution-ShareAlike 3.0 United
States License.
Notas del editor
How many users are using
IN the late 1800s English economist William Stanley Jevons noticed that despite huge increases in the efficiency of coal burning, demand actually increased rather than decreased. In cloud computing despite huge cost savings the consumption of cloud computing is and will continue to grow as demand increases. The downside is that this increased consumption has the ability to increase management burden if users don’t adopt leveraged methods for managing cloud computing.
With a credit card and a single administrator can spin up 100s of servers in minutes.
How does the management personnel for the cloud scale? …Automation.
DevOps is a popular movement among cloud operations professionals who are advocating many of the same methodologes used in Agile software development.
Ideally for the cloud you create management toolchains that automate the management of your cloud.
Cloud computing promises highly available systems, but if you have a reactive approach you won’t achieve that goal. Build redundant, highly automated systems that can self-heal.
Other disciplines like back-up, log management, performance and security (virus,intrusion detection) are important but not core to the delivery of cloud computing systems
Tools that can automatically build your systems/install systems
Configuration management tools abstract infrastructure into variables that are implemented via “recipes”Interesting new development is the PXE boot Crowbar extension for Chef Server used to provision virtual machines in OpenStack.
API lets you develop to one API and move between cloudseasilyTypesof Tasks Accomplished by an APIProvisioning (creating, re-creating, moving, or deleting components e.g. virtual machines, vlans)Configuration (assigning or changing attributes of the architecture such as security and network settings)Cloud ProvidersJclouds – java API Abstraction Libcloud – started by CloudKick (now Rackspace) to abstract clouds, Apache incubator projectDeltacloud – started by Red Hat to abstract clouds, Apache incubator projectFog – started by EngineYard is a ruby gem that allows you to interact with number of cloud API.
Java applications are developed and checked into Source Control Manager (SCM)Applications built and dependencies resolved via MavenBuilt and unit-tested by HudsonChanges recorded in Change Management Database (CMDB)Deployed by ControlTier (RunDeck)