Presentation given by Greg Althaus & Rob Hirschfeld @ SUSEcon 2012 about Dell Crowbar, DevOps and Cloud Deployment.

1. Crowbar:
The Cloud Operations
Framework
Greg Althaus
Dell Principal Engineer and Crowbar Co-Founder

2. Agenda
• What Led to Crowbar?
• What is Crowbar?
• Crowbar Architecture
• SUSE + Crowbar = A Path to Clouds
• Crowbar 2.0 and Beyond
• Q&A
2 Revolutionary Cloud & Big Data Solutions

3. What Led to
Crowbar?
3 Revolutionary Cloud & Big Data Solutions

4. Crowbar <3 OpenStack
“Boxes to working OpenStack in <4 hours”
• Crowbar’s birthright was “an OpenStack installer”
– But we started it knowing there was a much bigger problem
• Like many open source projects, OpenStack is complex and changes
constantly
– Want the innovation and afraid of change
• To create an OpenStack project, needed:
– To get customers running quickly
– Have a way to cope with frequent updates
– Connect a lot of parts together
– Be part of the open source community
• Got & get substantial uplift from
the OpenStack community
4 Confidential Revolutionary Cloud & Big Data Solutions

5. Problem: Clouds do not operate themselves!
Operations
SW
Ops
Servers HW
Networks
Storage
Etc. Expertise
5 Confidential Revolutionary Cloud & Big Data Solutions

6. Ops automation (“devops”) jumpstarts learning,
reduces complexity and manages scale
Operations
Automation
(DevOps)
SW
Servers HW
Networks
Storage Ops
Etc.
Operations
Expertise
6 Confidential Revolutionary Cloud & Big Data Solutions

7. What is DevOps? Why is it important?
• DevOps extended to infrastructure is “CloudOps”
• Operational Model for Software
– “Operations as Code”
– Highly automated
– Layered deployments (no image based)
– Continuous deployment
• Shaped by Amazon Cloud
– API only access to infrastructure
– Elastic resources that are easier to create and destroy than change
• Configuration Management Tooling
– Chef & Puppet
– Builds applications from open source modules/recipes
– Not quite coding, but far from manual
7 Confidential Revolutionary Cloud & Big Data Solutions

8. OpenStack has a lot of interlocking parts!
8 Confidential Revolutionary Cloud & Big Data Solutions

9. …and it matters how they are deployed!
Cookbooks
9 Confidential Revolutionary Cloud & Big Data Solutions

10. Images vs. Layers: Lifecycle
Images: Replacement Layers: Upgrade
Config Config Config
I I
Foo Foo
Config
Config
I+A+U+ I+A+U+ I+A+U+
Bar v1 Bar v2
O/S O/S O/S
U U
OS OS
Config Bar v2
I+A+U+
O/S
10 Confidential Revolutionary Cloud & Big Data Solutions

11. CloudOps for OpenStack
There are lot of pieces, but they share many elements!
I I I I I I
Crowbar Dash Nova Nova Nova
Config
Config
Config
Config
Config
Config
Nagios
Chef Board Cntrl DB API
U U U U U U
OS OS OS OS OS OS
I I I I I I
Nova Nova Nova Nova Nova Nova
Config
Config
Config
Config
Config
Config
Node Node Node Node Node Node
U U U U U U
OS OS OS OS OS OS
11 Confidential Revolutionary Cloud & Big Data Solutions

12. CloudOps for OpenStack
Plus interconnections are highly complex and dynamic
I I I I I I
Crowbar Dash Nova Nova Nova
Config
Config
Config
Config
Config
Config
Nagios
Chef Board Cntrl DB API
U U U U U U
OS OS OS OS OS OS
I I I I I I
Nova Nova Nova Nova Nova Nova
Config
Config
Config
Config
Config
Config
Node Node Node Node Node Node
U U U U U U
OS OS OS OS OS OS
12 Confidential Revolutionary Cloud & Big Data Solutions

13. What is
Crowbar?
13 Revolutionary Cloud & Big Data Solutions

14. Crowbar, the Cloud Deployer++
• What is Crowbar
– Not only reduces the effort for deploying clouds
– Also provides an operational model for managing them
• Field proven technologies
– Build on locally deployed Chef Server
– Raw servers to full cloud in <2 hours, multi-node from the ground up
– Hardened with over a year of deployments
• Apache 2 open source (on github/dellcloudedge)
– Multi-apps (OpenStack & Hadoop), multi-OS (Ubuntu, RHEL, CentOS, SUSE),
Development “in the open” w/ community engagement
• Hadoop & OpenStack releases (dell.com/OpenStack)
– Cactus, Diablo and Essex!
• Open version is not Dell hardware Limited
– This is been an important factor in adoption
14 Confidential Revolutionary Cloud & Big Data Solutions

15. Crowbar in the Open (Community!)
• Community
– Focused on OpenStack, expanding in other areas
– Engineering benefits from community test and dev
– Dell engineering active participants
• Developed in the Open
– Most code development is current & live on github
– Needed for collaboration
– Allows faster response to customers and partners
• Active Community
– 300+ email list users & nearly 500 Github followers
– 500+ downloads of the Crowbar image
– Partners: Intel, Opscode, SUSE, etc
– Customers: telecom, cloud, financial, etc
15 Confidential Revolutionary Cloud & Big Data Solutions

16. Cloud Taxonomy
Software as a Service IT as a Service Platform as a Service Infrastructure as a Service Everything as a Service
Admin Software
Web Services & APIs Self Service Portal
SSO
Customer Management
Entitlement, rights Billing Metering
Information Service Management
LDAP/AD
Infrastructure Software
Workload Lifecycle
Legacy Management
Ser Gov/Workflow Automation Monitoring
Management
Orchestration Intelligent Resource Manager Platform Provisioning
Reporting
Analytics
Security
Abstraction Software
OS Data Store Operating System Application Run-Time
Hardware Virtualization Virtualization Virtualization
IPS
Physical
Compute Switch Storage
Firewall
Network
HVAC Power Facility
Overarching Environmentals
Revolutionary Cloud & Big Data Solutions
Systems

17. Crowbar Builds the Cloud Taxonomy
Software as a Service IT as a Service Platform as a Service Infrastructure as a Service Everything as a Service
Admin Software
OpenStack APIs OpenStack Dashboard
SSO
Customer Management
Entitlement, rights Billing Metering
Information Service Management
LDAP/AD
Infrastructure Software
Workload Lifecycle
Legacy Management
Ser Gov/Workflow Automation Monitoring
Management
Orchestration Intelligent Resource Manager Crowbar
Reporting
Analytics
Keystone
Abstraction Software
OS Data Store Operating System Application Run-Time
HardwareKVM
Virtualization Virtualization Virtualization
IPS
Physical
Nova Switch
Glance Swift
Firewall
Quantum
BIOS RAID NETWORK
Overarching Environmentals
Revolutionary Cloud & Big Data Solutions
Systems

18. Crowbar Builds Up Clouds in layers
APIs, User Access, &
Ecosystem Partners
Ops Management
Dell “Crowbar”
Cloud Infrastructure
Core Components &
Operating Systems
Physical Resources
18 Confidential Revolutionary Cloud & Big Data Solutions

19. 1) Hardware Configuration
APIs, User Access, &
Ecosystem Partners
Ops Management
Dell “Crowbar”
Cloud Infrastructure
Core Components &
Operating Systems
Physical Resources
19 Confidential Revolutionary Cloud & Big Data Solutions

20. 2) OS and Ops Infrastructure
APIs, User Access, &
Ecosystem Partners
Ops Management
Dell “Crowbar”
Cloud Infrastructure
Core Components &
Operating Systems
Physical Resources
20 Confidential Revolutionary Cloud & Big Data Solutions

21. 3) Cloud Platform Integration
APIs, User Access, &
Ecosystem Partners
Ops Management
Dell “Crowbar”
Cloud Infrastructure
Core Components &
Operating Systems
Physical Resources
21 Confidential Revolutionary Cloud & Big Data Solutions

22. 4) Cloud App Management
APIs, User Access, &
Ecosystem Partners
Ops Management
Dell “Crowbar”
Cloud Infrastructure
Core Components &
Operating Systems
Physical Resources
22 Confidential Revolutionary Cloud & Big Data Solutions

23. Late Binding Matters!
It’s “Just-in-time” Deployment
Late binding is a critical component of Crowbar flexiblity
• Integrated configuration of RAID, BIOS,
Allocate with System View
provision & networking, so no longer have
to do that work in advance.
• Do NOT need to make these decisions
early in the process
Inventory Provision
• Can adapt and change over time.
Discover Deploy Layer N User App & Ops
decisions drive the entire
Network Config
config & deploy process!
23 Confidential Revolutionary Cloud & Big Data Solutions

24. Crowbar
Architecture
+
Demo
24 Revolutionary Cloud & Big Data Solutions

25. OpenStack Orchestration: Chef & Crowbar
Orchestration
State Machine
OpenStack Components
Crowbar Magic Happens!
Ops Management
Dell “Crowbar”
Run List
Network Configuration
Chef
Server
Operating System
Config.
Hardware Config
(uses “Sledgehammer”)
DHCP
Server(s)
25 Confidential Revolutionary Cloud & Big Data Solutions

26. Dell Cloud Installer: Crowbar Barclamps
APIs, User Access, & Nagios Ganglia Dashboard
Ecosystem Partners
enStratus Cloudera Horizon
Ops Management
Nova Swift Glance
Dell “Crowbar”
Cloud Infrastructure
& Dell IP Extensions
Block Hadoop Keystone
Core Components &
Crowbar DNS Logging
Operating Systems
Deployer NTP
Provisioner
BIOS IPMI
Physical Resources
Network RAID
* Crowbar has potential to service other programs beyond OpenStack
26 Confidential Revolutionary Cloud & Big Data Solutions

27. What is a Barclamp?
Barclamp
Crowbar API O/S App Components
Chef Recipes
& Partial UI Packages & Scripts
Ops Management
• Crowbar API & Partial UI
Dell “Crowbar”
• Adds states and transitions to orchestration
• Customizes UI specific to barclamp
• Visual extensions to nodes/networks/utils
• Chef Recipes
• DevOps description of application
• Enforced repeatable configuration
• O/A App Packages
• Operating system dependencies (Debs & RPMs)
• Components & Scripts
• Ad hoc configuration not covered by Chef
27 Confidential Revolutionary Cloud & Big Data Solutions

28. What is a Barclamp?
Barclamp
Crowbar API O/S App Components
Chef Recipes
& Partial UI Packages & Scripts
Ops Management
• Interact as service layers
Dell “Crowbar”
• Have dependencies and pre-requisites
• Can operate on multiple operating systems
• Have independent version streams
• Leverage/reuse Chef cookbooks
Barclamp
Barclamp
28 Confidential Revolutionary Cloud & Big Data Solutions

29. Crowbar’s Proposal Workflow
• Proposals manage deployments
• Proposals are a core design objective to ensure customer flexibility
and system level management
• Workflow:
• Templates are default configurations without resources
• Proposals all users to choose config and nodes (resources)
• Applying a proposal tells Crowbar to “make it so”
Discovered
Nodes
Proposal Nodes Apply Active
Barclamp
Deployed
Services
Proposal Nodes Apply Active
29 Confidential Revolutionary Cloud & Big Data Solutions

30. Crowbar Orchestration = State Machine
30 Confidential Revolutionary Cloud & Big Data Solutions

31. Networking in Crowbar IP
Allocations
Network Barclamp
DHCP
Chef Client Crowbar
IMPI Discovers
VLAN Switches
VLAN
and Ports
eth0
VLAN VLAN
eth1 Crowbar
Teamed Teamed Does NOT
Configure
eth#
The Switch!
not O/S Only*
mgm’d
Server * For now
31 Confidential Revolutionary Cloud & Big Data Solutions

32. Networking in Crowbar IP
Allocations
Network Barclamp
DHCP
Other
Chef Client Barclamps
IMPI Request
VLAN
Conduit “Conduits”
VLAN
1g1
eth0
VLAN VLAN
Conduit
Interface &
1g2
eth1 Conduits
Teamed Teamed
Conduit Normalize
1g3
eth# Networks
Between
not Nodes
mgm’d
Server
32 Confidential Revolutionary Cloud & Big Data Solutions

33. SUSE + Crowbar
=
A Path to Clouds
33 Revolutionary Cloud & Big Data Solutions

34. SUSE Cloud with Crowbar
• Version of Crowbar that runs on SUSE
• Packaged to take advantage of SUSE
repositories and methods
• Provides OpenStack Essex
• Extends and builds SUSE Studio
• Released and available as of the beginning
of September
34 Confidential Revolutionary Cloud & Big Data Solutions

35. Team Work
• A huge key to the success
has been team work
• Joint development
• Design and planning
discussion
• Not just limited to SUSE, but
definite leader
35 Confidential Revolutionary Cloud & Big Data Solutions

36. Crowbar 2.0
and
Beyond
36 Confidential Revolutionary Cloud & Big Data Solutions

37. Crowbar 2.0 (currently in progress)
http://bit.ly/crowbar2
• Simplify use of Chef, eliminate Crowbar requirements in Chef recipes
– Reduce the initial effort required to leverage Crowbar
– Opens Crowbar to a broader audience (see Upstreaming)
• Provide heterogeneous/multiple operating system deployments to
enable:
– Multiple versions of the same OS running for upgrades
– Different OS operating simultaneously (and deal with heterogeneous
packaging issues)
– Accommodation of no-agent systems like locked systems (e.g.: virtualization
hosts) and switches (aka external entities)
– UEFI booting in Sledgehammer
• Strengthen networking abstractions
– Allow networking configurations to be created dynamically (so that users are
not locked into choices made before Crowbar deployment)
– Better manage connected operations
– Enable pull-from-source deployments that are ahead of
(or forked from) available packages
37 Confidential Revolutionary Cloud & Big Data Solutions

38. Crowbar 2.0 (currently in progress)
http://bit.ly/crowbar2
• Improve Crowbar's core database and state machine to enable
– Larger scale concerns
– Controlled production migrations and upgrades
• Packaging rework
– Lead by some of the SUSE work
– Create distribution-based packages
• Upgrade components
– Start the process of enabling security updates
– Staged Updates from test to production environments
– Start considering application
• Other important items
– Make documentation more coupled to current features and easier to
maintain
– Upgrade to Rails 3 to simplify code base, security and performance
– Deepen automated test coverage and capabilities
38 Confidential Revolutionary Cloud & Big Data Solutions

39. Crowbar 2.0 (currently in progress)
http://crowbar.sync.in/crowbar2
• Network models (http://youtu.be/eTSTLecNlGM ) -
http://crowbar.sync.in/crowbar2-network
• Online mode (http://youtu.be/U05apbOqq0k ) -
http://crowbar.sync.in/crowbar2-online
• Pull from source (http://youtu.be/Bo4V6Dax3LU ) -
http://crowbar.sync.in/crowbar2-pullsource
• Heterogeneous OS (http://youtu.be/xL-2eefqH8s) -
http://crowbar.sync.in/crowbar2-multi-os
• Attribute driven recipes (http://youtu.be/wtUNJfrHU1Q) -
http://crowbar.sync.in/crowbar2-attributeconfig
39 Confidential Revolutionary Cloud & Big Data Solutions

40. Q&A
40 Confidential Revolutionary Cloud & Big Data Solutions

41. Thanks for
Listening!
• Greg Althaus
Gregory_Althaus@dell.com
Twitter: @galthaus
41 Confidential Revolutionary Cloud & Big Data Solutions