This was co-presented at the OpenStack Summit 2013 in Portland by Kamesh Pemmaraju, Product Manager from Dell and Neil Levine Inktank. Inktank Ceph is a transformational open source storage solution fully integrated into OpenStack providing scalable object and block storage (via Cinder) using commodity servers. The Ceph solution is resilient to failures, uses storage efficiently, and performs well under a variety of VM Workloads. Dell Crowbar is an open source software framework that can automatically deploy Ceph and OpenStack on bare metal servers in a matter of hours. The Ceph team worked with Dell to create a Ceph barclamp (a crowbar extention) that integrates Glance, Cinder, and Nova-Volume. As a result, it is lot faster and easier to install, configure, and manage a sizable OpenStack and Ceph cluster that is tightly integrated and cost- optimized. Hear how OpenStack users can address their storage deployment challenges: Considerations when selecting a cloud storage system Overview of the Ceph architecture with unique features and benefits Overview of Dell Crowbar and how it can automate and simplify Ceph/OpenStack deployments Best practices in deploying cloud storage with Ceph and OpenStack

1. Wicked Easy Ceph Block
Storage & OpenStack
Deployment with
Crowbar
Kamesh Pemmaraju, Dell
Neil Levine, Inktank

2. Outline
• Introduction
– Dell OpenStack-Powered Cloud Solution
– Ceph
– Crowbar
• Why Crowbar + Ceph in OpenStack?
– OpenStack Block Storage gap
– Automation, scale
• What we have done to enable things?
• Customer benefits, drivers – an example
• What’s coming next?

3. • Clouds demand significant operational
and process controls
• Operational decisions drive hardware
and software decisions
• We are finding ways to productize
operations into best practices
HW
SW
OPS
Clouds Require an Operational Focus

4. OPS
SW
HW
SW
OPS
“Crowbar”
CloudOps
Software
Services &
Consulting
Reference
Architecture
Dell OpenStack-Powered Cloud Solution

5. Dell OpenStack-Powered Cloud Solution
HW + SW + Services
Hardware
HW reference
architecture
• C6220, C6105,R720, R720XD servers
• Storage and compute
• Force 10 S60 and PowerConnect
Configuration • Min 6 nodes. Max 60 nodes
Software
Software
• OpenStack Installer and continuous integration (Crowbar)
• OpenStack cloud SW
• Ceph for distributed storage
Operating system
• Ubuntu 12.04 (host)
• Windows (guests)
• Linux (guests)
Hypervisor • KVM
Services
Deployment and
consulting
• Hardware integration – onsite or merge center
• Software installation – onsite
• OpenStack consulting services
− Assessment, design and installation
− Delivered via partnerships with Mirantis and Canonical
Support
• HW/Crowbar: Dell ProSupport
• OpenStack support via Canonical (coming soon)

6. OBJECTS VIRTUAL DISKS FILES & DIRECTORIES
CEPH
FILE SYSTEM
A distributed, scale-out
filesystem with POSIX
semantics that provides
storage for a legacy and
modern applications
CEPH
OBJECT STORAGE
A powerful S3- and Swift-
compatible gateway that
brings the power of the
Ceph Object Store to
modern applications
CEPH
BLOCK STORAGE
A distributed virtual block
device that delivers high-
performance, cost-effective
storage for virtual machines
and legacy applications
Ceph – Unified Storage Software

7. Ceph – Key Differentiators
• Unified storage platform (Object + Block + File)
• Ceph Intelligent Placement (CRUSH)
• Ceph Intelligent Devices (Self-healing, P2P)
• Ceph Intelligent Objects (Embedded Software Classes)
• Ceph Integration (Linux Kernel, OpenStack, Cloudstack)

8. Ceph Storage Cluster (RADOS)
A reliable, autonomous, distributed object store comprised of self-healing, self-
managing, intelligent storage nodes
LIBRADOS
A library allowing
apps to directly
access RADOS,
with support for
C, C++, Java,
Python, Ruby,
and PHP
Ceph Block
Device
(RBD)
A reliable and fully-
distributed block
device, with a Linux
kernel client and a
QEMU/KVM driver
Ceph File
System
(CephFS)
A POSIX-compliant
distributed file
system, with a Linux
kernel client and
support for FUSE
Ceph Object
Gateway
(RGW)
A bucket-based REST
gateway, compatible
with S3 and Swift
APP APP HOST/VM CLIENT
Ceph Architecture

9. Ceph Monitor Nodes:
– Maintain cluster map
– Provide consensus for
distributed decision-making
– Must have an odd number
– These do not serve stored
objects to clients
M Ceph OSD Nodes:
• One Ceph OSD daemon per disk
(recommended)
• At least three nodes in a cluster
• Serve stored objects to clients
• Intelligently peer to perform
replication tasks
• Supports object classes
Ceph RADOS Components

10. DISK
OSD
FS
btrfs
xfs
ext4
MMM
Ceph
Storage
Cluster
(RADOS)
Ceph
OSD
Node
DISK
OSD
FS
DISK
OSD
FS
DISK
OSD
FS
DISK
OSD
FS

11. Mission: “A Zero Touch Cloud Installer”
Servers in boxes to full function cloud in under 2 hours
• Fast & Flexible
– Bare metal install including BIOS & RAID config
– Users can choose how their system is configured (“barclamps”)
• DevOps Embracing
– Ongoing Operations Model (DevOps for Clouds)
– Leverages & Wraps Opscode Chef
• Open
– Not specific to OpenStack – Dell using for other Apps
– Not restricted to Dell hardware
– Apache 2 licensed
What is Crowbar?

12. Images vs. Layers: Overview
Images: Single Unit Layers: Stacked Pieces
Integrations +
Applications +
Utilities + Operating
System
Configuration
Operating System
Utilities
Application Bar
Application Foo
Integrations
Configuration

13. Images vs. Layers: Lifecycle
Images: Replacement Layers: Upgrade
I+A+U+O/
S
Config
OS
U
Bar v1
Foo
I
Config
I+A+U+O/
S
Config
I+A+U+O/
S
Config
I+A+U+O/
S
Config
OS
U
Bar v2
Foo
I
Config
Bar v2

14. Crowbar Uses the Layer Model for Deployment
Dell“Crowbar”
OpsManagement
Core Components &
Operating Systems
Cloud
Infrastructure &
Dell IP Extensions
Physical Resources
APIs, User Access,
& Ecosystem
Partners
* Crowbar has potential to service other programs beyond OpenStack
• Includes all the components required to implement an
entire cloud infrastructure including ecosystems partners
• Pluggable components deploy cloud infrastructure
• Allow for addition of Dell IP and expansion by the
community services and customers
• Can integrate with Dell existing products
• Delivers basic data center services and required cloud
infrastructure.
• Provision bare-metal servers from box to cloud WITHOUT
user intervention (other than racking/cabling and some
minimal configuration questions) NOT EXCLUSIVE TO DELL
HARDWARE

15. Server
How Does Crowbar Work?
Dell“Crowbar”
OpsManagement
Operating System
Network
ConfigChef
Server
Network
Config
Apps
DHCP
CrowbarMagic
Happens!
Orchestration
State Machine
Run List
Config.
Hardware Config
(uses “Sledgehammer”)
Apps

16. Dell“Crowbar”
OpsManagement
What is a Barclamp?
Barclamp
Chef Recipes
O/S App
Packages
Crowbar API &
Partial UI
Components &
Scripts
• Crowbar API & Partial UI
• Adds states and transitions to Orchestration
• Customizes UI specific to barclamp
• Visual extensions to nodes/networks/utils
• Chef Recipes
• DevOps description of application
• Enforced repeatable configuration
• O/A App Packages
• Operating system dependencies (Debs & RPMs)
• Components & Scripts
• Ad hoc configuration not covered by Chef

17. Modular Design: Barclamps The Configuration You
Want to DeployDell“Crowbar”
OpsManagement
Core Components &
Operating Systems
Cloud Infrastructure
& Dell IP Extensions
Physical Resources
APIs, User Access, &
Ecosystem Partners
Crowbar
Deployer
Provisioner
Network RAID
BIOS IPMI
NTP
DNS Logging
Nova Swift Glance
Nagios Ganglia
enStratus RightScale
Dashboard
Ceph
Horizon
KeystoneForce10

18. “We should do
things the new
way!”
Use Object
Storage
“App can’t be
re-architected!”
Use local storage!
“No SPOFs
please!”
Use this expensive
SAN!
“How do the Web
2.0 folks do it?”
Use distributed
storage on utility
hardware!
The 4 Stages of Cloud Storage Thinking

20. Ceph Block Storage for Dell Openstack
Integrated: with Cinder and Nova (via qemu-kvm)
Compatible: with Glance
Foundations: no SPOFs, self-healing, CRUSH
Features: Copy-on-Write, Snapshots, Cloning
Integrated: with Crowbar for automation

21. If you can’t automate it, you can’t scale it
OS
U
Nova
Node
I
Config
OS
U
Nova
API
I
Config
OS
U
Nova
Node
I
Config
OS
U
Nova
Node
I
Config
OS
U
Nova
Node
I
ConfigOS
U
Nova
Node
I
Config
OS
U
Nova
Node
I
Config
OS
U
Nova
DB
I
Config
OS
U
Nova
Cntrl
I
Config
OS
U
Dash
Board
I
Config
OS
U
Nagios
I
Config
OS
U
Crowbar
Chef
I
Config
OS
U
Ceph Mon
Node
I
Config
OS
U
Ceph Mon
Node
I
Config
OS
U
Ceph Mon
Node
I
Config
OS
U
Ceph OSD
Node
I
Config OS
U
Ceph OSD
Node
I
Config
OS
U
Ceph OSD
Node
I
Config
OS
U
Cinder
API
I
Config
API &
Services
Compute
Nodes
Storage
Nodes
Scale-Out Operations With OpenStack

22. Ceph Related Barclamps for Crowbar
• https://github.com/ceph/barclamp-ceph
– ceph.com packages into the Crowbar Admin Node
– Ceph OSD roles (ceph-store)
– Ceph Mon roles (ceph-mon)
• https://github.com/crowbar/barclamp-nova
– Hypervisor with RBD (ceph-client)

23. • Dell and Inktank have partnered to bring a complete solution for
OpenStack + Ceph + Automated deployment with Crowbar
• The joint solution provides:
– Crowbar barclamp to
– Deploy Ceph clusters automatically in a very short time
– Connect those clusters to OpenStack
– Professional Services, Support, and Training
› Collaborative Support for Dell hardware customers
– Joint Solution
› Validated against Dell Reference Architectures via the
Technology Partner program
What Have We Done to Enable?

24. Customer:
University to Deploy OpenStack + Ceph Using Crowbar
Situation
• University employs close to 900 researchers and receives
hundreds of million dollars in Grants as a Top 10 Research
Institution primarily working on Cancer and Genomic
Projects. The University has a need to provide a centralized
data repository for Researchers in order to insure
compliances concerning retention of data.
• The intent of the Data Repository is to provide 2TB of free
storage space to each Researcher, with opportunity to
purchase from the university more capacity at a very
reasonable cost.
Decision Drivers
• University investigated using an traditional SAN storage
solution which was very expenses on a per TB basis. They
also investigated using public cloud storage options which
also proved to be too expensive. Finally, they looked at
Hadoop for the project but found that it was not a good fit for
this use case.
• In the end, University
chose Dell/Inktank to
architect a platform
that would be very
scalable and provide
lost costs per TB and
was the best of all
worlds that provide
compute and storage
on the same
hardware.

25. What’s Next?
• May
– Ceph Cuttlefish
• Summer 2013
– Dell Grizzly support
– Ceph Dumpling (August)
• https://github.com/crowbar/crowbar/wiki/Crowbar-2.0
– Crowbar v2.0 work is happening in the open. Check it out and
participate!

26. Don’t Miss the OpenStack Summit Sessions
Havana Sessions:
– Planning the Ceph Roadmap for Openstack
Wednesday April 17, 2013
1:50pm - 2:30pm in Room - B113
– Features for Ceph with Cinder and Beyond
Thursday, April 18, 2013
9:50 – 10:30am in Rooms – C120+121+122

27. Contact Information
Reach Kamesh and Neil for additional information:
Dell.com/OpenStack
Dell.com/Crowbar
Inktank.com/Dell
Kamesh_Pemmaraju@Dell.com
@kpemmaraju
Neil.Levine@Inktank.com
@neilwlevine
Visit the Dell and Inktank booths in the OpenStack Summit Expo Hall