CloudStack is an open source cloud management platform that allows users to provision virtual machines and cloud infrastructure resources. It supports multiple hypervisors and public/private cloud deployments. CloudStack provides a self-service user portal, resource monitoring, and APIs to automate administration tasks. It uses a multi-tier architecture with zones, pods, clusters, and hosts to scale horizontally.

1. CloudStack Overview
Written by: Chiradeep Vittal, Alex Huang @ Citrix
Revised by: Gavin Lee, Zhennan Sun @ TCloud Computing

2. Outline
• Overview of CloudStack
• Problem Definition
• Feature set overview
• Network
• Storage
• MS internals
• System VMs
• System Interactions
• Roadmap
• Comparisons

4. What is CloudStack?
• Multi-tenant cloud
orchestration platform
– Turnkey
– Hypervisor agnostic
– Scalable
Build your cloud the way the – Secure
world’s most successful clouds – Open source, open standards
are built
– Deploys on premise or as a hosted
solution
– BSS, self service portal. (Not ASL)
– Extensive networking service
• Deliver cloud services faster
and cheaper

5. CloudStack Supports Multiple Cloud Strategies
Private Clouds Public Clouds
On-premise Hosted Multi-tenant
Enterprise Cloud Enterprise Cloud Public Cloud
• Dedicated • Dedicated • Mix of shared and
resources resources dedicated
• Security & total • Security resources
control • SLA bound • Elastic scaling
• Internal network • 3rd party owned • Pay as you go
• Managed by and operated • Public
Enterprise or 3rd internet, VPN
party access

6. CloudStack Provides On-demand Access to
Infrastructure Through a Self-Service Portal
Org A Org B
Users
Admin Admin
End User Users Users
Compute Network Storage
Admin

7. Open Flexible Platform
Compute Hypervisor
XenServer VMware Oracle VM KVM Bare metal
Storage Block & Object
Fiber
Local Disk iSCSI NFS Swift
Channel
Primary Storage Secondary Storage
Network Network & Network Services
Connection Load
Isolation Firewall VPN
Type balancer

8. Problem Definition
• Offer a scalable, flexible, manageable IAAS platform that
follows established cloud computing paradigms
• IAAS
– Orchestrate physical and virtual resources to offer self-service
infrastructure provisioning and monitoring
• Scalable
– 1 -> N hypervisors / VMs / virtual resources
– 1 -> N end users
• Flexible
– Handle new physical resource types
• Hypervisors, storage, networking
– Add new APIs
– Add new services
– Add new network models

9. Problem Definition (contd)
• Manageable
– Hide complexity of underlying resources
– Rich functional end-user and admin UI
– Admin API to automate operations
– Easy install, upgrade for small -> large clouds
– Simple scaling, automated resilience
• Established Paradigms
– EC2 –inspired
• Semantic variations based on cloud provider
needs, hypervisor capabilities

10. Feture Set Overview

11. Create Custom Virtual Machines via Service Offerings
Select Operating System
• Windows, Linux
Select Compute Offering
• CPU & RAM
Select Disk Offering
• Volume Size
Select Network Offering
• Network & Services
Create VM

12. Dashboard Provides Overview of Consumed Resources
• Running, Stopped &
Total VMs
• Public IPs
• Private networks
• Latest Events

13. Virtual Machine Management
Users
Change
VM Operations VM Access VM Status
Service Offering
Start
• CPU Utilized 2 CPUs 4 CPUs
Stop 1 GB RAM 4 GB RAM
• Network Read
20 GB 200 GB
Restart • Network Writes
20 Mbps 100 Mbps
Destroy

14. Volume & Snapshot Management
VM 1
Add / Delete
Volumes Volume
Create Templates Volume Template
from Volumes
Hourly Weekly
Schedule Now
Snapshots Daily Monthly
….
View Snapshot
History

15. Network & Network Services
• Create Networks and attach VMs
• Acquire public IP address for NAT &
load balancing
• Control traffic to VM using ingress
and egress firewall rules
• Set up rules to load balance traffic
between VMs

16. Core CloudStack Components
VM
• Hosts
• Servers onto which services will be provisioned Host
VM
• Primary Storage
• VM storage Host
• Cluster
Primary
• A grouping of hosts and their associated storage Storage
• Pod
• Collection of clusters Cluster
• Network
Secondary
• Within the same L2 switch Storage Network Cluster
• Secondary Storage
• Template, snapshot and ISO storage CloudStack Pod
• Zone
• Collection of pods, network offerings and secondary
storage CloudStack Pod
• Management Server Farm
• Responsible for all management and provisioning tasks
Zone

17. CloudStack Deployment Architecture
Internet  Hypervisor is the basic unit of
CloudStack
Management scale.
Server
Zone 1  Cluster consists of one ore
more hosts of same hypervisor
L3 core
 All hosts in cluster have access
to shared (primary) storage
Pod 1 Access Layer Pod N
Secondary
 Pod is one or more clusters,
…. Storage usually with L2 switches.
Cluster N
 Availability Zone has one or
more pods, has access to
…. secondary storage.
 One or more zones represent
Cluster 1
cloud
Host 1
Primary
Storage
Host 2

18. CloudStack Cloud Architecture
Cloud
Data Center 1 Data Center 2
Data Center 2
Data Center 3
Zone 2
Zone 2
Zone1 Zone 3
Zone 4 3
Zone
CloudStack Cloud can have
one or more Availability
Zones (AZ).
Data Center 2
Data Center 2
Data Center 2
Zone 2
Zone 2
ZoneZone 3
2
Zone 3
Zone 3

19. Management Server Managing Multiple Zones
Cloud
Data Center 1 Data Center 2  Single Management Server can
Data Center 2
Mgmt Data Center 3 manage multiple zones
Server
Zone 2  Zones can be geographically
Zone 2 distributed but low latency links are
Zone 3 expected for better performance
Zone1
Zone 4 3
Zone
 Single MS node can manage up to
10K hosts.
 Multiple MS nodes can be deployed
Data Center 2 as cluster for scale or redundancy
Data Center 2
Data Center 2
Zone 2
Zone 2
Zone Zone 3
2
Zone 3
Zone 3

20. Management Server Deployment Architecture
Single-node Multi-node
Deployment Deployment
MS
User API User API
MS MySQL Load
MS
DB Balancer
Admin API Admin API
MySQL
MS DB
Back Up
Replication DB
 MS is stateless. MS can be deployed
as physical server or VM
 Single MS node can manage up to
Infrastructure 10K hosts. Multiple nodes can be Infrastructure
Resources deployed for scale or redundancy Resources
 Commercial: RHEL 5.4+; FOSS:
Ubuntu 10.0.4, Fedora 16

21. CloudStack Storage
Primary Storage
• Configured at Cluster-level. Close to hosts for better
performance
L3 switch
• Stores all disk volumes for VMs in a cluster
• Cluster can have one or more primary storages
Pod 1 L2 switch
• Local disk, iSCSI, FC or NFS Secondary
Cluster 1 Storage
Host 1
Primary
Secondary Storage Storage
Host 2
• Configured at Zone-level
• Stores all Templates, ISOs and Snapshots
• Zone can have one or more secondary storages
• NFS, OpenStack Swift

22. Understanding the Role of Storage and Templates
• Primary Storage
• Cluster level storage for VMs
Host
• Connected directly to hosts
• NFS, iSCSI, FC and Local Host
• Secondary Storage Primary Storage
• Zone level storage for template, ISOs and Cluster
snapshots
• NFS or OpenStack Swift via CloudStack Pod
System VM
• Templates and ISOs
• Imported into CloudStack
• Can be private or public Secondary Storage
Zone
Template

23. Provisioning Process
1. User Requests Instance VM
2. Provision Optional Network Services Host
3. Copy instance template from Host
secondary storage to primary storage Primary Storage
on appropriate cluster
Cluster
4. Create any requested data volumes on
primary storage for the cluster Pod
5. Create instance Template
6. Start instance
Secondary Storage
Zone

24. Citrix XenServer
CloudStack
• Integrates directly with XenServer Manager
Pool Master
• Snapshots at host level XenServer Pool
Master Host
• System VM control channel at host
level XenServer Host
• Network management is host level XenServer Host
XenServer Host
XenServer Host
XenServer
Resource Pool

25. Oracle VM
• Integrates with ovs-agent CloudStack
Manager
• Snapshots at host level
• System VM control channel at OVS Agent
host level OVM Host
• Network management is host OVS Agent
level OVM Host
• Does not use OVM Manager OVS Agent
• All templates must be from Oracle OVM Host
• CloudStack configures ocfs2 nodes OVS Agent
OVM Host
• Requires “helper” cluster
• XenServer, KVM or vSphere

26. RedHat Enterprise Linux (KVM)
• Integrates with libvirt using Cloud
CloudStack
Agent Manager
• Snapshots at host level
Cloud Agent
• System VM control channel at
host level Libvirt
KVM Host
• Network management is host
level
Cloud Agent
• Only RHEL 6, not RHEV
Libvirt
• Also supports Ubuntu 10.04
KVM Host

27. VMware vSphere
CloudStack
• Integration through vCenter Manager
vSphere Host
• System VM control channel via
vCenter
CloudStack private network vSphere Host
• Snapshot and volume vSphere Cluster
management via Secondary
Storage VM vSphere Host
• Networking via vSphere vSwitch vSphere Host
vSphere Host
vSphere Cluster
Data Center

28. Management Server Interaction with Hypervisors
Management
Server
XAPI HTTPS
vCenter Agent Agent
XenServer
KVM OVM
ESX
• XS 5.6, 5.6FP1, 5.6 SP2, 6.0 • ESX 4.1, 5.0 (coming) • RHEL 6.0, 6.1, 6.2 (coming) • OVM 2.2
• Incremental Snapshots • Full Snapshots • Full Snapshots (not live) • No Snapshots
• VHD • VMDK • QCOW2 • RAW
• NFS, iSCSI, FC & Local disk • NFS, iSCSI, FC & Local disk • NFS, iSCSI & FC • NFS & iSCSi
• Storage over-provisioning: • Storage over-provisioning: • Storage over-provisioning: • No storage over-
NFS NFS, iSCSI NFS provisioning

29. Multi-tenancy & Account Management
Cloud
Resources
Domain
VMs, IPs, Snapshots…
• Domain is a unit of
Org A isolation that represents
Admin a customer org, business
unit or a reseller
Domain
Reseller A
• Domain can have
Admin Resources arbitrary levels of sub-
Sub-Domain
Org C
VMs, IPs, Snapshots… domains
Admin
• A Domain can have one
Account
or more accounts
Group A
• An Account represents
Account one or more users and is
Group B the basic unit of isolation
User 1 • Admin can limit
resources at the Account
User 2
or Domain levels

30. CloudStack Network

31. Network Terminology
• Traffic type
– Guest: The tenant network to which instances are attached
– Storage: The physical network which connects the hypervisor to primary storage
– Management: Control Plane traffic between CloudStack management server
and hypervisor clusters
– Public:
“Outside” the cloud [usually Internet]
Shared public VLANs trunked down to all hypervisors
• Network type
– Shared, one subnet serve multiple users
Direct. 1 subnet
Direct tagged. VLAN, multiple subnet
– Isolated, different subnet for different user
Virtual (tagged)
• All traffic can be multiplexed on to the same underlying physical network
using VLANs
– Usually Management network is untagged
– Storage network usually on separate nic (or bond)
• Admin informs CloudStack how to map these network types to the
underlying physical network
– Configure traffic labels on the hypervisor
– Configure traffic labels on Admin UI

32. VM Instance
• Choose the instantiated guest network
• IP is arbitrary
Guest Network
• Instance of Network Offering
• Shared: created by Admin
• Isolated: Created and owned by user
• One virtual router for one network
• Cross pod, within Zone
• VLAN id picked from the pool
Physical Network Network Offering
• Only for Guest traffic
• Zone level
• Guest network type: Shared or Isolated
• Defined by NIC
• Defined a set of network services, such
• Assigned with traffic type (P, G, M, S)
as DHCP, Firewall, VPN, NAT…
• Associated by label/vswitch name
• Bandwidth
• Attached with device as service provider
Tag

33. Physical Network
Operations
Users
Admin and
Cloud API
CloudStack MS
Cluster
Router
MySQL
Load Balancer
Availability Zone
L3 Core Switch
Access
Layer
Switches
Secondary
Servers
… … … … … Storage
Pod 1 Pod 2 Pod 3 Pod N

34. Network Isolation
Web DB Web
VM VM VM
Web DB
Security Security
Group Group
Web Web DB
VM VM VM
… … …
Web Web
VM VM

35. Network Isolation (Security Group, L3)
Public Guest 1 10.1.0.2
Internet VM 1
10.1.0.1 Guest 2
Pod 1 L2 10.1.0.3
Switch VM 1
Guest 1 10.1.0.4
VM 2
L3 Core
Pod 2 L2
Switch
10.1.8.1
…
Switch
Guest 2 10.1.16.12
Load 10.1.16.1 VM 2
Pod 3 L2
Balancer Switch
Guest 2
10.1.16.21
VM 3
… Guest 1
VM 3
10.1.16.47
Guest 1
10.1.16.85
VM 4

36. Network Isolation (VLAN, L2)
Core (L3) Network
Pod K Pod M Pod N
Access Switch(es) V
Hypervisor
V
V
Hypervisor
R
…
CLUSTER 1
Hypervisor 1
R
VLAN 101 Traffic …
Hypervisor 8
VLAN 102 Traffic
…
CLUSTER 4
V V
Hypervisor N
V Tenant VM
Hypervisor N+1
V
R Tenant Virtual Router

37. Guest virtual network
Guest Virtual Network
10.1.1.0/24
Public Public IP Guest 1
Gateway Guest
Network address VM 1
address address
65.37.141.11
10.1.1.1 10.1.1.2
65.37.141.36
Guest 1 Virtual Guest 1 Guest
Public Router VM 2 address
Internet 10.1.1.3
NAT
Guest 1 Guest
DHCP
VM 3 address
Load Balancing
VPN 10.1.1.4
Guest 1 Guest
VM 4 address
10.1.1.5
Guest Virtual Network
Public IP 10.1.1.0/24
address Gateway Guest 2 Guest
65.37.141.24 address VM 1 address
65.37.141.80 10.1.1.1 10.1.1.2
Guest 2 Virtual Guest 2 Guest
Router VM 2 address
10.1.1.3
NAT
Guest 2 Guest
DHCP
VM 3 address
Load Balancing
VPN 10.1.1.4

38. Guest Virtual Network With Physical Device
CS Virtual Router provides Network Services External Devices provide Network Services
Guest Virtual Network 10.1.1.1/8 Guest Virtual Network 10.1.1.1/8
VLAN 100 VLAN 100
Public Public
Network/Internet Network/Internet
Guest Private IP Guest
10.1.1.1 VM 1 Public IP 10.1.1.1 VM 1
65.37.141.111 Juniper 10.1.1.111
Gateway
Public IP SRX
address
CS Firewall
65.37.141.11 10.1.1.1 Guest Guest
Virtual
10.1.1.3 VM 2 10.1.1.3 VM 2
Router
Public IP Private IP
DHCP, DNS NetScaler 10.1.1.112
65.37.141.112
NAT Guest Load Guest
Load Balancing 10.1.1.4 VM 3 Blancer VM 3
10.1.1.4
VPN
Guest Guest
10.1.1.5 VM 4 10.1.1.5 VM 4
CS
Virtual DHCP, DNS
Router

39. Layer-3 Guest Network
Network Services Managed Externally Network Services Managed by CS
Public Network
65.11.0.0/16
Security Group 1
Security Group 1
Public
Network/Internet 10.1.2.3
65.11.1.2 Guest VM
Guest VM
1
1
65.11.1.2 10.2.12.4
65.11.1.3 NetScaler Guest VM
Guest VM L3
65.11.1.3 Load 2
2 switch
Blancer
65.11.1.4
EIP,
ELB 10.5.2.99
65.11.1.4 Guest VM
Guest VM
3
3
10.1.2.18
65.11.1.5 Guest VM
Guest VM
4
4
CS
CS Virtual
Virtual DHCP, Security Group 2
DHCP, Security Group 2 Router
Router DNS
DNS

40. Multi-tier network
Multi-tier network
Virtual Network Virtual Network
Virtual Network 10.1.2.0/24 10.1.3.0/24
10.1.1.0/24 VLAN 1001 VLAN 141
Public VLAN 100
Network/Intern App VM
10.1.2.31 1 10.1.3.21
et Web VM
Public IP Private IP 10.1.1.1 1 10.1.2.21
65.37.141.11 Juniper 10.1.1.111
1 SRX App VM
Firewall 10.1.2.24 10.1.3.45
Web VM 2
10.1.1.3 2 10.1.2.18
Public IP Private IP
65.37.141. Netscaler 10.1.1.112
112 Load Web VM
Balancer 10.1.1.4 3 10.1.2.38 10.1.3.24 DB VM 1
Web VM
10.1.1.5 4 10.1.2.39
CS DHCP, CS DHCP,
Virtual Virtual DNS CS
DHCP, DNS, Router Virtual
Router
User- User-
DNS Router
data data,
User-
Source
data Public IP
-NAT,
65.37.141.115

41. Multi-tier unified [vision]
Internet
IPSec or SSL site-to-site VPN
CS
Virtual Router / Customer
Loadbalancer Other Premises
Monitoring VLAN
Virtual Router Services
App VM
• IPAM 10.1.2.31 1
• DNS 10.1.1.1
Web VM
1
• LB [intra]
• S-2-S VPN App VM
10.1.2.24
• Static Routes Web VM 2
• ACLs 10.1.1.3 2
• NAT, PF
• FW [ingress & egress]
Web VM
• BGP 10.1.1.4 3 10.1.3.24 DB VM 1
Web VM
10.1.1.5 4
Virtual Network Virtual Network Virtual Network
10.1.1.0/24 10.1.2.0/24 10.1.3.0/24
VLAN 100 VLAN 1001 VLAN 141

42. Multi-tier unified with SDN[vision]
Internet
IPSec or SSL site-to-site VPN
CS
Loadbalancer Virtual Router / Customer
Other Premises
Virtual Appliance
Monitoring VLAN
Virtual Router Services
App VM
• IPAM 10.1.2.31 1
• DNS 10.1.1.1
Web VM
1
• LB [intra]
• S-2-S VPN App VM
10.1.2.24
• Static Routes Web VM 2
• ACLs 10.1.1.3 2
• NAT, PF
• FW [ingress & egress]
Web VM
• BGP 10.1.1.4 3 10.1.3.24 DB VM 1
Web VM
10.1.1.5 4
Overlay Overlay Overlay
Network Network Network
10.1.1.0/24 10.1.2.0/24 10.1.3.0/24

43. Network Offerings
• Cloud provider defines the
feature set for guest networks
• Toggle features or service levels
– Security groups on/off
– Load balancer on/off
– Load balancer software/hardware
– VPN, firewall, port forwarding
• User chooses network offering
when creating network
• Enables upgrade between
network offerings
• Default offerings built-in
– For classic CloudStack networking

44. CloudStack Storage

45. Storage
• Primary Storage
Zone-Level Layer 3 Switch
Private Network – Block device to the VM
– IOPs intensive
– Accessible from host or
Pod 1 Pod Pod cluster wide
Pod-Level Layer-2
Switch
2 N
– Supports storage tiering
Scale-Out
NFS • WORM Storage
– Secondary Storage or Object
Computing
Primary
Server 1
Storage
Cluster 2
Store for templates, ISO, and
Computing
Primary
snapshot archiving
Server 2
Storage – High capacity
Computing
Scale-Out
NFS
• CloudStack manages the
Server 3 storage between the two to
Cluster 1
Primary
Storage achieve maximum benefit and
Computing
Server 4 resiliency

46. Primary Storage Support Matrix
Type XenServer VmWare KVM
Local Disk Supported Supported Supported
iSCSI Supported Supported Not Supported
Fiber Channel Supported Supported Not Supported
NFS Supported Supported Supported
VM Storage Supported Supported Supported
Network

47. Storage Tiering
• Supported via storage tags for primary storage
• Specify a tag when adding a storage pool
• Specify a tag when adding a disk offering
• Only storage pools with the tag will be
allocated for the volume

48. WORM Storage
• Write Once Read Many storage pattern is
supported by two different storage types
– Secondary Storage (NFS Server within an availability
zone)
– Object Store (Swift implementation for cross-zone)
• Objective for WORM storage
– High capacity, cheap storage
– Easy to increase capacity
• Used to store templates, ISOs, and snapshots

49. Snapshots
• Snapshots are used as backups for DRS
• Taken on the primary storage and moved to
secondary storage
• Supports individual snapshots and recurring
snapshots
• Full snapshots on VmWare and KVM. Need help.
• Incremental snapshots on XenServer
• Allows backup network traffic to be specified in
zone to segregate the backup network traffic
from other network traffic types

50. MS Internals
• Architecture
• Workflow
• High Availability
• Scalability

51. Inside a Management Server
Plugins
cmd.execute() Plugins
Cmds Plugins
Async
CS API API Job
Services
Servlet Queue
API
Mgr Kernel
Responses
Agent API
(Commands) Agent Resources
Manager Local
Or
Remote
Hypervisor Network
Native Device
APIs API
MySQL

52. Old Architecture
API Layer
Pros
EC2 CloudStack
Access Control • Agile development for
existing developers
Virtual Machine Manager
Console Proxy Manager
• Scales well horizontally
Async Job Manager
Snapshot Manager
Template Manager
Network Manager
Storage Manager
Cons
… • Monolithic
• Difficult to educate
new and third-party
Agent Manager
XenServ KVM SRX F5
NetScal Other
developers
er er
• Easy to introduce bugs
Resour Resour Resour Resourc
Resourc ce ce Resour es
ce
e ce
52

53. New Deployment Architecture
• Scales horizontally to
different pressure points
• Automatically scales
service VMs in zones to
facilitate most efficient
data path transfers
• Fault isolation between
API servers and
Execution Servers and
resources within zones

54. New Architecture – API Server
UI
Cloud
CLI
Other
Clients
• API Server isolates
Portal
integration code from
REST
Execution Server
API Server • API Server can
OAM&P API
Pluggable API Engine
End User EC2 Other
horizontally scale to
Management Services
API API
ACL &
APIs
Integration
handle traffic
- Resource
management
- Configuration
Authentication
- Accounts,
Domains, and
• Easily adds other API
- Additional
operations added
Projects
- ACL, limits
compatibility
• Easily exposes API
by third party checking
Framework
- Job Queue
-
-
Database Access Layer
OSGi
needed by third party
vendors

55. New Architecture – Execution Server
Execution Server • Execution Server protected by
job queue
• Kernel kept small for stability. It
Services API
Kernel Plugins
• Drives long running VM operations • Storage only drives processes.
• Syncs between resources managed
and DB •
Handling
Network • Plugins provide mappings of
• Generates events
•
Handling
Deployment
virtual entities to physical
•
planning
Hypervisor
resources
Handling
• Third party plugins to provide
vendor differentiation in
CloudStack
• Cluster Management •
Framework
Component Framework • Communicates with resources
•
•
Job Management
•
Alert & Event Management
(OSGi)
Transaction Management
within data center over message
•
•
Database Access Layer
Messaging Layer
bus

56. New Architecture – Resources
Agent
• Resources are carried in
Hypervisor Resources
service VMs to be in close
network proximity to the
Network Resources
physical resources it
Storage Resources manages
Image & Template Resources
• Easily scales to utilize the
most abundant resource in
Snapshot Resources
data center (CPU & RAM)
• Communicates with
Execution Server over
message bus (JSON)
• Can be replicated for fault
tolerance

57. Cloud Other
UI CLI Clients
Portal
Management Server
REST API
OAM&P API End User API EC2 API Other APIs Pluggable Service API Engine
Console Proxy ACL & Authentication Security Adapters
Management - Accounts, Domains, and Projects
- ACL, limits checking Account Management
Connectors
Template Services API
Access
Deployment Planning
Plugin API
HA
Kernel
- Drives long running VM
Services API
Network Configurations
Usage operations
Calculations - Syncs between resources
managed and DB Network Elements
Additional - Generates events
Services
Hypervisor Gurus
Cluster Resource Job Alert & Event Database
Management Management Management Management Access
Event Bus
Message Bus
Hypervisor Network Storage Image Snapshot
Resources Resources Resources Resources Resources

58. Kernel Module
• Understands how to orchestrate long running
processes (i.e. VM starts, Snapshot
copies, Template propagation)
• Well defined process steps
• Calls Plugin API to execute functionalities that
it needs

59. Plugins
• Various ways to add more capability to
CloudStack
• Implements clearly defined interfaces
• All operations must be idempotent
• All calls are at transaction boundaries
• Compiles only against the Plugin API module

60. Anatomy of a Plugin
Rest API
- Optional. Required only if needs to expose configuration API to admin. ServerResource
- Optional. Required if
Plugin needs to be co-
located with the
resource
- Implements translation
layer to talk to resource
- Communicates with
Plugin API
Implmentation server component via
JSON
Data Access Layer

61. Anatomy of a Plugin
• Can be two jars: server component to be
deployed on management server and an optional
ServerResource component to be deployed co-
located with the resource
• Server component can implement multiple Plugin
APIs to affect its feature
• Can expose its own API through Pluggable Service
so administrators can configure the plugin
• As an example, OVS plugin actually implements
both NetworkGuru and NetworkElement

62. Plugin Interfaces Available
• NetworkGuru – Implements various network isolation technologies
and ip address technologies
• NetworkElement – Facilitate network services on network elements
to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)
• DeploymentPlanner – Different algorithms to place a VM and
volumes.
• Investigator – Ways to find out if a host is down or VM is down.
• Fencer – Ways to fence off a VM if the state is unknown
• UserAuthenticator – Methods of authenticating a user
• SecurityChecker – ACL access
• HostAllocator – Provides different ways to allocate host
• StoragePoolAllocator – Provides different ways to allocate volumes

63. Adding a Plugin to CloudStack
• Components are configured through
components.xml
• Supports DAO, Manager, and Adapter patterns
• Open to other component frameworks (OSGi a
possibility)

64. Components.xml Example
<components.xml>
<system-integrity-checker class="com.cloud.upgrade.DatabaseUpgradeChecker">
<checker name="ManagementServerNode" class="com.cloud.cluster.ManagementServerNode"/>
<checker name="EncryptionSecretKeyChecker"
class="com.cloud.utils.crypt.EncryptionSecretKeyChecker"/>
<checker name="DatabaseIntegrityChecker" class="com.cloud.upgrade.DatabaseIntegrityChecker"/>
<checker name="DatabaseUpgradeChecker" class="com.cloud.upgrade.PremiumDatabaseUpgradeChecker"/>
</system-integrity-checker>
<interceptor library="com.cloud.configuration.DefaultInterceptorLibrary"/>
<management-server class="com.cloud.server.ManagementServerExtImpl"
library="com.cloud.configuration.PremiumComponentLibrary">
<adapters key="com.cloud.storage.allocator.StoragePoolAllocator">
<adapter name="LocalStorage" class="com.cloud.storage.allocator.LocalStoragePoolAllocator"/>
<adapter name="Storage" class="com.cloud.storage.allocator.FirstFitStoragePoolAllocator"/>
</adapters>
<pluggableservice name="VirtualRouterElementService"
key="com.cloud.network.element.VirtualRouterElementService"
class="com.cloud.network.element.VirtualRouterElement"/>
</management-server>
</components.xml>

65. ServerResource
• Translation layer between CloudStack
commands and resource API
• May be Co-located with resource
• Have no access to DB
• API defined in JSON messages

66. DAO
• SQL generation done mostly in GenericDaoBase
• Uses JPA annotations
• Very little code to write for each individual DAO
• Database Access Layer for Kernel
• No support for more complicated features such
as fetch strategy
• Welcome to use other types of ORM in other
modules but like to hear about preferred library.
(Hibernate is out due to licensing issues)

67. Example DAO
// ExampleVO.java // ExampleDao.java
@Entity public interface ExampleDao
@Table(name=“example”) extends GenericDao<ExampleVO, Long> {
public class ExampleVO { }
@Id
@GeneratedValue(strategy= // ExampleDaoImpl.java
GenerationType.IDENTITY) @Local(value=ExampleDao.class)
@Column(name=“id”) public class ExampleDaoImpl
long id; extends GenericDaoBase<ExampleVO, Long>
implements ExampleDao {
@Column(name=“name”)
String name; protected ExampleDaoImpl() {
}
@Column(name=“value”) }
String value;
}

68. Sequence Flow for deploy VM Kernel
End User Security User VM VirtualMac Network Storage Network Job
Rest API Checkers Mgr hine Mgr Mgr Mgr Guru Scheduling
Deploy VM
ACL Checks
Allocate Entity in CS
Allocate VM
Allocate NIC
Allocate IP
Allocate Volume
Schedules Deploy Job
Returns with job id, VM id
Query Job Result
Returns with job status

69. Sequence Flow for deploy VM
User VM VirtualMac Network Storage Network Network Template Deployment Server
Job Threads Services API
Mgr hine Mgr Mgr Mgr Guru Element Mgr Planner Resource
Start VM
Start User VM
Start VM
Get a Deployment Plan (Host and StoragePool)
Prepare Nics
Reserve resources for Nic
Notify that Nic is about to be started in network
Agent Calls
Prepare Volumes
Prepare template on Primary Storage
Agent Calls
Agent Start VM Call
Stores job result

70. High Availability

71. High Availability
• Service Offering contains a flag for whether
HA should be supported for the VM
• Does not use the native HA capability of
hypervisors for XenServer and KVM
• Uses adapters to fine tune HA process

72. Triggering High Availability
VM HA are triggered via the following methods:
• VM Sync detects out of band VM changes
• Resource Management detects that a resource is
unreachable and its state can not be determined.
• VM start/stop has been sent to the resource but
resource does not return
• Details of how high availability is done is at
http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/CloudStack_High_Availability_-
_Developer's_Guide

73. High Availability
Has VM
changed since Yes
Cancel Work
• Investigation
work scheduled?
No
– Uses investigators to find out if
Investigation
VM is alive or down
– Each investigator returns three
No
Needed?
Yes
states
• Up
Up
• Down
Failure Start VM
Down
Is VM Up or Up Is hypervisor
Down? host Up or
Success
Down?
• Unknown
Unknown Down
• Fencing
Completed Work
Has more
– Uses fencers to fence off the VM
Investigators
?
Yes
from accessing storage to ensure
No
VM is not corrupted
Reschedule Work
– Each Fencer returns three states
Yes
Fence off
VM?
• Fenced
• Unable to Fence
No
• Don’t know how to fence
• Restart
– Restarts the VM
More Yes
Fencers??
No

74. Scalability

75. Current Status
• 10k resources managed per management server
node
• Scales out horizontally (must disable stats
collector)
• Real production deployment of tens of thousands
of resources
• Internal testing with software simulators up to
30k physical resources with 30k VMs managed by
4 management server nodes
• We believe we can at least double that scale per
management server node

76. Balancing Incoming Requests
• Each management server has two worker thread pools for incoming
requests: effectively two servers in one.
– Executor threads provided by tomcat
– Job threads waiting on job queue
• All incoming requests that requires mostly DB operations are short
in duration and are executed by executor threads because incoming
requests are already load balanced by the load balancer
• All incoming requests needing resources, which often have long
running durations, are checked against ACL by the executor threads
and then queued and picked up by job threads.
• # of job threads are scaled to the # of DB connections available to
the management server
• Requests may take a long time depending on the constraint of the
resources but they don’t fail.

77. Comparison of two Approaches
• Stats Collector – collects capacity statistics
– Fires every five minutes to collect stats about host CPU and
memory capacity
– Smart server and dumb client model: Resource only
collects info and management server processes
– Runs the same way on every management server
• VM Sync
– Fires every minute
– Peer to peer model: Resource does a full sync on
connection and delta syncs thereafter. Management
server trusts on resource for correct information.
– Only runs against resources connected to the management
server node

78. Numbers
• Assume 10k hosts and 500k VMs (50 VMs per host)
• Stats Collector
– Fires off 10k requests every 5 minutes or 33 requests a second.
– Bad but not too bad: Occupies 33 threads every second.
– But just wait:
• 2 management servers: 66 requests
• 3 management servers: 99 requests
– It gets worse as # of management servers increase because it did not auto-balance across
management servers
– Oh but it gets worse still: Because the 10k hosts is now spread across 3 management servers.
While it’s 99 requests generated, the number of threads involved is three-fold because
requests need to be routed to the right management server.
– It keeps the management server at 20% busy even at no load from incoming requests
• VM Sync
– Fires off 1 request at resource connection to sync about 50 VMs
– Then, push from resource as resource knows what it has pushed before and only pushes
changes that are out-of-band.
– So essentially no threads occupied for a much larger data set.

79. Resource Load Balancing
• As management server is added into the cluster, resources are rebalanced
seamlessly.
– MS2 signals to MS1 to hand over a resource
– MS1 wait for the commands on the resources to finish
– MS1 holds further commands in a queue
– MS1 signals to MS2 to take over
– MS2 connects
– MS2 signals to MS1 to complete transfer
– MS1 discards its resource and flows the commands being held to MS2
• Listeners are provided to business logic to listen on connection status and
adjusts work based on who’s connected.
• By only working on resources that are connected to the management
server the process is on, work is auto-balanced between management
servers.
• Also reduces the message routing between the management servers.

80. CloudStack System VMs

81. CloudStack System VMs
• System VMs optimize and scale the data path on behalf of CloudStack
– Stateless, can be destroyed and recreated from database state
– Highly Available
– Communicates with Management Server over management network
– Usually have 3 interfaces: control(linked-local), mgmt and public
• Console Proxy VM
– Provides AJAX-style HTTP-only console viewer
– Grabs VNC output from hypervisor
– Scales out (more spawned) as load increases
– Java-based server Communicates with MS
• Secondary Storage VM
– Provides image (template) management services
– Download from HTTP file share or Swift
– Copy between zones
– Scale out to handle multiple NFS mounts
– Java-based server communicates with MS

82. CloudStack System VMs
• Virtual Router VM
– Provides multiple network services
– IPAM (DHCP), DNS, NAT, Source NAT, Firewall, Port Forwarding, VPN
– User-data, Meta-data, guest SSH keys and password change server
– Redundancy via VRRP
– MS configures VR over SSH
• Proxied via the hypervisor on XS and KVM

83. System VM spec
• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security
APT repository. No extraneous accounts
• 32-bit for enhanced performance on Xen/VMWare
• Only essential software packages are installed. Services such
as, printing, ftp, telnet, X, kudzu, dns, sendmail are not installed.
• SSHd only listens on the private/link-local interface. SSH port has been changed to a non-
standard port (3922). SSH logins only using keys (keys are generated at install time and are
unique for every customer)
• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum
performance on all hypervisors. Xen tools inclusion allows performance monitoring
• Template is built from scratch and is not polluted with any old logs or history
• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved
security and speed
• Latest version of jre from Sun/Oracle ensures improved security and speed

84. System VM contd
• SSH keys and password are unique to cloud
installation
• Code can be patched by restarting system vm
– Mounts a special ISO file with latest code at boot
– If ISO contents differ, patch and reboot
• Same system vm works on XS, KVM, VMWare
– Bootstrap step for the cloud is to install the template
for this system vm
• Ready to be re-purposed for other specialized
tasks

85. Interactions
OVM Cluster Primary
Storage
vcenter
Monitoring Primary
CS API vSphere Cluster
Storage
End
User UI
Primary
XS Cluster Storage
Admin
UI
Clustered
CloudStack XAPI
Domain CS Admin & CloudStack
CloudStack
Admin End-user API Primary
UI
Management JSON KVM Cluster Storage
Server
NetConf
Juniper SRX
Cloud user Nitro API
{API client (Fog/etc)} VNC
JSON
ec2 API JSON Netscaler
Cloud user Console
Console
{ec2 API client } Proxy VM
Proxy VM NFS
MySQL Server
{Proxied} SSH Sec. Storage NFS NFS
Sec. Storage
VM
Ajax HTTPS VM
Console
Router VM HTTP (Template Download)
Router VM HTTP (Template Copy)
Router VM
Cloud user HTTP (Swift)

86. CloudStack Roadmap
2012 2013
Feb Apr Jul Oct Feb
Acton Bonita Burbank Campo ?
► Swift Integration ► OpenvSwitch Support ► Inter-Vlan Routing ► AWS-style Regions
► Hyper-V (win 8)
► Support XenServer 6 ► VMWare Distributed ► Multi-tier App ► IPv6
► Support Vsphere 5 vSwitch Support ► Site-to-Site VPNs ► Resource Scaling
► Netscaler Integration ► Cisco Nexus 1000v ► AWS-style tags ► Dedicated Resource
Support Module
► Refine Resource ► VM Tiers
Management ► Upload Volume ► Scalability (50K hosts)
► UI refinement ► Plugin Architecture
► LDAP/AD Authentication ► Hypervisor
Enhancement
► Clustered LVM support

87. CloudStack vs. OpenStack vs. Eucalyptus

88. CloudStack
• Mainly written in Java
• ASL2.0 license
• Has more than 100 production clouds (Around May, 2012)
• Support private/hybrid/public cloud
• Scale to 30K physical host in commercial environment
• Support XenServer/Vsphere/KVM/OVM/Hyper-V/Baremetal
as hypervisor
• Multiple geographically distributed datacenters management
• Flexible and rich network functionality
• Easy installation and management
• Amazon EC2 API compatible
• Well documented
• Active community

89. OpenStack
• Mainly written in Python
• ASL2.0 license
• Support private/hybrid/public cloud
• Immature for commercial usage
• Support XenServer/Vsphere/KVM/Xen/Hyper-V as hypervisor
• Network is a single point of failure
• Weak VPN support for enterprise hybrid cloud
• All inter-module communication are based on MQ
• Not well documented
• A bit hard to install
• Amazon EC2 API partially compatible

90. Eucalyptus (Open Source edition)
• Mainly written in Java
• GPLv3 license
• Focus on private cloud
• Support KVM/Xen as hypervisor
• Fully compatible with Amazon EC2
• Fully compatible with Amazon S3 via Walrus
• EBS support via AoE and iSCSI
• Both web UI and command line tools for cloud administration
• Well documented
• Difficult to getting started