Apidays New York 2024 - The value of a flexible API Management solution for O...
Using next gen storage in Cloudstack
1. Using Next Generation Storage to
Improve Performance, Scalability &
Efficiency in your CloudStack
Infrastructure
Giles Sirett
CEO
Giles.sirett@shapeblue.com
Twitter: @ShapeBlue
2. Agenda
Key Design Considerations for a CloudStack IaaS Cloud
Primary & Secondary Storage
Scaling Storage
The IOPS Dilemma
Real World Storage Scenarios
Solidfire’s Next Generation Storage Platform
Demo of Solidfire / CloudStack Integration
Q&A
@ShapeBlue
5. About ShapeBlue
“ShapeBlue are expert builders of public & private
clouds. They are the leading global independent
CloudStack / CloudPlatform integrator &
consultancy”
@ShapeBlue
10. Secondary Storage
Configured at Zone level
Has to be accessible by Management Servers, Hosts and SSVMs
Stores all Templates, ISOs and Snapshots
Has to be NFS
Swift and S3 can be used to replicate data between Zones
Zones can have more than one Secondary Storage Volume
Public Templates & ISOs are copied to every store within a Zone
@ShapeBlue
11. Scaling Secondary Storage
Capacity determined by
Number of Public ISOs
Number of Public Templates
Data Consumed by Users
Volume Snapshots
Private ISOs
Private Templates
Uploaded Volumes
@ShapeBlue
12. Primary Storage
Configured at Cluster or Zone level
Stores all Disk Volumes and VM Snapshots for VMs
Options are:
NFS
iSCSI
FC
Local Storage
High Performance is critical
@ShapeBlue
14. Considerations for Scaling Primary Storage
Workload is a key factor
Optimal LUN Size for iSCSI ~ 500GB
Capacity required for each VM
Disk performance required for each VM
20 VMs per iSCSI LUN for optimal performance
Significant admin overhead to create and manage
Admins tend to create large LUNs for ease of management
Performance Suffers
Scale-Out is a manual process
@ShapeBlue
15. Cloud Scale Requires Automation
Cloud scale requires automation of all components
CloudStack can orchestrate
Hypervisors
Networks (SDN)
Network Appliances – F5, LB, Netscaler etc
Virtual Machines
Storage is a very significant piece of the infrastructure
Choose a storage vendor who integrates with CloudStack
@ShapeBlue
16. Scaling Primary Storage - Example
4 XenServer Clusters, each with 4 Hosts
28 usable cores (after Hypervisor overhead)
764GB usable RAM (after Hypervisor overhead)
Mapping of 6vCPU to 1 core
@ShapeBlue
17. Scaling Primary Storage - Example
Average VM Specification is 1vCPU / 4GB of RAM
Based on total host capacity
2528 Guest VMs
75,840 IOPs (30 IOPS per VM)
50.5 TB of Storage (20GB per VM)
1264 VRs
12640 (10 IOPS per VR)
2.5 TB of Storage (2 GB per VM)
@ShapeBlue
18. Storage Array IOPS vs Capacity - Example
What we need
What can a traditional storage array provide (real world figures)
53 TB capacity
88,480 IOPS
50TB Capacity
9,500 IOPS
Therefore we need 10 Storage Arrays
500TB Capacity
95,000 IOPS
10x the required capacity to
achieve required IOPS
@ShapeBlue
22. Other Factors Affecting Capacity
Linked Clones
Each VM within a LUN/Volume is linked to a ‘Template’ image
Template may have 20GB allocated but only 2GB on disk
Linked VMs only contain the differences – starts at 200MB
Thin Provisioning
Deduplication
Actual storage space required is much lower than you think!
@ShapeBlue
23. IOPS vs Capacity Problem
Total IOPS per Storage Array – not per VM
Traditional Multi-Tiered Storage has its limits
Forced to add excess capacity to achieve IOPS
Deduplication actually makes matters worse
@ShapeBlue
25. Key Primary Storage
Considerations
Guaranteed
Performance
Fine-grain performance
management with Guaranteed
Quality of Service (QoS)
Scale-out
Simultaneous scaling of both
capacity and performance
In-line
efficiency
In-line data reduction reduces data
footprint, requires less purchased capacity
High
Availability
RAID-less data protection maintains data
availability, security and performance
regardless of failure condition.
Management
Automation
Intuitive UI, CLI, and REST-based
API for ease of management
complete automation
26. Scale-Out Architecture
Linear Scalability
96TB
5, SF3010 Nodes
• 60 TB
• 250,000 IOPS
Starter 5 node
SF3010 configuration
Capacity
84TB
400,000 IOPS
350,000 IOPS
72TB
300,000 IOPS
60TB
250,000 IOPS
Performance
• New nodes are added as demand dictates
• Performance and capacity instantly available to all volumes
• Nodes added on the fly without down time
27. High Availability
•
Cluster wide RAID-less data protection,
SolidFire Helix™
•
Self-healing – Automatic and complete restoration of
redundancy after failure
•
Complete HA – No single points of failure
•
Non-disruptive upgrades
9
Maintains all QoS settings in the face of component
and/or system level failure
•
7
Non-controller based architecture
•
6
4
3
Failure Isolation – without performance impact
•
1
2
A
7
5
5
8
8
4
6
3
2
1
A
9
28. Guaranteed Performance
• Allocate: Storage performance
independent of capacity
• Manage: performance real-time without
impacting other volumes
• Guarantee: performance to every
volume with fine-grain QoS settings
29. Efficiency, In-line
•
Always-on, in-line Data Reduction
– Deduplication
– Compression
– Thin Provisioning
•
Executed across entire data store without
performance impact
•
Delivers drastic reduction in power, cooling,
and floor space
30. System Automation
• Integrated REST-based API
• Enables complete automation of any SolidFire
function
• Supports development of user-facing storage
controls
• Seamless integration into current
billing/charge-back systems and management
stacks
• API Integration Support from SolidFire
34. Further Information or questions slide
Shapeblue.com/blog
Slides can be found at
www.slideshare.net/shapeblue
Giles.sirett@shapeblue.com
Geoff.higginbottom@shapeblue.com
@shapeblue
@cloudstackgurou
@ShapeBlue
35. Using Next Generation Storage to
Improve Performance, Scalability &
Efficiency in your CloudStack
Infrastructure
Giles Sirett
CEO
Giles.sirett@shapeblue.com
Twitter: @ShapeBlue
Notas del editor
RegionA grouping of Availability Zones within a geographic areaManagement Server Infrastructure to manage the RegionAvailability ZoneTypically one Zone per DCContains at least 1 POD, 1 Cluster and Secondary StoragePodTypically a Rack containing one or more Clusters and NetworkingClusterGroup of identical Hosts running a common HypervisorPrimary StoragePrimary StorageTraditionally Unique to each ClusterKVM and VMware now support Zone-Wide Primary StorageHosts the Guest Virtual Machines and VM SnapshotsCan be any format the Hypervisor SupportsSecondary StorageZone WideNFS + S3 or Swift for Region Wide ReplicationStores Templates, ISOs and Volume Snapshots (backups)
Highlight the lack of Automation so traditional use of large LUNs = lower performanceHorizontalScale out requires lots of Human interaction
Cloud Automation is Key to achieving ScaleHypervisors, CISCO UCS, SDN, Load Balancers, Firewalls, Netscalersetc can all be Orchestrated
Example specs of a 16 Host PODNote – 4 Socket Hosts, but could equally be 32 Host POD using lower Dual SocketHosts – same total RAMXenServer and VMware have limits on max number of LUNs per Cluster
Capacity based on Linux VMs with 1vCPU, 4GB RAM and 20GB Disk,Using Redundant VR Feature – Public Cloud
Real World Design – not an exampleOriginal vendor replaced with alternate due to lack of performance
6 Arrays Deployed – ½ Peta-Byte of excess capacity600VMs20GB per VM30 IOPS per Vm150 VRs2GB per VR10 IOPS per VR
Highlight the last ParagraphStorage was being designed by Enterprise Storage Team based on IOPS requirements
Customer Example vs. NetAppCustomer needed to deploy 11,000 VM over 8 months.Took advantage of high deduplication rates and SF cloning technology and achieved the following results.SolidFire CapEx Spend is $2.45M less than NetAppSolidFire is 93% more dense than NetApp14 racks are not needed110 sq. ft. of data center space available for other useEquates to $264,000 in floor space savings97% fewer kWh used resulting in cost avoidance of $2.14 million over 5 years in power costsThis comparison is with a FAS6240AEShelf ConfigsFAS6240AE with 10GB UTADisk Shelves Disk Type:900GB 10k SAS Controllers4 (2 per controller) NIC #1 - HBA SAS 4-Port Copper 3/6 Gb QSFP PCIe,-C 2 (1 per controller) 512GB Flashcache4 (2 per controller) NIC #2-ADPT 2-Pt Unified Tgt 10GbE SFP+ PCIe,EN,-C Cabinet Type 42U NetApp Cabinet 32amp single phaseShelves = 2 U 24 x 900GB 10k SAS Controllers = 3 U with 512 GB FlashCache 1 per controller