Deploying Flash Storage For Ceph With High Performance Networking

Ceph Day Chicago – August 16, 2015
Deploying Flash Storage For Ceph
Rob Davis – robd@mellanox.com

© 2015 Mellanox Technologies 2
End-to-End Networking Solutions
Storage
Front / Back-End
Server / Compute Switch / Gateway
56-100G InfiniBand
10-100GbE
Host/Fabric SoftwareICs Switches/GatewaysAdapter Cards Cables/Modules
Comprehensive End-to-End InfiniBand and Ethernet Portfolio
Metro / WAN
56-100G InfiniBand
10-100GbE

Embedded Inside Storage Platforms Across the Industry
SMB Direct

Disk Drive Shipments Continue to Grow

Solid State Disk Growth
Today

HDD vs. SDD cost projections
Source:Wikibon2014,fromnumeroussources

The Performance Difference is Dramatic
0.1
10
1000
HD SSD NVM
AccessTime(micro-Sec)
Storage Media Technology
AccessTimeinMicroSeconds
Hard
Drives
NAND
Flash
Next Gen
NVM

Four Year Life Cycle TCO – HDD vs. SSD
Source:Wikibon2014,fromnumeroussources

Ceph and Network Traffic
APIs enable access to object, block & file data Fully distributed scale out architecture
Source: http://ceph.com/docs/master/rados/configuration/network-config-ref/

Cluster Network Traffic – Sunny Day Scenarios
Client OSD
Write
Write Ack
OSD OSD
Replications
N
Write Operation
Client OSD
Read
Read Reply
OSD OSD
N
READ Operation
No extra cluster network traffic Typically 2x cluster network traffic
(N-1)∙x

Flash Can Move the Bottleneck from the Disk to the Network
Network I/O should match Disk I/O

Where best to plug in?

Classic Network Architecture

New Leaf-Spine Architecture
Greater Cross-sectional Bandwidth

Recovery backend traffic 1x times lost data
Cluster Network Traffic – Recovery (Replication)
Example - Time to recover
 Net networking time to move data
 10GbE
• 20TB system @10GE 4.4hrs
 40GbE
 100GbE
• 20TB system 100GE 0.4hrs
Client OSD OSD OSD OSD
Read
Read Reply

The Effect of Replication vs. Erasure Coding

Cluster Network Traffic – Sunny Day Scenarios (Erasure Coding)
Client OSD
Read
Read Reply
K
OSD OSD OSD OSD OSD
M
Read
Shards
decode
READ Operation
Client OSD
Write
Write Ack
K
OSD OSD OSD OSD OSD
M
Write
Shards
encode
Write Operation
~1x cluster network traffic
((k-1)/k)∙x
Typically ~1.4x cluster network traffic
((k+m-1)/k) ∙x

Ceph with Flash is Best with a Separate Cluster Network
 High performance networks enable maximum cluster availability
• Clients, OSD, Monitors and Metadata servers communicate over multiple network layers
• Real-time requirements for heartbeat, replication, recovery and re-balancing
• Flash only adds to the network performance requirement
 Cluster (“backend”) network performance dictates cluster’s performance and scalability
• “Network load between Ceph OSD Daemons easily dwarfs the network load between Ceph Clients
and the Ceph Storage Cluster” (Ceph Documentation)

Ceph Flash Deployment Using Cluster Network
 Cluster (Private) Network @ 40GbE
• Smooth HA, unblocked heartbeats, efficient data balancing
 Clients @ 1, 10 or 40GbE
• 40 GbE guaranties line rate for high ingress/egress clients
 IOPs and Throughput to Clients @ 1, 10, or 40GbE
2.5x Higher Throughput , 15% Higher IOPs with 40Gb Ethernet vs. 10GbE
(http://www.mellanox.com/related-docs/whitepapers/WP_Deploying_Ceph_over_High_Performance_Networks.pdf)
Cluster Network
Admin Node
40GbE
Public Network
10GbE/40GBE
Ceph Nodes
(Monitors, OSDs, MDS)
Client Nodes
10GbE/40GbE

Reference Architecture for Ceph and Flash
 Red Hat, Supermicro, Seagate, Mellanox, Intel
 Extensive Ceph Performance Testing: Disk, Flash, Network, CPU
 Reference Architecture Published
40GbE Network
Setup
 Key Test Results
• 2 SSDs as fast as many disks, more flash is
faster
 40GbE Advantages
• Up to 2x read throughput per server
• Up to 50% decrease in latency
http://www.redhat.com/en/resources/red-hat-ceph-storage-clusters-supermicro-storage-servers

Ceph Flash Optimized Solution
Highlights Compared to Stock Ceph
• Read performance up to 8x better
• Write performance up to 2x better with tuning
Optimizations
• All-flash storage for OSDs
• Enhanced parallelism and lock optimization
• Optimization for reads from flash
• Improvements to Ceph messenger
Tested Configuration
• InfiniFlash Storage with IFOS 1.0 EAP3
• Up to 4 RBDs
• 2 Ceph OSD nodes, connected to InfiniFlash
• 40GbE NICs from Mellanox
SanDisk InfiniFlash

SanDisk InfiniFlash, Maximizing Ceph Random Read IOPS
Random Read IOPs Random Read Latency (ms)
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
25% Read 50% Read 75% Read 100% Read
8KB Random Read, QD=16
Stock Ceph IF-500
0
2
4
6
8
10
12
14
25% Read 50% Read 75% Read 100% Read
8KB Random Read, QD=16
Stock Ceph IF-500
http://www.sandisk.com/about-sandisk/press-room/press-releases/2015/sandisk-creates-new-storage-category-with-infiniflash%E2%84%A2-all-flash-storage-system/

Ceph/Flash Media & Entertainment Solution
 Turnkey Object Storage
• Built on Ceph
• Pre-configured for rapid deployment
• Mellanox 10/40GbE networking
 High-Capacity Configuration
• SSDs and 6-8TB Helium-filled drives
• Up to 2PB in 18U
 High-Performance
• Single client read 2.2 GB/s
• SSD caching + Hard Drives
 www.storagefoundry.net

Flash Performance Creates Bottleneck in Network Software
50%
100%
Networked Storage
Storage Protocol (SW) Network
Storage Media
Network
HDD
SSD
Storage
Media
0.01
1
100
HD SSD NVM
FC, TCP RDMA RDMA+
AccessTime(micro-Sec)
Protocol and Network
NVM
HDD
The
Network
and the
Protocol
MUST get
faster

NVMe Express – Protocol Designed for Flash
 NVMe: Optimized for flash and next-gen NV-
memory
• Traditional SCSI interfaces designed for spinning disk
• NVMe bypasses unneeded layers
 NVMe Flash Outperforms SAS/SATA Flash
• 2x-2.5x more bandwidth, 40-50% lower latency, Up to 3x more
IOPS

RDMA Enables More Efficient Networking for Ceph
Higher Bandwidth
Lower Latency
More CPU Power For
Applications

Adding RDMA to Ceph
 RDMA Beta Included in Hammer
• Open Source: Mellanox, Red Hat, CohortFS, and Community collaboration
• Full RDMA expected in Infernalis
• New RDMA messenger layer called XioMessenger, built on Accelio (RDMA abstraction layer)

 Open source
• https://github.com/accelio/accelio/ && www.accelio.org
 Faster RDMA integration to application
 Asynchronous
 Maximize msg and CPU parallelism
• Enable >10GB/s from single node
• Enable <10usec latency under load
 In Giant and Hammer
• http://wiki.ceph.com/Planning/Blueprints/Giant/Accelio_RDMA_Messenger
Accelio, High-Performance Reliable Messaging Library

RDMA Enables Efficient Data Movement for Windows SDS
 Without RDMA
• 5.7 GB/s throughput
• 20-26% CPU utilization
• 4 cores 100% consumed by moving data
 With Hardware RDMA
• 11.1 GB/s throughput at half the latency
• 13-14% CPU utilization
• More CPU power for applications, better ROI
x x x x
100GbE With CPU Onload 100 GbE With Network Offload
CPU Onload Penalties
• Half the Throughput
• Twice the Latency
• Higher CPU Consumption
2X Better
Bandwidth
Half the Latency
33% Lower CPU
See the demo: https://www.youtube.com/watch?v=u8ZYhUjSUoI

Conclusions
 Flash storage is main stream and approaching price parity with hard disks
 The performance difference between flash and hard disks is dramatic and with the
right network, Ceph can easily leverage this
 Its important to balance the storage and network performance in OSDs
 A separate high performance “Cluster” network is usually needed to realize the full
performance advantage of flash in Ceph systems
 RDMA can further optimize Ceph flash performance

Deploying Flash Storage For Ceph With High Performance Networking

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Deploying Flash Storage For Ceph With High Performance Networking

Similar a Deploying Flash Storage For Ceph With High Performance Networking (20)

Último

Último (20)

Deploying Flash Storage For Ceph With High Performance Networking

Notas del editor