Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.

1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. Amazon Elastic Block Store Deep Dive
•Larger and faster volumes

6. EBS !=

9. A few definitions…
•IOPS: Input/output operations per second (#)
•Throughput: Read/write rate to storage (MB/s)
•Latency: Delay between request and completion (ms)
•Capacity: Volume of data that can be stored (GB)
•Block size: Size of each I/O (k)

10. When performance matters, use SSD-backed volumes

12. •New default volume type for Amazon EBS
•Every volume can burst up to 3,000 IOPS
-Larger volumes can burst for longer periods
•3 IOPS per GB baseline performance
-Maximum of 3,000 on a 1TB volume
•99% performance consistency

13. (2) Max IO credit per bucket is 5.4M
(1) Always accumulating 3 IOPS per GB per second
(3) You can spend up to 3000 IOPS per second 13

14. 14

15. 15

16. 16

17. VolumeType
BootTime
AccessTime
OS
GP2
3:31
4:33
WindowsServer 2012
Standard
4:30
7:16
WindowsServer 2012
GP2
0:36
0:45
CentOS6
Standard
0:57
1:16
CentOS6
40% Reduction in boot times by using General Purpose (SSD)

18. 80% Cost Savings. 50% more peak I/O with
General Purpose (SSD)

19. 19

20. •Best for I/O intensive databases that require highest consistency
•Provision up to 4,000 IOPS per volume
•Supports IOPS to GB ratio of 30
•Designed to offer 99.9% consistency

21. •Best for cold workloads
•Rarely accessed data that needs always on access
•IOPS: ~100 IOPS steady-state, with best-effort bursts
•Throughput: variable by workload, best effort to 10s of MB/s
•Latency: Varies, reads typically ~20-40 ms, writes typically ~2-10ms

22. General Purpose (SSD)
Provisioned IOPS (SSD)
Magnetic
Recommend use cases
Boot volumes
Small to med DBs Dev and test
I/O intensive
Large DBs
Cold storage
Storage media
SSD-backed
SSD-backed
Magnetic-backed
Volume size
1GB-1TB
1GB-1TB
1GB-1TB
Max IOPS per volume
3 IOPS/GB Burst up to 3000 IOPS
4,000 IOPS
~100 IOPS
Read and write peak throughput
128 MB/s
128 MB/s
~50-90 MBps
Max I/O per node(16k)
48,000
48,000
48,000
Peak throughput node
800 MB/s
800 MB/s
800MB/s
Latency (random read)
1-2ms
1-2 ms
20-40 ms
API name
gp2
io1
standard
Price*
$.10/GB-Month
$.125/GB-Month $.065/provisioned IOPS
$.05/GB-Month $.05/ 1M I/O

24. Always use General Purpose (SSD) for boot volumes

29. •LAW
–L
–A
–W

30. 1.IOPS:Read/write I/O rate (IOPS)
2.Latency:Time between I/O submission and completion (ms)
3.Throughput:Read/write transfer rate (MB/s)
–Throughput = IOPS X I/O Size

31. (1) Amazon EC2
(2) I/O
(4) Amazon
EBS
(3) Network
link

32. 1.EC2 instance:Network bandwidth(Mbps)
2.EBS-optimized:EC2 instance option (On/Off)
3.Workload:Block size, read/write ratio, serialization
4.Queue depth:The number of outstanding I/Os
5.RAID:Stripe volumes to maximize performance
6.Pre-warm:Eliminates first touch penalty

33. 1. Amazon EC2 instance
• Compute optimized– C3
• Memory optimized- R3
• General Purpose – M3
Amazon
EBS
Amazon
EC2
Select Amazon EC2 instance that has the right network, RAM, and
CPU resources for your applications.

34. 2. Amazon EBS-optimized
b

35. Use EBS-optimized instances for consistent EBS performance

36. 3. Workload
EBS SSD-backed volumes measure I/O size up to 256KB
EBS SSD-backed delivers same performance for read and write
NEW

37. 4,000 IOPS
PIOPS volume
4,000 IOPS
128MB/s throughput
You can achieve 4,000 IOPS
when driving smaller I/O
You can achieve up to 128MB/s
when driving larger I/O

38. 4,000 IOPS
PIOPS volume
4,000 IOPS
128MB/s throughput
4,000 X 4KB = 16MB/s
4,000 X 32KB =128MB/s
8,000 x 8KB = 64MB/s
4,000 x 8KB = 32MB/s
2,000 x 128KB=256MB/s
1,000 x 128KB =128MB/s

39. Block (I/O) size determines whether your application is IOPS bound or throughput bound

40. 4. Queue depth
An I/O
Amazon
EBS
After it’s gone, it’s gone
Amazon
EC2
Queue depth is the pending I/O in flight

41. 41

42. I/O Latency

43. 0.075
35.1
0
5
10
15
20
25
30
35
40
1
4
8
12
16
20
24
28
32
Latency TP90 ( ms)
Queue Depth
Random READ latency across various QDs
Latency (TP90)
Read latency linearly increases with increase in queue depth

44. 0.075
35.1
2.09
1,865
4,152
3,851
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
5
10
15
20
25
30
35
1
4
8
12
16
20
24
28
32
Latency TP90 ( ms)
Queue Depth
16 KB random READ IOPS, Latency across various QDs
Latency (TP90)
Avg Read IOPS
IOPS
Queue depth of one has the lowest latency, but has the lowest IOPS

45. 0.075
35.1
2.09
1,865
4,152
3,851
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
5
10
15
20
25
30
35
1
4
8
12
16
20
24
28
32
Latency TP90 ( ms)
Queue Depth
16 KB random READ IOPS, Latency across various QDs
Latency (TP90)
Avg Read IOPS
IOPS
Queue depth between 4-8 has the optimal IOPS and latency performance

46. 0.075
35.1
2.09
1,865
4,152
3,851
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
5
10
15
20
25
30
35
1
4
8
12
16
20
24
28
32
Latency TP90 ( ms)
Queue Depth
16 KB random READ IOPS, Latency across various QDs
Latency (TP90)
Avg Read IOPS
IOPS
Higher queue depths have negative impact on IOPS and latency

47. 0.08
7.71
845
4,152
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
1
2
3
4
5
6
7
8
9
10
1
4
8
12
16
20
24
28
32
Latency TP90 ( ms)
Queue Depth
16 KB random WRITE IOPS, Latency across various QDs
Latency (TP90)
AvgIOPS
IOPS
Write latency, queue depth and IOPS interaction is similar to that of read latency

48. Optimal queue depth to achieve lower latency and highest IOPS is typically between 4-8; ~1 QD per 500 IOPS
Amazon EBS-optimized offers consistent latency experience

49. 5. RAID
• Increase performance or capacity or both
• Stripe multiple volumes to achieve 10’s of TB and up-to
48,000 IOPS per Instance
• Don’t mix volume types
• Typically RAID 0 or LVM stripe
• Avoid RAID for redundancy Amazon
EBS
Amazon
EC2

50. Maximum performance per instance
12×400 GB PIOPS, pre-warmed, RAID 0 LVM, Stripe size 128 KB, attached to CR1 instance

51. Use stripe size of 128KB or 256KB

52. 6. Pre-warminghttp://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html

53. Use large block size to speed up your pre-warming.
ex: sudoddif=/dev/xvdfof=/dev/xvdfconv=notruncbs=1M

54. Final tips
•Ext4 or XFS (understand journal impact!)
•Amazon EBS encryption has no visible IOPS performance impact for many Amazon EC2 instances
•Alignment can matter, check tools use 4k

55. Workload/ Software
Typical block size
Random/ Seq?
Max EBS @ 500 Mb/s
Max EBS @ 1 Gb/s
MaxEBS @ 10Gb/s instances
Oracle DB
Configurable:2 KB–16 KB
Default 8 KB
random
~7800 IOPS
~15,600 IOPS
~96,000 IOPS
Microsoft SQL Server
8 KB w/ 64 KB extents
random
~7800 IOPS
~15,600 IOPS
~80,000 IOPS
MySQL
16 KB
random
~4000 IOPS
~7,800 IOPS
~48,000 IOPS
PostgreSQL
8 KB
random
~7,800 IOPS
~15,600 IOPS
~96,000 IOPS
MongoDB
4 KB
serialized
~15,600 IOPS
~31,000 IOPS
~96,000 IOPS
Cassandra
4 KB
random
~15,600 IOPS
~31,000 IOPS
~96,000 IOPS
GlusterFS
128 KB
sequential
~500 IOPS
~1000 IOPS
~6,000 IOPS

56. EBS-optimized
Oh, YEAH!!
Amazon
EC2
A “boatload” of I/O
Right sized EBS

57. 1.EC2 instance:Network transfer rate (Mbps)
2.EBS-optimized:EC2 instance option (On/Off)
3.Workload:Block size, read/write ratio, serialization
4.Queue depth:The number of outstanding I/Os(#)
5.RAID:Stripe volumes to maximize performance
6.Pre-warm:Eliminates first touch penalty

59. Performance
(per volume)
GP2
PIOPS
GP2
PIOPS
Max Size
1 TB
1 TB
16 TB
16 TB
Max IOPS
3,000
4,000
10,000
20,000
Max Throughput
128 Mbps
128 Mbps
160 MBps
320 MBps
PREVIOUSLY
W/ LARGER & FASTER VOLUMES

62. Amazon EBS larger and faster volume types
General Purpose (SSD)
Provisioned IOPS (SSD)
Magnetic
Recommend use cases
Boot volumes
Small to med DBs Dev and test
I/O intensive
Large DBs
Cold storage
Storage media
SSD-backed
SSD-backed
Magnetic-backed
Volume size
1GB-16TB
4GB-16TB
1GB-1TB
Max IOPS per volume
10,000 IOPS
20,000 IOPS
~100 IOPS
burst
< 1TB to3000 IOPS
baseline
baseline
Read and write peak throughput
160 MB/s
320 MB/s
~50-90 MBps
Max IOPS per node (16k)
48,000
48,000
48,000
Peak throughput node
800 MB/s
800 MB/s
800MB/s
Latency (random read)
1-2ms
1-2 ms
20-40 ms
API name
gp2
io1
standard
Price*
$.10/GB-Month
$.125/GB-Month
$.065/provisioned IOPS
$.05/GB-Month
$.05/ 1M I/O

64. 64

66. 66

67. • 2-tiered key hierarchy using envelope
encryption
• Unique data key encrypt customer data
• AWS KMS master keys encrypt data keys
• Benefits of envelope encryption:
• Limits risk of a compromised data key
• Better performance for encrypting large data
• Easier to manage a small number of master
keys than millions of data keys
Master Key(s)
Data Key 1
S3 object EBS
volume
Redshift
cluster
Data Key 2 Data Key 3 Data Key 4
Custom
application
AWS KMS

68. Select the right volume for your workload
Select the right instance for your workload
Take snapshots
Use encryption if you need it

69. Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals