2. Switch your database to flash
now. Or you’re doing it wrong.
Brian Bulkowski, Aerospike CTO and co-founder
July 8, 2013 2
http://highscalability.com/blog/2012/12/10/switch-your-
databases-to-flash-storage-now-or-youre-doing-it.html
5. NAND Flash Memory
July 8, 2013 5
Flash is a persistent memory technology invented by
Dr. Fujio Masuoka at Toshiba in 1980.
Bit
Line
Source
Line Word Line
Control Gate
Float Gate
NPN
7. Direct Cut Through Architecture
7/8/2013 Fusion-io Confidential 7
PCIe
DRAM
Host
CPU
App
OS
LEGACY APPROACH FUSION DIRECT APPROACH
PCIeSAS
DRAM
Data path
Controller
NAND
Host
CPU
RAID
Controller
App
OS
Goal of every I/O operation to move data to/from DRAM and flash.
SC
Super
Capacitors
11. Working Set Getting Bigger
July 8, 2013 11
▸ Key Design Principle:
▸ Working Set < DRAM
12. EEP
July 8, 2013 12
▸ Key Design Principle:
▸ Working Set < DRAM
13. DOLLARS
Cost of DRAM Modules
July 8, 2013 13
0
200
400
600
800
1000
1200
1400
1600
4GB 8GB 16GB 32GB
$ $$
$$$
$$$$$$
14. HBase Server
July 8, 2013 14
▸ A typical server…
CPU Cores: 32 with HT
Memory: 128 GB
Is your working set larger than 128GB?
15. HBase Cluster
July 8, 2013 15
▸ With NoSQL Databases, we tend to scale out for
DRAM
Combined Resources
CPU Cores: 96
Memory: 384 GB
More cores than needed to serve reads and writes.
16. The All Flash Option
July 8, 2013 16
HBase Node
Data Node
Daemon
Region Server
Daemon
Memory Store Storage Directories
Disk Flash
HBase Node
Data Node Daemon
Storage Directories
20. YCSB Suite Latency – Uniform Distribution
July 8, 2013 20
Workload Average
Latency
95th Percentile
Latency
99th Percentile
Latency
50/50 Read/Write
Update Latency
81 µs 0 ms 0 ms
50/50 Read/Write
Read Latency
13.5 ms 34ms 128ms
95/5 Read/Write
Update Latency
69.3 µs 0 ms 0 ms
95/5 Read/Write
Read Latency
8.5 ms 26 ms 39 ms
Read-only 9.2 ms 26 ms 38 ms
21. Write Amplification
July 8, 2013 21
Workload Type Amplification Factor
Bulk Load 14.8
Normal Operations
(80/20 update/insert split)
4.2
Amplification Factor =
Physical Bytes Written
Workload Bytes Written
22. The HBase BucketCache (HBase-
7404)
July 8, 2013 22
Committed to HBase trunk. Will be in 0.96 release, backport patch for
0.94 available.
+
https://issues.apache.org/jira/browse/HBASE-7404
23. BucketCache Configuration
July 8, 2013 23
▸ In hbase-site.xml
<property> <name>hbase.bucketcache.ioengine</name> <value>file:/path
/to/bucketcache.dat</value> </property>
<property> <name>hbase.bucketcache.size</name> <!-- 2TB: unit is MB
-->
<value>2097152</value>
</property>
28. What Next?
July 8, 2013 28
1. Can we do something about that write
amplification?
2. There is minimal penalty to in-place updates in
flash on modern FTLs; can we devise a way to do this
compatible with HDFS?
3. Does HDFS need to be more aware of different
storage technologies? (DRAM, Flash, PCM)
29. f u s i o n i o . c o m | R E D E F I N E W H A T ’ S P O S S I B L E
THANK YOU
f u s i o n i o . c o m | R E D E F I N E W H A T ’ S P O S S I B L E
THANK YOU