Introduction to Memory-Style Storage in Linux (COSCUP 2017 Taipei, Taiwan)

1. Introduction to Memory-
Style Storage in Linux
Clay Chang
COSCUP’17
6-Aug-2017

2. About Myself
• Senior System Software Architect at Hewlett Packard Enterprise
• Ph.D. Candidate at NTU CSIE
• Father of

3. Agenda
• Traditional Storage vs Memory-Style Storage
• Introduction to Persistent Memory / NVM / SCM
• PMEM Support in Linux
• Emulating PMEM in Linux
• References

4. Traditional Storage Stack
• Application
• VFS
• Page Cache
• Block Layer
• SCSI Upper-Level Drivers
• SCSI Low-Level Drivers
• Physical Devices
Application
VFS
(ext4, xfs, nfs, …)
Page Cache
Block Layer
(I/O scheduler, blkmq)
SCSI Upper-Level Drivers
(/dev/sd*, …)
SCSI Low-Level Drivers
(libata, ahci, …)
Physical Devices
(HDD, SSD, …)
open
read
write
close
….
mmap
Block I/Os (BIOs)
Driver Requests
* https://www.thomas-krenn.com/en/wiki/Linux_Storage_Stack_Diagram

5. Memory-Style Storage
• Application
• VFS
• Page Cache
• Block Layer
• SCSI Upper-Level Drivers
• SCSI Low-Level Drivers
• Physical Devices (Persistent
Memory)
Application
VFS
(ext4, xfs, nfs, …)
Page Cache
Block Layer
(I/O scheduler, blkmq)
SCSI Upper-Level Drivers
(/dev/sd*, …)
SCSI Low-Level Drivers
(libata, ahci, …)
Physical Devices
(Persistent Memory)
open
read
write
close
….
mmap
Block I/Os (BIOs)
Driver Requests
Load/Store
* https://www.thomas-krenn.com/en/wiki/Linux_Storage_Stack_Diagram

6. Introduction to Persistent
Memory / NVM / SCM
• Emerging Memory Technology
• Intel 3D XPoint, Phase Change
Memory (PCM), Memoristor, etc…
• DRAM-like Performance
• Storage-like Persistence
• No refresh charge / Non-volatile
• High Capacity / Low Cost
• Byte-addressable
• NVM: Non-Volatile Memory
• SCM: Storage Class Memory

7. 107 - 108
105 - 106
NAND Flash
HDD
Mass storage
Mass storage, archive
Memory Hierarchy Shift in NVM Era
1
10
100
Register
SRAM
DRAM
L1 cache
L2,L3 cache
Main memory
Fast,
byte-addressable
Volatile
High refresh power
Slow
Large, low-cost,
non-volatile
No refresh
Memory-Storage GapNVM
ReRAM
PCM
3D XPoint
103 - 105
Fast,
byte-addressable
Large, low-cost,
non-volatile
NVM blurs the line between Memory and Storage

8. 3D XPoint
1000X
FASTERTHAN NAND
1000X
ENDURANCEOF NAND
10X
DENSERTHAN CONVENTIONAL MEMORY
Source: http://www.intel.com/content/www/us/en/architecture-and-technology/3d-xpoint-unveiled-video.html
Intel 3D XPoint

9. Storage Architecture Shift in NVM Era
• We can attach NVM to the traditional I/O
bus as a drop-in replacement for Storage
• Or, directly attach NVM to the fast memory
bus as Storage
• Best way to unlock the performance of NVM
• NVDIMM - Non-Volatile Dual-Inline Memory
Module
DRAM
CPU
Main Memory
Memorybus
Flash/
Disk
Northbridge Southbridge
Storage
I/Obus
NVM

10. Software Architecture for Persistent Memory

11. DAX-enabled File System (1/2)
• Page Cache
• System software to mediate between
fast memory and slow storage
• Elegant Design
• Read-ahead
• Write-back policy
• …
• Valid for decades until now,
• Becoming NEW bottleneck
Storage
Memory
Page CacheCPU
Read
Write

12. DAX-enabled File System (2/2)
• DAX-enabled
CPU
DRAM NVM
Memory Controller
cache
file
data
• Before DAX
CPU
DRAM NVM
Memory Controller
cache
file
data
Page Cache:
Un-necessary data copy
• DAX – Direct Access
• Use existing mmap
semantics
• In-place update
• No storage stack involved
• True device performance

13. Is DAX all enough?
• Simple answer: No
• In-place update
• Cannot do Journaling
• Out-place required for Journaling
• What? Crash consistency not guaranteed!

14. Crash Consistency
Crash!
strcpy(pmem, “Hello World!”);
• “Ensure that the file system keeps the on-disk image in a reasonable
state given that crashes can occur at arbitrary points in time.”
[Remiz14]
X Hello
V Hello World!
V

15. Load/store
How to ensure Crash Consistency?
Application
NVM-Lib*
NVM
mmap
Direct Access
* NVM-Lib: Helper Library for user-handled data consistency due to file system bypassing
Ext4-DAX maps the physical
page on the NVM to user space
directly
Consistent is not guaranteed
when system crash or power
loss; needs NVM-Lib
Ext4-DAX
Requires Program Change!
Big Issue!
TX_BEGIN(TX_LOCK_MUTEX, &op->lock) {
TX_STRCPY(buffer, ”Hello World!”);
} TX_END

16. Playing with DAX / Emulating PM in Linux
• PM is supported in Linux since
v4.2
• Can be emulated by DRAM
• Can test the new software
stack
• # vi /etc/default/grub
GRUB_CMDLINE_LINUX="memmap=16G!16G”
• # update-grub2
• # mkdir /mnt/pmem
• # mkfs.ext4 /dev/pmem0
• # mount -o dax /dev/pmem0 /mnt/pmem
• # echo ”Hello World!” > /mnt/pmem/Hello.txt
• # cat /mnt/pmem/Hello.txt

17. Key Takeaways!
• New memory technology makes storage ultra-fast!
• DRAM-like Performance
• Storage-like Persistence
• System Software becoming the NEW bottleneck!
• New Programming Model (sometimes may not be a good idea)
• New File System!
• New Operating System?
• Try it now!

18. References
• http://pmem.io
• http://pmem.io/2016/02/22/pm-emulation.html
• http://events.linuxfoundation.org/sites/events/files/slides/Managing
%20Persistent%20Memory_0.pdf
• http://snia.org
• https://www.usenix.org/system/files/conference/fast16/fast16-
papers-xu.pdf - NOVA
• [Remiz14] http://pages.cs.wisc.edu/~remzi/OSTEP/file-journaling.pdf

19. Questions?
Thank You!