This document proposes a software-defined approach called SDPM (Software-Defined Persistent Memory) to abstract the heterogeneity of emerging persistent memory technologies and enable their use across different hardware configurations. It describes SDPM's design goals of supporting various local and remote persistent memory attach points while providing a unified programming model. The proposed architecture introduces a persistent memory manager and a file system to manage data placement and provide memory-like and storage-like access. An evaluation shows the prototype delivering near-optimal performance for local and remote persistent memory configurations.

1. Towards Software-defined Persistent Memory:
Rethinking Software Support for Heterogeneous Memory Architectures
Swaminathan Sundararaman*
NishaTalagala*
Dhananjoy Das Amar Mudrankit*
Dulcardo Arteaga*
*Work done at Fusion-io/SanDisk

2. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 2
Memory-Storage Convergence (Trend 1)
L1, L2, L3 CPU Caches DRAM Hard Drives
MicrosecondsNanoseconds
CPU WAIT CYCLES
Tiered Memory Solutions
Main Memory System Storage Systems
Milliseconds
ACCESS DELAY
2 cycles 1,000,000s100s 1,000s 10,000s
chasm
2
Flash
Memory
Persistent
Memories
PM blurs the line between storage and memory

3. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 3
Challenges with Current Persistent Memory Solutions
Access Granularity Byte (Memory) Block (I/O) Hybrid
Memory Technology PCM
ReRAM/
Memristor
SRAM
(backed by Cap.)
NVDIMM
Capacity 1 - 100s GB 1 - 100s GB 32K – 2GB 4 – 32GB
Local Attach Point PCIe NVMe SAS DDR
Access Mechanism File System Object Store KV Store …
Memory Location Local Remote Replicated
Network Connect Infiniband Ethernet PCIe …
3
Many possible combinations!

4. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 4
• Rewrite applications for different deployments
▪ Not practical given the number of scenarios
• What about existing applications / deployments?
• User data is constantly growing and needs not fit in persistent
memory
What Should Application Developers Do?
4

5. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 5
Moving Towards a Software-defined World… (Trend 2)
Software-Defined Networking (SDN)
Enables administrators to manage network services
by abstracting higher level functionality
Abstraction of logical storage services and capabilities
from the underlying physical storage systems
Software-Defined Storage (SDS) Software-Defined Flash (SDF)
Abstract or expose flash specific details to enable software to
realize the raw bandwidth and storage capacity of Flash
Software-Defined Data Center (SDDC)
All elements of the infrastructure such as networking, storage,
CPU and security are virtualized and delivered as a service
5
Abstraction of logical storage services and capabilities from the
underlying physical persistent memory hardware and interconnect
Software-Defined Persistent Memory (SDPM)

6. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 6
Our SDPM Solution
• The first instance of a software defined approach to PM that can bring
the benefits of PM to a gamut of practical deployments.
• Abstract the heterogeneity in PM hardware from applications
• Provide file system API & programming libraries to access PM
• Use currently available PM hardware to show the feasibility of an
SDPM
▪ PCIe & DDR4 attached PM (both local & remote attach)
▪ Using Infiniband & 10G Ethernet for remote access
• The prototype architecture provides good performance and near
optimal acceleration for a range of local and remote PM configurations
6

7. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 7
• Introduction
• Design
• Evaluation
• Conclusion
Outline
7

8. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 8
• Support a variety of local and remote attach points with differing
performance but identical functionality and semantic guarantees.
• Enable tiering of data between PM and flash, with caching in DRAM,
to enable different cost/performance configurations
• Support hybrid (i.e., both memory and I/O) access, traditional
storage management, and persistence guarantees to combine the
best of memory and storage worlds.
• Enable a single application programming model to work across a
variety of hardware
SDPM Design Goals
8

9. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 9
Software Defined Persistent-Memory Architecture
Applications
Programming
Libraries
File System
Persistent Memory
Manager
Block Device
Driver
SDPM
PCIe/
DDR
SAS/
SATA/
NVMe
PM
Infiniband/
Ethernet/
PCIe/…
Memory/
Block (I/O)
• Persistent Memory Manager
▪ Abstracts hardware and interconnect
details from the file system / applications
▪ Exports APIs to guarantee persistence
• File System
▪ Unified persistent namespace to PM &
Flash
▪ Hybrid access to PM
▪ Transparent & non-transparent
acceleration
• Programming Libraries
▪ Unified access APIs to applications
▪ OS bypass for remote access

10. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 10
• A flash-optimized POSIX compliant Linux file system
▪ Extended NVMFS to support PM in addition to Flash
• Provides a unified and persistent namespace to both PM and flash
▪ Hybrid (memory & I/O) access to PM and applications can switch back and
forth
▪ Transparent application acceleration (by tiering data between PM & flash)
▪ Supports “direct” mmap mode to directly map and use PM without caching in
DRAM
▪ Supports a single programming model via a combination of application specific
libraries over direct mmap and transparent access via POSIX APIs
Non-Volatile Memory File System (NVMFS)
10

11. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 11
Persistent Memory Manager: Hardware Abstractions
• Memory Mapping Types
▪ PM can be mapped in multiple ways depending on the hardware. We need to
ensure that each memory type is default mapped to the optimal model possible
for its physical attach (by default map it to write combining)
▪ Enable FS operations that allow the app. to control the per file memory mapping.
• Guaranteeing Persistence
▪ We need mechanism(s) to guarantee all acknowledged in-flight data (such as in
CPU caches, registers, etc.) have reached the PM device independent of its
attach point
▪ PMM provides a barrier() operation (to NVMFS and user space libraries) that
guarantees that all data is moved to the persistence domain as needed for the
attach point
11

12. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 12
• Introduction
• Design
• Evaluation
• Conclusion
Outline
12

13. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 13
Evaluation - Configuration
Configurations Attach Method Local or Remote
Config-1 DDR NVDIMM Local
Config-2 PCIe MMIO Local
Config-3 DDR NVDIMM Remote Ethernet
Config-4 PCIe MMIO Remote Ethernet
Config-5 DDR NVDIMM Remote Infiniband
Config-6 PCIe MMIO Remote Infiniband
Config-1, Config-2
(Local)
Config- 3, 4, 5, 6
(Remote)
System Configuration HP DL380, 96GB DDR DRAM, x86_64 Linux 3.14 kernel
MySQL Version Percona 5.5
Flash Device (PCIe) Fusion-io Gen 2 ioMemory 1.2TB
Persistent Memory PCI-e: ACM (512KB)
DDR4: Viking NVDIMM (8GB)
Network Interconnect N/A Infiniband: ConnectX-3 56 Gbit IB Ethernet:
Intel 82599ES 10-Gigabit
13

14. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 14
Evaluation - Setup
Host A
(Source)MySQL
Programming
Libraires
NVMFS PMM
ioMemory VSL
Replication Src
PM
SDPM
Local
Config-2
Config-1
Host B
(Sink)
Programming
Libraires
NVMFSPMM
ioMemory VSL
Replication Sink
(daemon)
PM
SDPM
Config-4
Config-3
10Gbit Ethernet
56Gbit Infiniband
Remote
14 Config-5
Config-6

15. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 15
Local PCIe MMIO Vs DDR4 (config-1 Vs Config-2)
Bandwidth(MB/sec)
0
1000
2000
3000
4000
Size (bytes)
64 128 256 512 1024 2048 4096
DDR-BW MMIO-BW
Latency(nsec)
0
2000
4000
6000
8000
Size (bytes)
64 128 256 512 1024 2048 4096
DDR-Latency
MMIO-Latency
15
Smaller data sizes -> barrier operation
Larger data sizes -> transport media

16. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 16
Barrier Overheads: Local (config-1) Vs Remote (config-3 & config-5)
Latency(us)
0
12500
25000
37500
50000
Synchronization Frequency (#Ops)
1 10 100 1000
Ethernet FLUSH InfiniBand RDMA Local ACMPM
16
20x
4x
# of operations between synchronizations increases, the
performance becomes closer

17. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL
0
15,000
30,000
45,000
60,000
REMOTE LOCAL VANILLA NO_LOG
Bin & Tr Log Tr log ONLY
MySQL: Insert Heavy Workload
17
Config-3 Config-1 on Flash Infinitely fast logging

18. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL
Operations/sec(in
x1000)
0
7500
15000
22500
30000
MySQL: LinkBench 10x Workload
18
No Logs Vanilla Local
(config-1)
Remote
(config-5)
31%
17%
Facebook’s social graph workload
10x: 100 million nodes
Inserts, delete, update, and lookup
30% writes (insert/update/delete)
70% reads (lookup)
Infiniband performance for small updates is sufficient
3%

19. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 19
• Introduction
• Design
• Evaluation
• Conclusion
Outline
19

20. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL
Conclusions
• PM is going to change the storage-memory landscape
• Many different forms / capacities / attach points / performance
• SDPM: a software-defined approach to using persistent memory
• Abstracts heterogeneity in memory hardware
• Applications can transparently run on local & remote persistent memory
• Selectively abstracts PM characteristics to provide optimal performance
• Transparently tier data between PM & Flash to exploit both existing & new
applications written to run on PM.
• Our evaluation shows near optimal performance for local & remote attach PM
20

21. CONFIDENTIAL
PARALLEL MACHINES CONFIDENTIAL 21
Thank You
Iif you are interested in trying out SDPM: Dhananjoy.Das@sandisk.com