1. HDFS
Fisher Liao
2013/01/17

2. Goals

Hardware Failure
 Streaming Data Access
 Large Data Sets
 Appending-Writes and File Syncs
 Hflush
 Append

Moving Comuptation
 Portable

3. NameNode & DataNodes

master/slave

4. File System Namespace

replication factor

5. Data Replication

block size/replication factor configurable per
file
 namenode receive Heartbeat/Blockreport
from datanodes

Heartbeat

Blockreport
 replica placement
 Policy

Rack

6. Data Replication(Cont.)

replica selection - closest to reader
 safemode(namenode)
 on startup

no replication

exit after namenode data block check > x%
 replicate

7. Persistence of File System
Metadata

Editlog
 FsImage
 Checkpoint
 datanode
 each block a file
 on starup, scan local > blockreport

8. Communication Protocol

TCP/IP
 ClientProtocol
 DataNode Protocol

9. Robustness

failures
 NameNode failure
 DataNode failure
 network partitions

data disk failure/heartbeats/re-replication
 cluster rebalancing - free space, threshold
 data integrity – checksum

meatadata disk failure
 snapshot(HDFS not support yet)

10. Data Organization

data blocks
 replication pipelining – write
1. namenode receive list of datanode by algorism
2. client write to 1st datanode
3. 1st datanode receive small portions(4KB)
4. 1st datanode copy this portion to 2nd datanode

11. Accessibility

API
 FS Shell
 DFSAdmin
 Browser

12. Space Reclamation

Delete
 Undelete
 decrease replication factor