4. • When any namespace modification is performed
it durably logs a record of the modification to JNs
• The Standby reads the edits from the JNs and applies
them to its own namespace
JournalNodes’ job
7/26/2013 Copyright 2013 Trend Micro Inc.
JN
1
JN
2
JN
3
NN
Active
NN
Standby
Edits Edits Edits
Edits Edits EditsEdits Edits Edits
Safe
Mode
5. • Specify path on local disk
• tolerate at most (N - 1) / 2 failures
JournalNodes’ storage
7/26/2013 Copyright 2013 Trend Micro Inc.
6. • JournalNodes will only allow a single NameNode to be a
writer at a time.
• no potential for corrupting the file system metadata from
a split-brain scenario.
JournalNodes’ fencing
7/26/2013 Copyright 2013 Trend Micro Inc.
JN
1
JN
2
JN
3
NN
Active
NN
Standby
WRITE READ
7. • Whenever a NameNode becomes active, it first
generate an epoch number.
• first active NameNode after the namespace is initialized
starts with epoch number 1
• any failovers or restarts result in an increment of the
epoch number
JournalNodes’ fencing
7/26/2013 Copyright 2013 Trend Micro Inc.
8. • When a new NameNode becomes active, it has an
epoch number higher than any previous NameNode
• Call JournalNodes to increment their promised epochs
• Fencing:
– JNs receive newer epoch
update majority of JNs’ promised epochs accept
– JNs receive older epoch
reject
JournalNodes’ fencing
7/26/2013 Copyright 2013 Trend Micro Inc.
9. • previous Active NameNode could serve read requests
to clients which may be out of date until a write access
performed
• You can specify some fencing method to avoid this
happened
But…
7/26/2013 Copyright 2013 Trend Micro Inc.
11. • sshfence
SSH to the Active NameNode and kill the process
Fencing Method
7/26/2013 Copyright 2013 Trend Micro Inc.
12. • shell
run a shell command to fence the Active NameNode
• The script may have properties with the '_' character
replacing any '.'
ex : dfs_namenode_rpc-address
Fencing Method
7/26/2013 Copyright 2013 Trend Micro Inc.
16. • Health monitoring
– the ZKFC pings its local NameNode on a periodic basis with a
health-check command. (healthy/unhealthy)
• ZooKeeper session management
– when the local NameNode is healthy, the ZKFC holds a session
open in ZooKeeper.
– If the local NameNode is active, it also holds a special "lock"
znode.
– if the session expires, the lock node will be automatically
deleted.
ZKFailoverController
7/26/2013 Copyright 2013 Trend Micro Inc.
17. • ZooKeeper-based election
– if the local NameNode is healthy, and no other node currently
holds the lock znode, it will itself try to acquire the lock.
– If it succeeds, then it has "won the election“
• Failover
– the previous active is fenced
– local NameNode transitions to active state.
ZKFailoverController
7/26/2013 Copyright 2013 Trend Micro Inc.
20. • Client connect to Active Namenode via proxy
• When Active Namenode down, client receive Exception
retry and send RPC to another namenode
(implement by ConfiguredFailoverProxyProvider)
Client Failover
7/26/2013 Copyright 2013 Trend Micro Inc.
22. • If setting up a fresh HDFS cluster,
hdfs namenode –format
• copy over the contents of your NameNode metadata
directories to the other
hdfs namenode –bootstrapStandby
./format-failover-namenode.sh
• hdfs –initializeSharedEdits to initialize edits log in
journalnode
• Startup both Namenode
converting a non-HA-enabled cluster
to be HA-enabled
7/26/2013 Copyright 2013 Trend Micro Inc.
