Slides Tamc07

TAMC 2007 25 th May, 2007

A Distributed Algorithm of Fault
Recovery For Stateful Failover

Indranil Saha
HTS (Honeywell Technology Solutions) Research
Bangalore, India
Email: indranil.saha@honeywell.com
and
Debapriyay Mukhopadhyay
Ixia Technologies
Kolkata, India
Email: dmukhopadhyay@ixiacom.com

A Distributed Algorithm of Fault Recovery For Stateful Failover 1

TAMC 2007 25 th May, 2007

Presentation Outline
I will talk about
• Introduction
• System Models
• Distributed Algorithm for Automated Fault Recovery
• Formal veriﬁcation of the Distributed Algorithm
• Conclusion


TAMC 2007 25 th May, 2007

• Introduction
• System Models
• Conclusion


TAMC 2007 25 th May, 2007

Introduction
• Critical business processes and mission critical systems should
provide a high degree of availability and reliability to the end
users.
• Redundancy techniques are mostly used to achieve
fault-tolerance.
• Redundancy can be achieved by using extra copies of its
components which include hardware, software and network
components.


TAMC 2007 25 th May, 2007

Stateful and Stateless Failover
• Stateless Failover:
- Occasional loss of application state information or data is
tolerable.
- The system can restart without any state or data restoration
after a failure.
- Any live node in the network is a promising candidate to take
over the processes of any failed node
• Stateful Failover
- Restoration of the state or data pertaining to the application
is required for highly accurate recovery.
- How to distribute the state information of a node across the
network is an important issue.


TAMC 2007 25 th May, 2007

Related Works
• Graph theoretic models have been extensively used to represent
processor-to-processor interconnection structure of fault
tolerant designs for speciﬁc multi-processor architectures
(Kuhl80, Yang88, Sridhar91, Mukhopadhyay92, Sung00,
Hung01).
• Minimum k-Hamilton graphs are widely used to meet
reliability considerations for loop type communication networks
(Mukhopadhyay92, Sung00, Hung01).
• Fault tolerant networks based on de Bruijn graph are proposed,
which can tolerate up to k − 2 node faults, where the graph is
regular of degree k and have k n number of vertices for some n
(Sridhar91).
None of these works talk about stateful failover.


TAMC 2007 25 th May, 2007

• Introduction
• System Models
• Conclusion


TAMC 2007 25 th May, 2007

System Model
• The network consists of the set of nodes N with |N | = n
• Each node is labeled with a unique id from 0 to n − 1.
• Each node handles one process initially, and is capable of
executing at most m processes simultaneously.
• Pi is the process node i starts executing initially when the
network becomes functional.
• Failures are of failstop kind, i.e., the nodes in the network can
stop operating at any point of time due to a crash.
• With a processor failed, all the links incident on that node also
becomes non-functional.
• k node faults are allowed in the network.


TAMC 2007 25 th May, 2007

Network Topology
Each node i ∈ N, (0 ≤ i ≤ n − 1), in the network is connected to
the set of nodes Pi ⊆ N, such that |Pi | = l = k + x, where
k + x(≤ n − 1) is even, and

Pi = {j ∈ N : j = (i + p)(mod n), where − l/2 ≤ p ≤ l/2, p = 0}

Underlying undirected graph modeling the network can be written
as (N, E) where
n−1
E = ∪i=0 {(i, j) : j ∈ Pi }.

The state information of processor i, i ∈ N , is periodically
forwarded to all the nodes in the set Fi ⊆ N such that |Fi | = k and

Fi = {j ∈ S : j = (i + p)(mod n), where − k/2 ≤ p ≤ k/2 , p = 0}


TAMC 2007 25 th May, 2007

Connectivity
- The graph (N, E) represents a regular network, for, the degree of
each node is l.

- For any n and k, the graph (N, E) corresponds to the Harary
Graph Hl,n , where

 k + 2, for k even,
l =k+x≥
 k + 1, for k odd,

The network is l-connected with χ(G) ≥ l(> k),

χ(G) denotes the connectivity of G.


TAMC 2007 25 th May, 2007

Theoretical Results
Theorem 1. A. Forwarding state information of each process to k
other nodes in the network ensures k-fault tolerance.
B. A suﬃcient condition to ensure k-fault tolerance is to forward
the state information by each node to at least k other nodes in the
network.
Theorem 2. As long as k ≤ m−1 .n , no live node has to execute
m
more than m processes including one of its own and an algorithm to
attain the same under the proposed framework can also be found.
Theorem 3. Minimum number of nodes with which any node in a
network with n > 2k (or n = 2k) is required to be connected
directly is 2k (or 2k − 1) to ensure that all the eligible nodes
corresponding to a process can be updated about its state
information all the time in one hop.


TAMC 2007 25 th May, 2007

Network Example
Illustration of a network with n = 10, m = 2 and k = 4


TAMC 2007 25 th May, 2007

• Introduction
• System Models
• Conclusion


TAMC 2007 25 th May, 2007

Message Types
1. INFO
• In the ﬁrst round, each node i sends an IN F O message to
all the nodes in the set Fi .
• Message consists of the tuple (j, Fj )
2. STATUS
• Starting from the second round in each successive rounds,
every live node i sends ST AT U S message for every process
pj that is running on it to all the live nodes in the set Fj .
• Message consists of the tuple (pj , Spj )
• Ommision of the Status message for a process for a round
indicates the failure of the process.
3. RESOLVED Message is sent to all the nodes in Fj by the node
who has resolved the failure of process j.


TAMC 2007 25 th May, 2007

Preference for the Neighbours
i
prefj . denotes the preference of node i to take process j in case of
its failure among the nodes in Fj .


TAMC 2007 25 th May, 2007

Distributed Algorithm: Example
Illustration of the distributed algorithm for a network with n = 10,
m = 2 and k = 4

Every node is running its own process.


TAMC 2007 25 th May, 2007


Node 9 is faulty


TAMC 2007 25 th May, 2007


Node 1 takes the process of node 9 after one round as it is the highest
preference node for process 9.


TAMC 2007 25 th May, 2007


Node 2 is faulty


TAMC 2007 25 th May, 2007




TAMC 2007 25 th May, 2007


Node 8 is faulty


TAMC 2007 25 th May, 2007




TAMC 2007 25 th May, 2007


Node 0 is faulty. Real problem begins...


TAMC 2007 25 th May, 2007


Node 7 takes the process of node 8 after 3 rounds as it is the third


TAMC 2007 25 th May, 2007


Node 1 stops running process 9 and starts running process 0 after 6
rounds of node 0’s failure.
According to Theorem 1 there is at least one node available to take
process 9.


TAMC 2007 25 th May, 2007


Node 7 stops running process 8 and starts running process 9 after 8
rounds when node 1 stops running process 9.


TAMC 2007 25 th May, 2007


Node 6 starts running process 8 after 4 rounds when node 7 stops
running process 8.
No more failure is possible.


TAMC 2007 25 th May, 2007

Analysis of the Algorithm
• At most 2k rounds are required to resolve a single fault.
• To resolve a single fault, the maximum number of
RESOLV ED messages that is required to be sent across the
network is (k − 2)m + 1, where m is the maximum number of
processes that a node is capable of executing.


TAMC 2007 25 th May, 2007

• Introduction
• System Models
• Conclusion


TAMC 2007 25 th May, 2007

Correctness of the Algorithm
• We show the correctness of the distributed algorithm through
formal veriﬁcation.
• We use Spin Model checker for modeling and veriﬁcation of the
algorithm.
• We have been able to verify our model for N=8, K=3 and M=2
and all lower instances.
• Due to the state-space explosion problem inherent in model
checker SPIN, we could not verity our algorithm for more than
8 processors.


TAMC 2007 25 th May, 2007

Spin Model Checker
• Tool for automatically model checking distributed algorithms
• Promela is a language for modeling systems of concurrent
processes that can interact via shared variables and message
channels
• Given a concurrent system modeled by a Promela program,
SPIN can check for deadlock, dead code, violations of user
speciﬁed assertions, and temporal properties expressed by LTL
formulas
• When a violation of a property is detected, SPIN reports a
scenario, i.e., a sequence of transitions, violating the property.


TAMC 2007 25 th May, 2007

Properties
Safety 1 Whenever a node becomes faulty, at least one of its
neighboring nodes is non-faulty.

Safety 2 No node has to take more than M processes at any point
of time.

Liveness Whenever a node becomes faulty, its process is
eventually taken up by some other live nodes.

Timeliness Every fault is recovered in no more than 2K rounds.


TAMC 2007 25 th May, 2007

• Introduction
• System Models
• Conclusion


TAMC 2007 25 th May, 2007

Conclusion
• We have presented a distributed algorithm of automated fault
recovery for stateful failover in a network.
• In whatever way the fault may arise the algorithm can handle
that fault
• In at most 2k rounds the processes of the faulty processor are
taken up by a(some) eligible live node(nodes) in the network.
• The message complexity of our algorithm is linear with the
number of nodes.
• The correctness of the algorithm has been proved by modeling
the algorithm in SPIN and verifying its desired properties.


TAMC 2007 25 th May, 2007

Thank You!!


Slides Tamc07

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to Slides Tamc07

Similar to Slides Tamc07 (20)

Recently uploaded

Recently uploaded (20)

Slides Tamc07