1. UNIT II
Title
Marcov Chains And Simple Queue
Part I
Discrete-Time Marcov Chains
Presented By
K.GURUNATHAN ME.,

2. 1.1 DTMC Vs CTMC
DTMC
 The arrival or departure of
an event can only occur at
the end of a time step.
 This property makes
DTMCs a little odd for
modeling computer system.
CTMC
 The arrival of an event can
happen at any moment in
time.
 This makes CTMCs
convenient for modeling
system.

3. 1.2 Definition of a DTMC
 A DTMC is a stochastic process with discrete
state space and discrete time {Xn, n>0} is a
discrete time marcov chain if
{Xn+1 | Xn = in, Xn-1 = in+1,… X0 = i0}
= P {Xn+1 = j | Xn = i}
= Pij(n)

4. 1.2 Definition of a DTMC (cont…)
 The past history impacts on the future
evolution of the system via the current state of
the system.
 Where, Pij(n) is called transaction probability
from state “i” to state “j” at time “n”.

5. 1.2 Definition of a DTMC (cont…)
Stochastic Process: SP
 It is simply a sequence of random variables.
 Suppose we observe some characteristics of a
system at discrete points in time.

6. 1.2 Definition of a DTMC (cont…)
Stochastic Process: SP
 Discrete time SP:
 A description of the relation between the random
variables (X0, X1, X2…)
Example, Observing the price of a share of intel, at
the beginning of each day.

7. 1.2 Definition of a DTMC (cont…)
Stochastic Process: SP
 Continuous time SP:
 A state of the system can be viewed at any time,
not just at discrete instants in time.
Example, the number of people in a supermarket t
minutes after the store opens for business.

8. 1.2 Definition of a DTMC (cont…)
Homogeneous DTMC:
 A DTMC is said to be homogeneous iff its transitions
probabilities do not depend on the time n (i.e..,)
P [Xn+1 | Xn = i] = P[ X1 = j | X0 = i] Put n = 0
Pij

9. 1.2 Definition of a DTMC (cont…)
Markovian Property:
 It states the conditional distribution of any
future state Xn+1, given past states X0, X1,.. Xn-1 and
given present state Xn.
It is independent of past states and depends only on
the present state Xn.

10. 1.2 Definition of a DTMC (cont…)
Transition Probability Matrix:
 Pij is the transition probability from state i to j.
 It is displayed as SxS transition probability matrix P.
 Each row in the P matrix most b e non-negative.
 The en tries in each row must sum to 1.

11. 1.3 Examples of finite state DTMCs
Repair facility problem:
 A machine is either working or in repair center.
 If it is working today then there is a 95% chance that
it will be working tomorrow.
 If it is in the repair centre today, then there is a 40%
chance that will be working tomorrow.

12. 1.3 Examples of finite state DTMCs
Umbrella problem:
 An absent minded professor has two umbrellas that
he uses when commuting from home to office and
back.
 If it rains, and an umbrella is available in his location,
he take it. If it is not raining, he always forgets to take
an umbrella.

13. 1.3 Examples of finite state DTMCs
Program analysis problem:
 A program has 3 types of instructions
 CPU instructions-C
 Memory instructions-M
 User instructions-U

14. 1.4 Powers of P: n-step Transition probability
Let Pn = P.P…P (i.e.) multiplied by n times.
We will use the notation Pn
ij to denote (Pn)ij
Examples:
Umbrella Problem
Repair facility problem

15. 1.5 Stationary Equations
 A Probability distribution is said to be
stationary for the marcov chain if

16. 1.6 Limiting Probability
 Consider the (i,j)th entry of the power matrix Pn for
large n.
Where

17. 1.6 Steady State
 If the initial state is chosen according to the
stationary probabilities then the state is said to be
steady state.
Examples,
 Repair facility problem with cost
Umbrella Problem

18. 1.6 Infinite state DTMCs
 For a marcov chain with an infinite number of states
one can still imagine a transition probability matrix P.
Infinite state DTMCs are common in modeling
system where the number of customers or jobs is
unbounded and thus the state space is unbounded.

19. THE END