Show that if a state in a communication class is either null-recurrent or positive recurrent or transient, every state shares the same property in the class.

A state “i” is recurrent if and only if

And it is transient otherwise. The states are recurrent or transient and lie under the same communication class. Consider the communication class is C and all the states are are transient and recurrent. The process can be easily seen from the following conditions.

1.      Suppose that two states are in the communication class that are “i” and “j”. now select an appropriate “n” so the statement gives . Now the condition “i” is recurrent and j for every time “i” is visited the positive probability p gives that “j” can be visited by n steps. Now the recurrent state of “i” can be visited infinite times under the Bernoulli trails. We can conclude that eventually there will be two states “i” and “j” that are recurrent. Eventually if “i” and “j” communicate and “i” is transient then “j” will also be transient.

2.      Considering the irreducible Markov chain, the states are recurrent together and all the states are transient. If any state is recurrent and Markov chain the recurrent conditions are transient. If two communication classes are

 The state spaces are finite for Markov Chain and states can be transient. Considering all irreducible Markov chain in the finite state spaces is recurrent for all the states. If two states communicate and one of them is recurrent then the state  with visiting states “i” for infinite number of times. In case of irreducible recurrent Markov Chains each state “j” will be visited repeatedly regardless of the initial state .

Question 2

An exercise on Sheet 9 proved that for a finite state space, a Markov chain is closed if and only if it is recurrent. Can a Markov chain on an infinite state space have a closed and transient communication class? Prove or disprove

In a Markov chain with the finite state spaces there is at least one closed communication class and it can be proved by saying  class that is not closed and now we take  class. The process ends up by reaching a closed class. Determining the recurrence and transience when number of the state as infinite.

Question 3

Show that every time-reversible probability distribution on the states of a Markov chain is stationary. Hint: Consider the detailed balance equations.

The Markov chain  is the time reversible if and only if the distribution  of the set  satisfies as  the equation becomes valid in the time reversible probability. The conditions for the time reversible probability and distribution on the states can be determined by using balance equation.

According to the equation  here the  factor is allowed to propagate through both sides of the product moving from left to the right side and reversing the direction of the transition induce minimum impact on the equation. The trick is applicable to deduce the general equality. The nice interpretation in terms of the probability flux is from “i” to “j” with . The flux remain same for both  “i” and “j” that is from i-flux to j-flux. The balance equation

Question 4

Show that a time-reversible probability distribution on the states of a Markov chain is uniform if and only if the transition matrix P is symmetric. 

The equation above with the detailed balance equation shows that the time reversible Markov Chains hold. As the Markov chain P is reversible with respect to  then the stationary distribution is for P. the equation gives

 

Question 5

Suppose N balls are placed into two buckets (in some way). Every step, one ball is chosen randomly, and it swaps buckets. Show that the associated Markov chain is has a time-reversible distribution. How could this be used to find the unique stationary distribution fast?

Consider the number of balls in the buckets are N and they are distributed in two buckets. Each time we have to select only one ball at random and then move the bucket with another one.

Question 6

Consider a two-state Markov chain of states a and b. Suppose that the transition probabilities contain pab = p and pba = q.

a.      For each value of p and q, determine the number of communication classes? When is it strongly connected? For each communication class, determine whether it is null-recurrent, positively recurrent or transient.

Consider state “b” that is reachable from the state “a” if the condition becomes  for some . In the state . The two states “a” and “b” are communicating if  and  the case becomes . The fact for the equivalence relation means that the relation partition is different at the state space “a” and the equivalent classes are called communication classes. Chain is irreducible for only the class. Graph is directed edge from “a” and “b” if  the communication classes are strongly connected components. The closed class C is the case where  and  imply . The state “a” is absorbing if {a} is the closed class.

b.      For what values of p and q does a stationary distribution exist? Find one whenever possible.

c.       For what values of p and q does a time-reversible distribution exist? Find one whenever possible.

The unique invariant P has irreducible and recurrent for positive recurrence. The equations for expected time spent between two events I and k as below

d.      For each value of p and q, find the expected time to reach state 1 from state 0

Consider the state of Markov chain at a time t is the value of . As the state changes from 0 to 1 that the process will be in state 1 at the time t. In the state space of the Markov chain the set of values for each   can be  and trajectory time will be 1 for one state transition.