Aperiodic Markov chains

Some sufficient conditions for a finite Markov chain to be ergodic are based on the following theorems, given without proof [17, pp.247-255]. The first one states that A finite irreducible aperiodic Markov chain is ergodic. Let ... 

On the basis of the following theorem, i.e., if the transition matrix P for a finite irreducible aperiodic Markov chain with Z states is doubly stochastic, then the stationary probabilities are given by... 

Recall the well-known theorem of the standard theory of the ergodic Markov chains one can state the following (e.g. Feller [4]) In any finite irreducible, aperiodic Markov chain with the transition matrix P, the limit of the power matrices/ exists if r tends to infinity. This limit matrix has identical rows, its rows are the stationary probability vector of the Markov chain, y = [v,Vj,...,v,...,v ], that is v = v P, fiuthermore v, >0 ( = 1,...,R) and... 

AH states in an eigodic (irreducible and aperiodic) Markov chain are positive recurrent if and only if there is a unique non-zero solution to the steady state Equation 5.18. 

By the choice of the action space, all stationary policies have transition probability matrices representing recurrent aperiodic Markov chains. If the number of possible states is limited, i.e. if the one-period demand is bounded, we can determine the optimal production policy. The optimal policy can be determined by a policy iteration method, but we will use the method of successive iteration, as described by Odoni(1969), since this method is faster in our situation. The optimal policy is the policy which achieves the minimum expected costs per transition, which will be denoted by g. Defining the quantity v (r) as the total expected costs from the next n transitions if the current state is r and if an optimal policy is followed, the iteration scheme takes the form described in the optimality principle by Bellman (1957) ... 

By the Ergodic Theorem for irreducible aperiodic Markov chains one also has that... 

If the n-steps transition probability elements are defined as the probability to reach the configuration j in n steps beginning from the configuration i and Ilj, = n (qjMarkov chain is ergodic (the ergodicity condition states that if i and j are two possible configurations with 0 and Ilj 0, for some finite n, pij(nl 0 ) and aperiodic (the chain of configurations do not form a sequence of events that repeats itself), the limits... 

We summarize the discussion of steady-state behavior of Markov chains as follows. For well-behaved chains, such as irreducible, aperiodic, finite-state chains, there is a unique steady-state distribution that is also the limiting distribution. Furthermore, this distribution gives the long-run fraction of time spent in each state. It may be computed easily, by solving the steady-state equations (22). The steady-state behavior of the chain does not depend on the initial state. 

Two important properties of Markov chains are aperiodicity and communication. For each state of the chain, we define the number d(i) to be the greatest common divisor of all integers I such that r > 0 (if the probability of return to state i is zero, then we set d(i) = 0). If d(i) = 1, for all i then we say that the chain is... 

The fundamental result for Markov chains is the following If a given Markov chain is both irreducible and aperiodic, then it can be shown that there is a unique invariant distribution such that... 

Two states accessible from each other in a Markov chain are said to communicate. A class of stafes is a group of sfafes fhaf communicafe wifh each other. If a Markov chain has only one class, fhen if is said to be irreducible. The period in a Markov chain is fhe minimum number of fransifions required to return to a state upon leaving it. A Markov chain of period one is called aperiodic. 

Let P = (pil) be the transition probability matrix of a Markov chain, where Pij is the probability of moving to a state j in the next stage while the process is in state i at the current stage that is, p = Pr[X,. i = X, = /]. A Markov chain is said to have steady state probabilities if the transition probability matrix converges to a constant matrix. Note that the term steady state probability is used here in a rather loose sense since only aperiodic recurrent Markov chains admit this property. 

Every Markov chain with a finite state set has a unique stationary distribution. In addition, if the Markov chain is aperiodic, then it admits steady state probabilities. Given the transition probability matrix, steady state probabilities of a Markov chain can be computed using the methods detailed in Kulkami (1995). 

Note that aperiodicity is often a strong assumption in Markov chains. One can still carry out the steady state probability computations even if a Markov chain has a period greater than one. Components of the steady state probability vector n can then be interpreted as long run proportions of time that the underlying stochastic process would be in a given state. 

A chain that only visits states on some multiple are called periodic Markov chains. We will restrict ourselves to aperiodic Maikov chains. In such a chain, after the chain has run long enough, it could be in any of the states at any time after that. We should note that many chains have the same long-run distribution. 

We will consider time-invariant Markov chains that are irreducible and aperiodic and where all states are positive recurrent. Chains having these properties are called ergodic. This type of chain is important as there are theorems which show that for this type of chain, the time average of a single realization approach the average of all possible realizations of the same Markov chain (called the ensemble) at some... 

Equation 5.16 says that the Uj area solution of the steady state equation. Thus the theorem says that if a unique non-zero solution of the steady state equation exists the chain is ergodic, and vice-versa. A consequence of this theorem is that time averages from an ergodic Markov chain will approach the steady state probabilities of the chain. Note, however, for an aperiodic irreducible Markov chain that has all null recurrent states, the mean recurrence times are infinite, and hence Uj = 0 for such a chain. The only solution to the steady state equation for such a chain is Uj = 0 for all j. It is very important that we make sure all chains that we use are ergodic and contain only positive recurrent states Note also that the theorem does not say anything about the rate the time averages converges to the steady state probabilities. 

An aperiodic irreducible Markov chain with positive recurrent states has a unique non-zero solution to the steady state equation, and vice-versa. These are known as ergodic Markov chains. 

In this case it can be shown that tt is the unique stationary distribution for the Markov chain P = pxy), and that the occupation-time distribution over long time intervals converges (with probability 1) to it, irrespective of the initial state of the system. If, in addition, P is aperiodic [this means that for each pair x,y S, p xy > 0 for all sufficiently large n], then the probability distribution at any single time in the far future also converges to tt, irrespective of the initial state— that is, = Tty for all x. 

---