Markov chains time invariant

Thus, we have seen that every binary DMS possesses the AEP in fact, so does every DMS. An information source need not be memoryless, though, to possess the AEP every stationary ergodic source—that is, a discrete source for which the output is a stationary ergodic random process—possesses the AEP (and, therefore, an entropy too). Even some nonstationary sources possess the AEP. For example, if the source sequence (wi, M2,. ..) is a time-invariant Markov chain, then its entropy is given by the formula... 

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

TIME-INVARIANT MARKOV CHAINS WITH FINITE STATE SPACE... 

We will restrict ourselves to time invariant Markov chains where the transition probabilities only depend on the states, not the time n. These are also called homogeneous Markov chains. In this case, we can leave out the time index and the transition probability matrix of the Markov chain is given by... 

We will consider time-invariant Markov chains that are irreducible and aperiodic and where all states are positive recurrent. Chains having these properties are called ergodic. This type of chain is important as there are theorems which show that for this type of chain, the time average of a single realization approach the average of all possible realizations of the same Markov chain (called the ensemble) at some... 

If the probability of transition from state i to state j is the same for all times, the Markov chain is called time-invariant or homogeneous. 

The one-step transition probabilities for a time-invariant Markov chain with a finite state space can be put in a matrix P, where the pij is the probability of a transition from state i to stated in one-step. This is the conditional probability... 

---