Time-Invariant Markov Chains with Finite State Space

3 TIME-INVARIANT MARKOV CHAINS WITH FINITE STATE SPACE 

First we will look at Markov chains that have a finite state space. Let us suppose there are states numbered from 1. K. In this case the transition probabilities can be put in a matrix. 

We will restrict ourselves to time invariant Markov chains where the transition probabilities only depend on the states, not the time n. These are also called homogeneous Markov chains. In this case, we can leave out the time index and the transition probability matrix of the Markov chain is given by 

Since the Markov chain satisfies the Markov property 

We recognize this is the formula for matrix multiplication. Hence the matrix of two-step transition probabilities is given by 

---

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

The one-step transition probabilities for a time-invariant Markov chain with a finite state space can be put in a matrix P, where the pij is the probability of a transition from state i to stated in one-step. This is the conditional probability... 

---