Markov Chains with Continuous State Space

7 MARKOV CHAINS WITH CONTINUOUS STATE SPACE 

Sometimes the state space of a Markov chain consists of all possible values in an interval. In that case, we would say the Markov chain has a one-dimensional continuous state space. In other Markov chains, the state space consists of all possible values in a rectangular region of dimension p and we would say the Markov chain has a p-dimensional continuous state space. In both of these cases, there are an uncountably infinite number of possible values in the state space. This is far too many to have a transition probability function associated with each pair of values. The probability of a transition between all but a countable number of pairs of possible values must be zero. Otherwise the sum of the transition probabilities would be infinite. A state to state transition probability function won t work for all pairs of states. Instead we define the transition probabilities from each possible state X to each possible measurable set of states A We call 

The main results that hold for discrete Markov chains continue to hold for Markov chains with a continuous state space with some modifications. In the single dimensional case, the probability of a measurable set A can be found from the probability of the transition CDF which is 

This is analogous to Equation 5.5, If the Markov chain possesses a limiting transition density independent of the initial state, 

This is analogous to Equation 5.11. When the continuous state space has dimension p, the integrals are multiple integrals over p dimensions, and the joint density function is found by taking p partial derivatives. 

---

In general, the probability of passing to state j from state i must be zero at all but at most a countably infinite number of states, or else they would not have a finite sum. This means, that we can t classify states by their first return probabilities and return probabilities as given in Tables 5.1 and 5.2, because these would equal 0 for almost all states. Redefining the definitions for recurrence is beyond the scope of this book, and Gamerman (1997) outlines the changes required. What is important to us is that under the required modifications, the main results we found for discrete Markov chains continue to hold for Markov chains with continuous state space. These are ... 

For a Markov chain with continuous state space, if the transition kernel is absolutely continuous, the Chapman-Kolmogorov and the steady state equations are written as integral equations involving the transition density function. 

In the case of a discrete time Markov chain with a continuous state space, we may simply suppress the variable t in the above formulas, and write... 

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

For Markov chains with a continuous state space, there are too many states for us to use a transition probability function. Instead we define a transition kernel which measures the probability of going from each individual state to every measurable set of states. 

This says the long-run probability of a state equals the weighted sum of one-step probabilities of entering that state from all states each weighted by its long-run probability. The comparable steady state equation that ir 0), the long-run distribution of a Markov chain with a continuous state space, satisfies is given by... 

Markov chains or processes are named after the Russian mathematician A.A.Markov (1852-1922) who introduced the concept of chain dependence and did basic pioneering work on this class of processes [1]. A Markov process is a mathematical probabilistic model that is very useful in the study of complex systems. The essence of the model is that if the initial state of a system is known, i.e. its present state, and the probabilities to move forward to other states are also given, then it is possible to predict the future state of the system ignoring its past history. In other words, past history is immaterial for predicting the future this is the key-element in Markov chains. Distinction is made between Markov processes discrete in time and space, processes discrete in space and continuous in time and processes continuous in space and time. This book is mainly concerned with processes discrete in time and space. 

The models discrete in space and continuous in time as well as those continuous in space and time, led many times to non-linear differential equations for which an analytical solution is extremely difficult or impossible. In order to solve the equations, simplifications, e.g. linearization of expressions and assumptions must be carried out. However, if this is not sufficient, one must apply numerical solutions. This led the author to a major conclusion that there are many advantages of using Markov chains which are discrete in time and space. The major reason is that physical models can be presented in a unified description via state vector and a one-step transition probability matrix. Additional reasons are detailed in Chapter 1. It will be shown later that this presentation coincides also with the fact that it yields the finite difference equations of the process under consideration on the basis of which the differential equations have been derived. 

Deflnitions. The basic elements of Markov chains associated with Eq.(2-24) are the system, the state space, the initial state vector and the one-step transition probability matrix. Considering refs.[26-30], each of the elements will be defined in the following with special emphasize to chemical reactions occurring in a batch perfectly-mixed reactor or in a single continuous plug-flow reactor. In the latter case, which may simulated by perfectly-mixed reactors in series, all species reside in the reactor the same time. 

The Poisson counting process of Section 2 is a continuous-time Markov chain N on the infinite state space 0, 1, 2,. . . , with generator... 

---