Continuous Markov processes, probability times

A continuous Markov process (also known as a diffusive process) is characterized by the fact that during any small period of time At some small (of the order of %/At) variation of state takes place. The process x(t) is called a Markov process if for any ordered n moments of time t < < t < conditional probability density depends only on the last fixed value ... 

Propagation of the fast subsystem - chemical Langevin equations The fast subset dynamics are assumed to follow a continuous Markov process description and therefore a multidimensional Fokker-Planck equation describes their time evolution. The multidimensional Fokker-Plank equation more accurately describes the evolution of the probability distribution of only the fast reactions. The solution is a distribution depicting the state occupancies. If the interest is in obtaining one of the possible trajectories of the solution, the proper course of action is to solve a system of chemical Langevin equations (CLEs). 

Let continuous one-dimensional Markov process x(f) at initial instant of time t = 0 have a fixed value x(0) = xo within the interval (c, d) that is, the initial probability density is the delta function ... 

The present stochastic model is the so-called particle model, where the substance of interest is viewed as a set of particles.1 We begin consideration of stochastic modeling by describing Markov-process models, which loosely means that the probability of reaching a future state depends only on the present and not the past states. We assume that the material is composed of particles distributed in an m-compartment system and that the stochastic nature of material transfer relies on the independent random movement of particles according to a continuous-time Markov process. 

If we consider the evolution of the liquid element together with the state of probabilities of elementary evolutions, we can observe that we have a continuous Markov stochastic process. If we apply the model given in Eq. (4.68), Pj(z, t) is the probability of having the liquid element at position x and time t evolving by means of a type 1 elementary process (displacement with a d-v flow rate along a positive direction of x). This probability can be described through three independent events ... 

If 21(f) is a Markov process with continuous transition probabilities and T(t) a process with non-negative independent increments, then X(T(t)) is also a Markov process. Thus, this process is subordinated to 21(f) with operational time T(t). The process 7 (f) is called a directing (controlling) process. 

Markov chains or processes are named after the Russian mathematician A.A.Markov (1852-1922) who introduced the concept of chain dependence and did basic pioneering work on this class of processes [1]. A Markov process is a mathematical probabilistic model that is very useful in the study of complex systems. The essence of the model is that if the initial state of a system is known, i.e. its present state, and the probabilities to move forward to other states are also given, then it is possible to predict the future state of the system ignoring its past history. In other words, past history is immaterial for predicting the future this is the key-element in Markov chains. Distinction is made between Markov processes discrete in time and space, processes discrete in space and continuous in time and processes continuous in space and time. This book is mainly concerned with processes discrete in time and space. 

The models discrete in space and continuous in time as well as those continuous in space and time, led many times to non-linear differential equations for which an analytical solution is extremely difficult or impossible. In order to solve the equations, simplifications, e.g. linearization of expressions and assumptions must be carried out. However, if this is not sufficient, one must apply numerical solutions. This led the author to a major conclusion that there are many advantages of using Markov chains which are discrete in time and space. The major reason is that physical models can be presented in a unified description via state vector and a one-step transition probability matrix. Additional reasons are detailed in Chapter 1. It will be shown later that this presentation coincides also with the fact that it yields the finite difference equations of the process under consideration on the basis of which the differential equations have been derived. 

So far we have considered a single mesoscopic equation for the particle density and a corresponding random walk model, a Markov process with continuous states in discrete time. It is natural to extend this analysis to a system of mesoscopic equations for the densities of particles Pi(x,n), i = 1,2,..., m. To describe the microscopic movement of particles we need a vector process (X , S ), where X is the position of the particle at time n and S its state at time n. S is a sequence of random variables taking one of m possible values at time n. One can introduce the probability density Pj(jc, n) = 9P(X < x,S = i)/dx and an imbedded Markov chain with the m x m transition matrix H = (/i ), so that the matrix entry corresponds to the conditional probability of a transition from state i to state j. 

The change in concentration of clusters of n molecules may be written as dCn(t)/dt = an-iCn-i(t) — (ccn + Pn)Cn(t) + Pn+iCn+i(t), which has the form of Kolmogorov differential equation for Markov processes in discrete number space and continuous time [21]. and fin are respectively the net probabilities of incorporation or loss of molecules by a cluster per unit time, and these may be defined formally as the aggregation or detachment frequencies times the surface area of the cluster of n molecules. Given the small size of the clusters, and fin are not simple functions of n and in general they are unknown. However, if and fin are not functions of time, then an equilibrium distribution C° of cluster sizes exists, such that dC°/dt = 0 for Cn t) = C°, and the following differential... 

The use of the embedded Discrete Time Markov Chain in a continuous stochastic process for deter-mining the events probability makes assumption that the system is in a stationary state characterizing by stationary distribution probabihties over its states. But the embedded DTMC is not limited to Continuous Time Markov Chain a DTMC can also be defined from semi-Markov or under some hypothesis from more generally stochastic processes. Another advantage to use the DTMC to obtain the events probability is that the probability of an event is not the same during the system evolution, but can depends on the state where it occurs (in other words the same event can be characterized by different occurrence probabilities). The use of the Arden lemma permits to formally determine the whole set of events sequences, without model exploring. Finally, the probability occurrence for relevant or critical events sequences and for a sublanguage is determined. 

Thus, the transitions are always from a state n to the state n + 1. The transitions are, of course, arrivals because they cause the count N to increase by 1. The probability of a transition in a short interval of time h is approximately Xh for any n by (26). This observation corresponds precisely with the description of the Poisson process in terms of coin tossing in Section 2. Moreover, the fact that the time tetween arrivals in a Poisson process is exponential may be seen now as a consequence of the fact, expressed in (33), that the holding times in any continuous-time Markov chain are exponentially distributed. 

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

---