Continuous Markov processes, probability

A continuous Markov process (also known as a diffusive process) is characterized by the fact that during any small period of time At some small (of the order of %/At) variation of state takes place. The process x(t) is called a Markov process if for any ordered n moments of time t < < t < conditional probability density depends only on the last fixed value ... 

The transition probability density of continuous Markov process satisfies to the following partial differential equations (WXo(x, t) = W(x, t xo, to)) ... 

Propagation of the fast subsystem - chemical Langevin equations The fast subset dynamics are assumed to follow a continuous Markov process description and therefore a multidimensional Fokker-Planck equation describes their time evolution. The multidimensional Fokker-Plank equation more accurately describes the evolution of the probability distribution of only the fast reactions. The solution is a distribution depicting the state occupancies. If the interest is in obtaining one of the possible trajectories of the solution, the proper course of action is to solve a system of chemical Langevin equations (CLEs). 

Let continuous one-dimensional Markov process x(f) at initial instant of time t = 0 have a fixed value x(0) = xo within the interval (c, d) that is, the initial probability density is the delta function ... 

Remark. The essential feature of our composite process is that i is an independent process by itself, while the transition probabilities of r are governed by i. This situation can be formulated more generally. Take a Markov process Y(t), discrete or continuous, having an M-equation with kernel... 

Markov processes whose master equation has the form (1.1) have been called continuous , because it can be proved that their sample functions are continuous (with probability 1). This name has sometimes led to the erroneous idea that all processes with a continuous range are of this type and must therefore obey (1.1). 

The present stochastic model is the so-called particle model, where the substance of interest is viewed as a set of particles.1 We begin consideration of stochastic modeling by describing Markov-process models, which loosely means that the probability of reaching a future state depends only on the present and not the past states. We assume that the material is composed of particles distributed in an m-compartment system and that the stochastic nature of material transfer relies on the independent random movement of particles according to a continuous-time Markov process. 

Two-phase polymerization is modeled here as a Markov process with random arrival of radicals, continuous polymer (radical) growth, and random termination of radicals by pair-wise combination. The basic equations give the joint probability density of the number and size of the growing polymers in a particle (or droplet). From these equations, suitably averaged, one can obtain the mean polymer size distribution. 

If we consider the evolution of the liquid element together with the state of probabilities of elementary evolutions, we can observe that we have a continuous Markov stochastic process. If we apply the model given in Eq. (4.68), Pj(z, t) is the probability of having the liquid element at position x and time t evolving by means of a type 1 elementary process (displacement with a d-v flow rate along a positive direction of x). This probability can be described through three independent events ... 

If 21(f) is a Markov process with continuous transition probabilities and T(t) a process with non-negative independent increments, then X(T(t)) is also a Markov process. Thus, this process is subordinated to 21(f) with operational time T(t). The process 7 (f) is called a directing (controlling) process. 

Markov chains or processes are named after the Russian mathematician A.A.Markov (1852-1922) who introduced the concept of chain dependence and did basic pioneering work on this class of processes [1]. A Markov process is a mathematical probabilistic model that is very useful in the study of complex systems. The essence of the model is that if the initial state of a system is known, i.e. its present state, and the probabilities to move forward to other states are also given, then it is possible to predict the future state of the system ignoring its past history. In other words, past history is immaterial for predicting the future this is the key-element in Markov chains. Distinction is made between Markov processes discrete in time and space, processes discrete in space and continuous in time and processes continuous in space and time. This book is mainly concerned with processes discrete in time and space. 

The reader can observe on the right- and left-hand sides of the desk a total of twelve books on Markov processes among which are, probably, refs. [2-8, 15-18, 84]. Some support to the fact that the books are abandoned is the prominent fact that the immediate continuation of the desk is the street... and that the books are leaning on the buildings. 

So far we have considered a single mesoscopic equation for the particle density and a corresponding random walk model, a Markov process with continuous states in discrete time. It is natural to extend this analysis to a system of mesoscopic equations for the densities of particles Pi(x,n), i = 1,2,..., m. To describe the microscopic movement of particles we need a vector process (X , S ), where X is the position of the particle at time n and S its state at time n. S is a sequence of random variables taking one of m possible values at time n. One can introduce the probability density Pj(jc, n) = 9P(X < x,S = i)/dx and an imbedded Markov chain with the m x m transition matrix H = (/i ), so that the matrix entry corresponds to the conditional probability of a transition from state i to state j. 

In the case of a Markov process with state space a subset of M, and with F the associated Borel a-algebra, and with a continuous index variable t e R+, the transition probabilities are defined instead for sets Ae... 

In contrast to the continuous models, the discrete models consider the processes at the level of individual structural elements, e.g. individual fibres, threads or loops, or individual stages of the process. In these models the processes are modelled as a series of states where the transition from one state to another happens with a probability. The underpinning theories for these models are theory of Markov processes (Kemeny and Snell, 1960), queuing theory (Gross et nf, 2008), and finite automata theory (Anderson, 2006 Hopcroft et al., 2007). 

The change in concentration of clusters of n molecules may be written as dCn(t)/dt = an-iCn-i(t) — (ccn + Pn)Cn(t) + Pn+iCn+i(t), which has the form of Kolmogorov differential equation for Markov processes in discrete number space and continuous time [21]. and fin are respectively the net probabilities of incorporation or loss of molecules by a cluster per unit time, and these may be defined formally as the aggregation or detachment frequencies times the surface area of the cluster of n molecules. Given the small size of the clusters, and fin are not simple functions of n and in general they are unknown. However, if and fin are not functions of time, then an equilibrium distribution C° of cluster sizes exists, such that dC°/dt = 0 for Cn t) = C°, and the following differential... 

The use of the embedded Discrete Time Markov Chain in a continuous stochastic process for deter-mining the events probability makes assumption that the system is in a stationary state characterizing by stationary distribution probabihties over its states. But the embedded DTMC is not limited to Continuous Time Markov Chain a DTMC can also be defined from semi-Markov or under some hypothesis from more generally stochastic processes. Another advantage to use the DTMC to obtain the events probability is that the probability of an event is not the same during the system evolution, but can depends on the state where it occurs (in other words the same event can be characterized by different occurrence probabilities). The use of the Arden lemma permits to formally determine the whole set of events sequences, without model exploring. Finally, the probability occurrence for relevant or critical events sequences and for a sublanguage is determined. 

Markov process A stochastic process is a random process in which the evolution from a state X(t ) to X(t +i) is indeterminate (i.e. governed by the laws of probability) and can be expressed by a probability distribution function. Diffusion can be classified as a stochastic process in a continuous state space (r) possessing the Markov property as... 

For many synthetic copolymers, it becomes possible to calculate all desired statistical characteristics of their primary structure, provided the sequence is described by a Markov chain. Although stochastic process 31 in the case of proteinlike copolymers is not a Markov chain, an exhaustive statistic description of their chemical structure can be performed by means of an auxiliary stochastic process 3iib whose states correspond to labeled monomeric units. As a label for unit M , it was suggested [23] to use its distance r from the center of the globule. The state of this stationary stochastic process 31 is a pair of numbers, (a, r), the first of which belongs to a discrete set while the second one corresponds to a continuous set. Stochastic process ib is remarkable for being stationary and Markovian. The probability of the transition from state a, r ) to state (/i, r") for the process of conventional movement along a heteropolymer macromolecule is described by the matrix-function of transition intensities... 

Chapter 4 is devoted to the description of stochastic mathematical modelling and the methods used to solve these models such as analytical, asymptotic or numerical methods. The evolution of processes is then analyzed by using different concepts, theories and methods. The concept of Markov chains or of complete connected chains, probability balance, the similarity between the Fokker-Plank-Kolmogorov equation and the property transport equation, and the stochastic differential equation systems are presented as the basic elements of stochastic process modelling. Mathematical models of the application of continuous and discrete polystochastic processes to chemical engineering processes are discussed. They include liquid and gas flow in a column with a mobile packed bed, mechanical stirring of a liquid in a tank, solid motion in a liquid fluidized bed, species movement and transfer in a porous media. Deep bed filtration and heat exchanger dynamics are also analyzed. 

The models discrete in space and continuous in time as well as those continuous in space and time, led many times to non-linear differential equations for which an analytical solution is extremely difficult or impossible. In order to solve the equations, simplifications, e.g. linearization of expressions and assumptions must be carried out. However, if this is not sufficient, one must apply numerical solutions. This led the author to a major conclusion that there are many advantages of using Markov chains which are discrete in time and space. The major reason is that physical models can be presented in a unified description via state vector and a one-step transition probability matrix. Additional reasons are detailed in Chapter 1. It will be shown later that this presentation coincides also with the fact that it yields the finite difference equations of the process under consideration on the basis of which the differential equations have been derived. 

Thus, the transitions are always from a state n to the state n + 1. The transitions are, of course, arrivals because they cause the count N to increase by 1. The probability of a transition in a short interval of time h is approximately Xh for any n by (26). This observation corresponds precisely with the description of the Poisson process in terms of coin tossing in Section 2. Moreover, the fact that the time tetween arrivals in a Poisson process is exponential may be seen now as a consequence of the fact, expressed in (33), that the holding times in any continuous-time Markov chain are exponentially distributed. 

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

---