Markov process Master equation

Contents A Historical Introduction. - Probability Concepts. -Markov Processes. - The Ito Calculus and Stochastic Differential Equations. - The Fokker-Planck Equatioa - Approximation Methods for Diffusion Processes. - Master Equations and Jump Processes. - Spatially Distributed Systems. - Bistability, Metastability, and Escape Problems. - Quantum Mechanical tokov Processes. - References. - Bibliogr hy. - Symbol Index. - Author Index. - Subject Index. 

Although Eq. (16) has a very simple form, the quantities a(x, y, t) may be very complicated. This is why the probabilistic master equation, Eq. (16), appears much simpler than the physical master equation which will be discussed presently. The equations which describe the time development of the conditional probabilities w2 are also obtained straightforwardly from Eqs. (lb) and (2). The results for a Markov process are... 

All that remains to be done for determining the fluctuation spectrum is to compute the conditional average, Eq. (31). However, this involves the full equations of motion of the many-body system and one can at best hope for a suitable approximate method. There are two such methods available. The first method is the Master Equation approach described above. Relying on the fact that the operator Q represents a macroscopic observable quantity, one assumes that on a coarse-grained level it constitutes a Markov process. The microscopic equations are then only required for computing the transition probabilities per unit time, W(q q ), for example by means of Dirac s time-dependent perturbation theory. Subsequently, one has to solve the Master Equation, as described in Section TV, to find both the spectral density of equilibrium fluctuations and the macroscopic phenomenological equation. 

On the other hand, (1.4) may be interpreted as to mean that any solution with initial condition P(y, t0) = 5(y — y0) is identical with the transition probability Pi i y> 11y0, to)- In fact, the master equation derived in the next chapter for Markov processes is of this type. Yet it cannot guarantee Markovian character, inasmuch as it still does not say anything about the higher distribution functions. ... 

The operator Q in the master equation is linear. It can be seen that the transition probability of a Markov process cannot obey a nonlinear equation of the form (1.4). The argument is similar to the one used by D. Polder, Philos. Mag. 45, 69 (1954). 

The master equation is an equivalent form of the Chapman-Kolmogorov equation for Markov processes, but it is easier to handle and more directly related to physical concepts. It will be the pivot of most of the work in this book. 

Consider a Markov process, which for convenience we take to be homogeneous, so that we may write Tx for the transition probability. The Chapman-Kolmogorov equation (IV.3.2) for Tx is a functional relation, which is not easy to handle in actual applications. The master equation is a more convenient version of the same equation it is a differential equation obtained by going to the limit of vanishing time difference t. For this purpose it is necessary first to ascertain how Tx> behaves as x tends to zero. In the previous section it was found that TX (y2 yl) for small x has the form ... 

This interpretation of the master equation means that is has an entirely different role than the Chapman-Kolmogorov equation. The latter is a nonlinear equation, which results from the Markov character, but contains no specific information about any particular Markov process. In the master equation, however, one considers the transformation probabilities as given by the specific system, and then has a linear equation for the probabilities which determine the (mesoscopic) state of that system. 

Exercise. If W is a 2 x 2 matrix the exponential in (2.4) can be evaluated directly by expanding in powers of W. Use this method for solving the master equation of the dichotomic Markov process. 

Exercise. For Markov processes that are not stationary or homogeneous one also has a forward, or master equation and a backward equation,... 

Solving this equation means determining the probability to find at t > 0 the system in (i, r) when it was at t = 0 in (i0, r0). This problem decomposes into two successive steps first find how the molecule jumps among the levels regardless of r, and subsequently add on the behavior in r. This is the reason why we use the name composite Markov process for any random process obeying a master equation of the type (7.4). ... 

The Fokker-Planck equation is a special type of master equation, which is often used as an approximation to the actual equation or as a model for more general Markov processes. Its elegant mathematical properties should not obscure the fact that its application in physical situations requires a physical justification, which is not always obvious, in particular not in nonlinear systems. 

Markov processes whose master equation has the form (1.1) have been called continuous , because it can be proved that their sample functions are continuous (with probability 1). This name has sometimes led to the erroneous idea that all processes with a continuous range are of this type and must therefore obey (1.1). 

Another, even simpler choice for is the two-valued Markov process (IV.2.3). The joint master equation reduces to two coupled equations for the pair of singlevariable functions (y, 1, t) = P (y, t),... 

In terms of the master equation for the Markov process the formal kinetics is nothing but the mean-field theory where the fluctuation terms like that on the r.h.s. of equation (2.2.43) are neglected. Strictly speaking, the macroscopic description, equation (2.1.2), were correct if the fluctuation terms vanished as V —> oo. In a general case the function P(N, t) does not satisfy the Poisson distribution [16, 27] in particular, °N (N> ... 

Note here that the relation between mesoscopic and microscopic approaches is not trivial. In fact, the former is closer to the macroscopic treatment (Section 2.1.1) which neglects the structural characteristics of a system. Passing from the micro- to meso- and, finally, to macroscopic level we loose also the initial statement of a stochastic model of the Markov process. Indeed, the disadvantages of deterministic equations used for rather simplified treatment of bimolecular kinetics (Section 2.1) lead to the macro- and mesoscopic models (Section 2.2) where the stochasticity is kept either by adding the stochastic external forces (Section 2.2.1) or by postulating the master equation itself for the relevant Markov process (Section 2.2.2). In the former case the fluctuation source is assumed to be external, whereas in the latter kinetics of bimolecular reaction and fluctuations are coupled and mutually related. Section 2.3.1.2 is aimed to consider the relation between these three levels as well as to discuss problem of how determinicity and stochasticity can coexist. 

A structure of the obtained set of equations derived by us in [81, 86] is very close to the famous BBGKI set of equations widely used in the statistical physics of dense gases and liquids [76]. Therefore, we presented the master equation of the Markov process in a form of the infinite set of deterministic coupled equations for averages (equation (2.3.34)). Practical use of these equations requires us to reduce them, retaining the joint correlation functions only. 

Note that willing to stress the relation between many-particle densities and master equation for the Markov process, we followed the formalism presented by us [81] rather than that used in the pioneering papers by Waite [84, 94, 95] and Leibfried [96], as well as in more recent studies [82, 97, 103-105] where... 

Given all these quantities, the master equation could be written for a set of reactions (2.1.27) as the Markov process. (Actual choice of the functions [Pg.474]

To formulate this stochastic model in terms of concentrations and joint correlation functions only, i.e., in a manner we used earlier in Chapters 2, 4 and 5, it is convenient to write down a master equation of the Markov process under study in a form of the infinite set of coupled equations for many-point densities. Let us write down the first equations for indices (m + m ) = 1 ... 

For a continuous Markov process, the master equation is of the form... 

For many physical applications, modeled by a homogeneous Markov process in time and space, the rate of transition is time independent and depends only on the difference of the starting and arriving states. Therefore, one can see that the master equation is given by... 

Stochastic analysis presents an alternative avenue for dealing with the inherently probabilistic and discontinuous microscopic events that underlie macroscopic phenomena. Many processes of chemical and physical interest can be described as random Markov processes.1,2 Unfortunately, solution of a stochastic master equation can present an extremely difficult mathematical challenge for systems of even modest complexity. In response to this difficulty, Gillespie3-5 developed an approach employing numerical Monte Carlo... 

Thus, Eq. (55) becomes identical to Eq. (2), thereby recovering an important result obtained many years ago [38] A Poisson process, with /(f) given by an exponential function of t, yields a Markov master equation. 

Chemical reaction networks are frequently modeled by Markov processes and can be formulated as master equations. Commonly, it is straightforward to write down the master equation, but when it comes to derive solutions, hard-to-justify approximations are inevitable see, for example, ref. 83. In essence, the same is true for polynucleotide replication described by a master... 

For translational long-range jump diffusion of a lattice gas the stochastic theory (random walk, Markov process and master equation) [30] eventually yields the result that Gg(r,t) can be identified with the solution (for a point-like source) of the macroscopic diffusion equation, which is identical to Pick s second law of diffusion but with the tracer (self diffusion) coefficient D instead of the chemical or Fick s diffusion coefficient. 

---