Stochastic process numerical algorithm

MCMC methods are essentially Monte Carlo numerical integration that is wrapped around a purpose built Markov chain. Both Markov chains and Monte Carlo integration may exist without reference to the other. A Markov chain is any chain where the current state of the chain is conditional on the immediate past state only—this is a so-called first-order Markov chain higher order chains are also possible. The chain refers to a sequence of realizations from a stochastic process. The nature of the Markov process is illustrated in the description of the MH algorithm (see Section 5.1.3.1). 

Gillespie s algorithm numerically reproduces the solution of the chemical master equation, simulating the individual occurrences of reactions. This type of description is called a jump Markov process, a type of stochastic process. A jump Markov process describes a system that has a probability of discontinuously transitioning from one state to another. This type of algorithm is also known as kinetic Monte Carlo. An ensemble of simulation trajectories in state space is required to accurately capture the probabilistic nature of the transient behavior of the system. 

In this section, we consider the description of Brownian motion by Markov diffusion processes that are the solutions of corresponding stochastic differential equations (SDEs). This section contains self-contained discussions of each of several possible interpretations of a system of nonlinear SDEs, and the relationships between different interpretations. Because most of the subtleties of this subject are generic to models with coordinate-dependent diffusivities, with or without constraints, this analysis may be more broadly useful as a review of the use of nonlinear SDEs to describe Brownian motion. Because each of the various possible interpretations of an SDE may be defined as the limit of a discrete jump process, this subject also provides a useful starting point for the discussion of numerical simulation algorithms, which are considered in the following section. 

---