Monte Carlo Markov process

Under certain circumstances it can be shown that in the limit of large number of iterations of the process, the random variable X tends to X (in distribution). Thus the Monte-Carlo Markov Chain method can be used to generate sets of independent realizations of the random variable which can be used to calculate an expectation. In practice, one prefers to use a long sequence of the iterates from a single starting value. The efficacy (specifically the ergodicity and rate of convergence) of the Monte-Carlo Markov-Chain method depends on the choice of the transition density. 

While static Monte Carlo methods generate a sequence of statistically independent configurations, dynamic MC methods are always based on some stochastic Markov process, where subsequent configurations X of the system are generated from the previous configuration X —X —X" — > with some transition probability IF(X —> X ). Since to a large extent the choice of the basic move X —X is arbitrary, various methods differ in the choice of the basic unit of motion . Also, the choice of transition probability IF(X — > X ) is not unique the only requirement is that the principle... 

The Monte Carlo method also starts from an initial configuration of the positions, but does not consider the momenta (r (0)). Next, a succession of configurations, kept at a constant temperature T, are computed as a Markov process ... 

Stochastic analysis presents an alternative avenue for dealing with the inherently probabilistic and discontinuous microscopic events that underlie macroscopic phenomena. Many processes of chemical and physical interest can be described as random Markov processes.1,2 Unfortunately, solution of a stochastic master equation can present an extremely difficult mathematical challenge for systems of even modest complexity. In response to this difficulty, Gillespie3-5 developed an approach employing numerical Monte Carlo... 

MCMC methods are essentially Monte Carlo numerical integration that is wrapped around a purpose built Markov chain. Both Markov chains and Monte Carlo integration may exist without reference to the other. A Markov chain is any chain where the current state of the chain is conditional on the immediate past state only—this is a so-called first-order Markov chain higher order chains are also possible. The chain refers to a sequence of realizations from a stochastic process. The nature of the Markov process is illustrated in the description of the MH algorithm (see Section 5.1.3.1). 

An HMM is essentially a Markov chain (—> Monte Carlo methods). Each state inside the Markov chain can produce a letter, and it does so by a stochastic Poisson process that chooses one of a finite number of letters in an alphabet. Each letter has a characteristic probability of being chosen. That probability depends on the state and on the letter. After the state produces a letter, the Markov chain moves to another state. This happens according to a transition probability that depends on the prior and succeeding state. 

Data augmentation using Markov Chain Monte Carlo (MCMC), which has its basis in Bayesian statistics, is much like the EM algorithm except that two random draws are made during the process. Markov... 

The presentation of the Monte Carlo method in terms of a Markov chain with a time parameter [see Eq. (2.1)] allows us to visualize the process of generating a succession of configurations as a dynamical one... 

One should be careful with this procedure, as in principle it renders a Monte Carlo simulation a non-Markov process. The effect is likely to be benign, but the safest way to proceed is to take the corrector updates of the pressure only during the equilibration phase of the simulation (i.e., those cycles normally granted to allow the system to equilibrate to the new state conditions). In our experience the corrector iteration usually converges very quickly, well before the end of the equilibration period. As a check one can... 

NMR [30] and mass spectroscopy [31] have been used to determine hard segment length in MDI-PTMO polyurethane chains extended with ethylene diamine. Monte Carlo simulations of Markov processes have also been used to derive hard segment molecular weight distributions under ideal and nonideal conditions [32-37]. 

M.H.C. Everdij and H.A.P. Blom (2003). Petri-nets and hybrid-state Markov processes in a power-hierarchy of dependability models. In Engel, Gueguen, Zaytoon (eds.), Analysis and design of hybrid systems, Elsevier, pp. 313-318 M.H.C. Everdij and H.A.P. Blom (2005), Piecewise deterministic Markov processes represented by dynamically coloured Petri nets. Stochastics Vol. 77, pp.1-29 P. Glasserman (2004), Monte Carlo methods in financial engineering. Springer. 

Dynamic Monte Carlo methods are based on stochastic Markov processes where subsequent configurations X are generated from the previous one (Xi X2 X3 ) with some transition probability W(Xi -> X2). To some extent, the choice of the basic move Xj X2 is arbitrary. Various methods, as shown in Fig. 4, just differ in the choice of the basic unit of motion. Furthermore, W is not uniquely defined We only require the principle of detailed balance with the equilibrium distribution Pg X),... 

The precise interpretation of the dynamics associated with the Monte Carlo procedure is that it is a numerical realization of a Markov process described by a master equation for the probabilty P(X, t) that a configuration X occurs at time t,... 

Wang, L.-P. and Stock, D.E. (1992). Stochastic Trajectory Models for Djibulent Diflusion Monte-Carlo Process versus Markov Chains. Atmos. Environ., Vol. 26, pp. 1599-1607. Wang, L.-P. and Stock, D.E. (1993). Dispersion of Heavy Particles by Tuibulent Motion. J. Atmos. Sd., Vol. 50, pp. 1897-1913. 

ABSTRACT This paper presents a holistic treatment to multiple dependent competing degradation processes, by employing the Piecewise-Deterministic Markov Process (PDMP) modeling framework. The proposed method can handle the dependencies between physics-based models, between multi-state models and between these two types of models. A Monte Carlo simulation algorithm is developed to compute the components/systems reliability. A case study on one subsystem of the Residual Heat Removal System (RHRS) of a nuclear power plant is illustrated. 

The state-transition model can be analyzed using a number of approaches as a Markov chains, using semi-Markov processes or using Monte Carlo simulation (Fishman 1996). The applicability of each method depends on the assumptions that can be made regarding faults occurrence and a repair time. In case of the Markov approach, it is necessary to assume that both the faults and renewals occur with constant intensities (i.e. exponential distribution). Also the large number of states makes Markov or semi-Markov method more difficult to use. Presented in the previous section reliability model includes random values with exponential, truncated normal and discrete distributions as well as some periodic relations (staff working time), so it is hard to be solved by analytical methods. 

ABSTRACT The paper presents analytical and Monte Carlo simulation methods applied to the reliability evaluation of a complex multistate system. A semi-Markov process is applied to construct the multistate model of the system operation process and its main characteristics are determined. Analytical linking of the system operation process model with the system multistate reliability model is proposed to get a general reliability model of the complex system operating at varying in time operation conditions and to find its reliability characteristics. The application of Monte Carlo simulation based on the constructed general reliability model of the complex system is proposed to reliability evaluation of a port grain transportation system and the results of this application are illustrated and compared with the results obtained by analytical method. 

The main use of acceptance-rejection-sampling and adaptive-rejection-sampling will be to sample single parameters as part of a larger Markov-chain Monte Carlo process. 

In this chapter we introduce Markov chains. These are a special type of stochastic process, which are processes that move around a set of possible values where the future values can t be predicted with certainty. There is some chance element in the evolution of the process through time. The set of possible values is called the state space of the process. Markov chains have the "memoryless" property that, given the past and present states, the future state only depends on the present state. This chapter will give us the necessary background knowledge about Markov chains that we will need to understand Markov chain Monte Carlo sampling. 

---