Sampling from a Markov Chain

Example 6 First, we will notice that there are many Markov chains that will have the same long-run distribution. We have two transition probability matrices Pi and P2 that describe the movement through a finite state-space with five elements. They are  

Row i represents the occupation probability distribution at time n given a start in state i. We see that the first chain has converged its long-run distribution (to within 6 significant digits). We see the second chain is far from convergence since these occupation probabilities are very different for the different rows. We let the second chain run further ton = 2 , and P now equals 

We see that the chain still has not converged to a long-run distribution since the rows are clearly not the same. We let the secotul chain run further to n = 2 and pl  

W see that the second chain has now converged to the same long-run distribution as the first chain. 

---

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

Using the Markov chain samples i, O2,..., TZt> in Equation (2.126) is estimated as the average Ht) of Q over the samples, which is the same as the usual MCS estimator, except that the samples are simulated from a Markov chain instead of being independent and identically distributed. Nevertheless, the estimator has similar statistical properties as those of the MCS estimators. In order to reduce the initial transient effect of the Markov chain on the estimate, the first few samples (say 10) may be discarded when computing x>- Thus, the Markov chain samples 1,62, On used for computing It-p are those simulated after the initial transient stage. 

In Chapter 7 we develop a method for finding a Markov chain that has good mixing properties. We will use the Metropolis-Hastings algorithm with heavy-tailed independent candidate density. We then discuss the problem of statistical inference on the sample from the Markov chain. We want to base our inferences on an approximately random sample from the posterior. This requires that we determine... 

We can use any of these methods to draw a sample from the posterior. However, the samples won t be random. Draws from a Markov chain are serially correlated. 

In this exereise, we compare the effectiveness of several ways to draw a sample from the Markov chain when the multiple parameters are strongly correlated. 

STATISTICAL INFERENCE FROM A MARKOV CHAIN MONTE CARLO SAMPLE... 

Chapter 7 Statistical Inference from a Markov chain Monte Carlo Sample... 

Sequential draws from a Markov chain are serially dependent. A Markov chain Monte Carlo sample will not be suitable for inference until we have discarded the draws from the burn-in period and thinned the sample so the thinned sample will approximate a random sample from the posterior. In Table A.8 we give the Minitab commands for thinning the output of a MCMC process using the macro ThinMCMC.mac. 

Since we will, in general, not have a direct algorithm to create samples Xi according to the distribution P xi) we will use a Markov process in which starting from an initial configuration xq a Markov chain of configuration is generated ... 

A series of probable transitions between states can be described with the Markov chain. A Markovian stochastic process is memoryless, and this is illustrated subsequently. We generate a sequence of random variables, (yo, yi, yi, ), so that each time t > 0, the next state yt+i would be sampled from a distribution P(y,+ily,), which would depend only on the current state of the chain, y,. Thus, given y, the next state y,+i would not depend additionally on the history of the chain (yo, yi, yi,---, y i). The name Markov chain is used to describe this sequence, and the transition kernel of the chain is i (.l.) does not depend on t if we assume that the chain is time homogeneous. A detail description of the Markov model is provided in Chapter 26. 

Here, the conformational space is sampled by a set of MC moves through a Markov chain process. A trial move from a conformation (or... 

By estimating the correlation sequence /o (n) from the Markov chain samples, in Equation (2.141), and hence S in Equation (2.140), can be estimated in a single simulation run. 

Figure 3. Line Sampling important unit vector a taken as the normalized center of mass of the failure domain F in the standard normal space. The center of mass of F is computed as an average of JV failure points generated by means of a Markov chain starting from an initial failure point (f (Koutsourelakis etal., 2004).

Variability measures The likelihood functions and prior distributions have been incorporated in a Bayesian inference procedure in which the posterior density n 01E) is computed. The Bayesian inference is performed by using a Markov Chain Monte Carlo Method (MCMC), which allows samples to be generated from a continuous unnormaUzed density (Chiband Greenberg, 1995). The MCMC method, which is frequently applied to Bayesian Inference problems (Gilks et al. 1996), results in a m-samples set S = 0,. .., 0, representing the... 

---