Transition probability matrix chains

Such a consideration demonstrated [56] that the sequence distribution in products of arbitrary equilibrium copolycondensation can always be described by some Markov chain with the elements of the transition probability matrix ex-... 

The basic elements of Markov-chain theory are the state space, the one-step transition probability matrix or the policy-making matrix and the initial state vector termed also the initial probability function In order to develop in the following a portion of the theory of Markov chains, some definitions are made and basic probability concepts are mentioned. 

According to [17, p.210], a closed communicating class C of states essentially constitutes a Markov chain which can be extracted and studied independently. If one writes the transition probability matrix P of a Markov chain so that the states in C are written first, and P can be written as ... 

The models discrete in space and continuous in time as well as those continuous in space and time, led many times to non-linear differential equations for which an analytical solution is extremely difficult or impossible. In order to solve the equations, simplifications, e.g. linearization of expressions and assumptions must be carried out. However, if this is not sufficient, one must apply numerical solutions. This led the author to a major conclusion that there are many advantages of using Markov chains which are discrete in time and space. The major reason is that physical models can be presented in a unified description via state vector and a one-step transition probability matrix. Additional reasons are detailed in Chapter 1. It will be shown later that this presentation coincides also with the fact that it yields the finite difference equations of the process under consideration on the basis of which the differential equations have been derived. 

Throughout this chapter it has been decided to apply Markov chains which are discrete in time and space. By this approach, reactions can be presented in a unified description via state vector and a one-step transition probability matrix. Consequently, a process is demonstrated solely by the probability of a system to occupy or not to occupy a state. In addition, complicated cases for which analytical solutions are impossible are avoided. 

Deflnitions. The basic elements of Markov chains associated with Eq.(2-24) are the system, the state space, the initial state vector and the one-step transition probability matrix. Considering refs.[26-30], each of the elements will be defined in the following with special emphasize to chemical reactions occurring in a batch perfectly-mixed reactor or in a single continuous plug-flow reactor. In the latter case, which may simulated by perfectly-mixed reactors in series, all species reside in the reactor the same time. 

In the following, a solution generated by the discrete Markov chains is presented graphically for a large number of chemical reactions of various types. The solution demonstrates the transient response Cj-nAt and emphasizes some characteristic behavior of the reaction. The solution is based on the transition probability matrix P obtained on the basis of the reaction kinetics by applying... 

It should be emphasized that the matrix representation becomes possible due to the Euler integration of the differential equations, yielding appropriate difference equations. Thus, flow systems incorporating heat and mass transfer processes as well as chennical reactions can easily be treated by Markov chains where the matrix P becomes "automatic" to construct, once gaining enough experience. In addition, flow systems are presented in unified description via state vector and a one-step transition probability matrix. 

Example The classic example of a Markov chain is the weather pattern modeling [240]. Again, consider a three-state Markov model in which the states, characterizing the weather on any given day t, are given as follows State 1 rainy State 2 cloudy State 3 sunny. The state transition probability matrix is defined as. 

The transition intensity matrix has the block form (1) but the matrices Qs and Qg are upper triangular. Similarly, the embedded Markov chain will have the transition probability matrix (7). 

When it is not possible or straightforward to derive the desired expression from direct mechanistic reasoning, Lowry [13] has shown how it is possible to combine the various probabilities of reaction in a Markov chain transition matrix and then obtain the CLD and its moments by matrix manipulation. For chains of finite length, termination ( absorption ) probabilities are included in the transition probability matrix and transition probability matrix P can be partitioned into submatrices in the following way ... 

Let P = (pil) be the transition probability matrix of a Markov chain, where Pij is the probability of moving to a state j in the next stage while the process is in state i at the current stage that is, p = Pr[X,. i = X, = /]. A Markov chain is said to have steady state probabilities if the transition probability matrix converges to a constant matrix. Note that the term steady state probability is used here in a rather loose sense since only aperiodic recurrent Markov chains admit this property. 

Every Markov chain with a finite state set has a unique stationary distribution. In addition, if the Markov chain is aperiodic, then it admits steady state probabilities. Given the transition probability matrix, steady state probabilities of a Markov chain can be computed using the methods detailed in Kulkami (1995). 

To model detectability, let us assume that with 0.8 probability, disruption information will be shared with nodes upstream (toward the buyer) and with 0.2 probability information will be shared with nodes downstream in the supply chain of Figure 7.11. Our computational experiments showed that allocating these probabilities yield much more reasonable outputs than assuming that information flows randomly. Then, the transition probability matrix of the supply chain in Figure 7.11 is given in Table 7.10 and the relevant MFPT matrix is given in Table 7.11. 

Transition Probability Matrix of the Supply Chain Network in Figure 7.11 (Example 7.6)... 

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

We will restrict ourselves to time invariant Markov chains where the transition probabilities only depend on the states, not the time n. These are also called homogeneous Markov chains. In this case, we can leave out the time index and the transition probability matrix of the Markov chain is given by... 

For simplicity let us assume that the state space S is discrete (i.e. finite or countably infinite) this is the case in nearly all the applications considered in this chapter. Consider a Markov chain with state space S and transition probability matrix P = p xy) = pxy satisfying the following two conditions ... 

Maximum number of nodes which can exist in a two-hop set Number of nodes in the two-hop neighbourhood of node x which have successfully acquired a time slot One-step transition probability matrix of a Markov chain n-step transition probability matrix of a Markov chain y) Conditional probability that discrete random variable X takes the value x given that discrete random variable Y takes the value y... 

Presently Monte Carlo calculations are based on the technique proposed by Metropolis [22] in 1953 which involves selecting the successive configurations in such a way that they build up a Markov chain [23], The one-step transition probabilities pij are defined as the probability that beginning from the i configuration with qj(N), the configuration j with qj,N> is reached in one step. These probabilities are the elements of the one-step probability matrix associated to the Markov chain and they must fulfill the following conditions ... 

Working with Markov chains, confusion is bound to arise if the indices of the Markov matrix are handled without care. As stated lucidly in an excellent elementary textbook devoted to finite mathematics,24 transition probability matrices must obey the constraints of a stochastic matrix. Namely that they have to be square, each element has to be non-negative, and the sum of each column must be unity. In this respect, and in order to conform with standard rules vector-matrix multiplication, it is preferable to interpret the probability / , as the probability of transition from state. v, to state s (this interpretation stipulates the standard Pp format instead of the pTP format, the latter convenient for the alternative 5 —> Sjinterpretation in defining p ), 5,6... 

In order to show these aspects, let us consider discrete-time Markov chains. The matrix of transition probabilities is denoted / ohbS which is the conditional probability that the system is in the state at time + 1 if it was in the state O) at time n. The time is counted in units of the time interval r. The matrix of transition probabilities satisfies... 

Next, we can consider a Markov chain with three states 1,2,3 with the following matrix of transition probabilities ... 

---