Discrete-time state transition matrix

It should be emphasized that the transition matrix, Eq.(2-91), applies to the time interval between two consecutive service completion where the process between the two completions is of a Markov-chain type discrete in time. The transition matrix is of a random walk type, since apart from the first row, the elements on any one diagonal are the same. The matrix indicates also that there is no restriction on the size of the queue which leads to a denumerable infinite chain. If, however, the size of the queue is limited, say N - 1 customers (including the one being served), in such a way that arriving customers who find the queue full are turned away, then the resulting Markov chain is finite with N states. Immediately after a service completion there can be at most N -1 customers in the queue, so that the imbedded Markov chain has the state space SS = [0, 1,2,. .., N - 1 customers] and the transition matrix ... 

An exhaustive statistical description of living copolymers is provided in the literature [25]. There, proceeding from kinetic equations of the ideal model, the type of stochastic process which describes the probability measure on the set of macromolecules has been rigorously established. To the state Sa(x) of this process monomeric unit Ma corresponds formed at the instant r by addition of monomer Ma to the macroradical. To the statistical ensemble of macromolecules marked by the label x there corresponds a Markovian stochastic process with discrete time but with the set of transient states Sa(x) constituting continuum. Here the fundamental distinction from the Markov chain (where the number of states is discrete) is quite evident. The role of the probability transition matrix in characterizing this chain is now played by the integral operator kernel ... 

In order to show these aspects, let us consider discrete-time Markov chains. The matrix of transition probabilities is denoted / ohbS which is the conditional probability that the system is in the state at time + 1 if it was in the state O) at time n. The time is counted in units of the time interval r. The matrix of transition probabilities satisfies... 

Assume that we successfully identified a metastable decomposition into the sets Di, Dm for a given lag time t. Due to our above results the dynamics is jumping from sets Dk to set Dj with probability p(r, Dk,Dj) during time t. Then, it is an intriguing idea to describe the effective dynamics of the system by means of the Markov chain with discrete states Di,..., Dm and transition matrix P = (Py) with Py = p t, D, Dj). This effective dynamics is Markovian and thus cannot take into account that there may be memory in the system that is much longer than the time span r used to compute the metastable decomposition. 

The Markov matrix defines the stochastic evolution of the system in discrete time. That is, suppose that at time t the probability of finding the system in state S is given by pt(S). If the probability of making a transition from state S to state S is P(S S) (sorry about the hat, we shall take it off... 

In this subsection, we develop a very simple queueing model. This model is a Markov chain IXf) representing the number of jobs present in a queueing system observed at regular discrete times t = 0, 1, 2,. . . The state space is 0, 1, 2,. . . . There are two types of transitions possible arrivals and departures. We write p for the probability that a job arrives in the next time step. We write q for the probability that a job will complete service in the next time step, assuming that there is at least one job present (L(t) > 0). If we write r for 1 — p — q, which is the probability of no state change when there is at least one job present, then the transition matrix of the chain is... 

Just as the rule whereby a discrete-time chain evolves is defined by a matrix P, the transition matrix, the rule whereby a continuous-time chain evolves is defined by a matrix Q, the generator matrix. The off-diagonal entry q, i + j, of Q has the interpretation that q jh is approximately the probability that the chain, starting from state i at a time t, makes a transition to state j by time t + h, where h is small ... 

We have related the continuous-time chain to a discrete-time chain with a fast clock, whose time unit is the small quantity h but whose transition probabilities Pifh) are proportionately small for i + j by (29). This allows us to analyze the continuous-time chain using discrete-time results. AU the basic calculations for continuous-time, finite-state Markov chains may be carried out by taking a Unlit as h — 0 of the discrete-time approximation. For example, the transition matrix P(t), defined in (28), may be derived as foUows. We divide the time intervM [0, t into a large number N of short intervals of length h = t/N, so that the transition matrix P(t) is the A-step transition matrix corresponding to P(h). It foUows from (29) that P(f) is approximately the A-step transition matrix corresponding to the transition matrix I + hQ. This approximation becomes exact as /t — 0, and we have... 

To model the state of the plant, a discrete Markov process is used. To calculate the transition matrix Q of a discrete Markov process, the transition probabilities between both states have to be estimated. All transitions of the recorded inflow data is used. The time series of plant states LOt are calculated by... 

In practice, assuming the discrete time case, the transition matrix includes the transition probabilities between the possible states. Therefore, in this model, market prices are used to find the credit spread and convert the matrix of transition probabilities to the time-dependent risk-neutral matrices Qt t+i- The credit spread is given by Equation (8.32) ... 

So far we have considered a single mesoscopic equation for the particle density and a corresponding random walk model, a Markov process with continuous states in discrete time. It is natural to extend this analysis to a system of mesoscopic equations for the densities of particles Pi(x,n), i = 1,2,..., m. To describe the microscopic movement of particles we need a vector process (X , S ), where X is the position of the particle at time n and S its state at time n. S is a sequence of random variables taking one of m possible values at time n. One can introduce the probability density Pj(jc, n) = 9P(X < x,S = i)/dx and an imbedded Markov chain with the m x m transition matrix H = (/i ), so that the matrix entry corresponds to the conditional probability of a transition from state i to state j. 

As mentioned on page 61, CTRWs are known as semi-Markov processes in the mathematical literature. In this section we provide a brief account of semi-Markov processes. They were introduced by P. Levy and W. L. Smith [253,415]. Recall that for a continuous-time Markov chain, the transitions between states at random times T are determined by the discrete chain X with the transition matrix H = (hij). The waiting time = T - for a given state i is exponentially distributed with the transition rate k , which depends only on the current state i. The natural generalization is to allow arbitrary distributions for the waiting times. This leads to a semi-Markov process. The reason for such a name is that the underlying process is a two-component Markov chain (X , T ). Here the random sequence X represents the state at the th transition, and T is the time of the nth transition. Obviously,... 

To solve for state probabilities, a row matrix indicating starting state probabilities is multiplied by the square transition matrix. Each multiplication represents one discrete time increment. 

Each rotational state is coupled to all other states through the potential matrix V defined in (3.22). Initial conditions Xj(I 0) are obtained by expanding — in analogy to (3.26) — the ground-state wavefunction multiplied by the transition dipole function in terms of the Yjo- The total of all one-dimensional wavepackets Xj (R t) forms an R- and i-dependent vector x whose propagation in space and time follows as described before for the two-dimensional wavepacket, with the exception that multiplication by the potential is replaced by a matrix multiplication Vx-The close-coupling equations become computationally more convenient if one makes an additional transformation to the so-called discrete variable representation (Bacic and Light 1986). The autocorrelation function is simply calculated from... 

The models discrete in space and continuous in time as well as those continuous in space and time, led many times to non-linear differential equations for which an analytical solution is extremely difficult or impossible. In order to solve the equations, simplifications, e.g. linearization of expressions and assumptions must be carried out. However, if this is not sufficient, one must apply numerical solutions. This led the author to a major conclusion that there are many advantages of using Markov chains which are discrete in time and space. The major reason is that physical models can be presented in a unified description via state vector and a one-step transition probability matrix. Additional reasons are detailed in Chapter 1. It will be shown later that this presentation coincides also with the fact that it yields the finite difference equations of the process under consideration on the basis of which the differential equations have been derived. 

Throughout this chapter it has been decided to apply Markov chains which are discrete in time and space. By this approach, reactions can be presented in a unified description via state vector and a one-step transition probability matrix. Consequently, a process is demonstrated solely by the probability of a system to occupy or not to occupy a state. In addition, complicated cases for which analytical solutions are impossible are avoided. 

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

---