Continuous state-space processes

More precisely, for diffusion processes the following conditions are fulfilled  

The verbal interpretation of (5.5) is that the process is continuous — this is the Lindeberg condition. The function a x, t) is the velocity of conditional expectation ( drift vector ), and bjj(x, t) is the matrix of the velocity of conditional covariance ( diffusion matrix ). The latter is positive semidefinite and symmetric as a result of its definition (5.7). 

The forward Kolmogorov equation (5.10) is referred to in physical literature as the Fokker-Planck equation. For the absolute density function g(x, t) (which contains less information than/) the following Kolmogorov-like equation holds  

We adopt the adjective Kolmogorov-like , since the Kolmogorov equation is used to refer only to (5.9) and (5.10) for conditional (or transitional) quantities. 

Equation (5.11) is a generalisation of the diffusion or heat conduction equation. 

---

In the t3rpical case the d3mamical process under investigation lives on a continuous state space such that the transfer operator does not have the form of a nice stochastic matrix. Therefore, discretization of the transfer operator is needed to 3deld a stochastic matrix with which one can proceed as in the example above. 

Noncontinuous processes in continuous state-space occur when the condition (5.5) is not fulfilled. In this case we need more general equations than the Kolmogorov equations. The main point is that analogously to (5.6) and (5.7) the yth velocity of conditional moments Dj can be defined ... 

The terminology is nonstandard, and in physical literature the Kramers-Mpyal expansion is given as a (nonsystematic) procedure to approximate discrete state-space processes by continuous processes. The point that we want to emphasise here is the clear fact that, even in the case of a continuous state-space, the process itself can be noncontinuous, when the Lindeberg condition is not fulfilled. The functions for the higher coefficients do not necessarily have to vanish. 

A great amount of stochastic physics investigates the approximation of jump processes by diffusion processes, i.e. of the master equation by a Fokker-Planck equation, since the latter is easier to solve. The rationale behind this procedure is the fact that the usual deterministic (CCD) and stochastic (CDS) models differ from each other in two aspects. The CDS model offers a stochastic description with a discrete state space. In most applications, where the number of particles is large and may approach Avogadro s number, the discreteness should be of minor importance. Since the CCD model adopts a continuous state-space, it is quite natural to adopt CCS model as an approximation for fluctuations. 

Other kinds of Fokker-Planck equations can be also derived. The continuous state-space stochastic model of a chemical reaction, which considers the reaction as a diffusion process , neglects the essential discreteness of the mesoscopic events. However, some shortcomings of (5.65) have been eliminated by using a direct Fokker-Planck equation obtained by means of nonlinear transport theory (Grabert et al., 1983). 

In Section 5.1 we introduce the stochastic processes. In Section 5.2 we will introduce Markov chains and define some terms associated with them. In Section 5.3 we find the n-step transition probability matrix in terms of one-step transition probability matrix for time invariant Markov chains with a finite state space. Then we investigate when a Markov ehain has a long-run distribution and discover the relationship between the long-run distribution of the Markov chain and the steady state equation. In Section 5.4 we classify the states of a Markov chain with a discrete state space, and find that all states in an irreducible Markov chain are of the same type. In Section 5.5 we investigate sampling from a Markov chain. In Section 5.6 we look at time-reversible Markov chains and discover the detailed balance conditions, which are needed to find a Markov chain with a given steady state distribution. In Section 5.7 we look at Markov chains with a continuous state space to determine the features analogous to those for discrete space Markov chains. 

Markov process A stochastic process is a random process in which the evolution from a state X(t ) to X(t +i) is indeterminate (i.e. governed by the laws of probability) and can be expressed by a probability distribution function. Diffusion can be classified as a stochastic process in a continuous state space (r) possessing the Markov property as... 

It may be useful to point out a few topics that go beyond a first course in control. With certain processes, we cannot take data continuously, but rather in certain selected slow intervals (c.f. titration in freshmen chemistry). These are called sampled-data systems. With computers, the analysis evolves into a new area of its own—discrete-time or digital control systems. Here, differential equations and Laplace transform do not work anymore. The mathematical techniques to handle discrete-time systems are difference equations and z-transform. Furthermore, there are multivariable and state space control, which we will encounter a brief introduction. Beyond the introductory level are optimal control, nonlinear control, adaptive control, stochastic control, and fuzzy logic control. Do not lose the perspective that control is an immense field. Classical control appears insignificant, but we have to start some where and onward we crawl. 

A class of continuous-time Markov processes with integer state space and with transitions allowed only between adjacent states plays a central role in the stochastic description of chemical kinetics.122 134 135... 

Each of the optional dynamical models mentioned above involves a homogeneous Markov process Xt = Xt teT in either continuous or discrete time on some state space X. The motion of Xt is given in terms of the stochastic... 

Markov chains or processes are named after the Russian mathematician A.A.Markov (1852-1922) who introduced the concept of chain dependence and did basic pioneering work on this class of processes [1]. A Markov process is a mathematical probabilistic model that is very useful in the study of complex systems. The essence of the model is that if the initial state of a system is known, i.e. its present state, and the probabilities to move forward to other states are also given, then it is possible to predict the future state of the system ignoring its past history. In other words, past history is immaterial for predicting the future this is the key-element in Markov chains. Distinction is made between Markov processes discrete in time and space, processes discrete in space and continuous in time and processes continuous in space and time. This book is mainly concerned with processes discrete in time and space. 

The models discrete in space and continuous in time as well as those continuous in space and time, led many times to non-linear differential equations for which an analytical solution is extremely difficult or impossible. In order to solve the equations, simplifications, e.g. linearization of expressions and assumptions must be carried out. However, if this is not sufficient, one must apply numerical solutions. This led the author to a major conclusion that there are many advantages of using Markov chains which are discrete in time and space. The major reason is that physical models can be presented in a unified description via state vector and a one-step transition probability matrix. Additional reasons are detailed in Chapter 1. It will be shown later that this presentation coincides also with the fact that it yields the finite difference equations of the process under consideration on the basis of which the differential equations have been derived. 

The characteristics of the state space being measured can be used to classify the Markov process. For most purposes, a discrete or finite space is assumed and this implies that there are a finite number of states that will be reached by the process (14). A continuous or infinite process is also possible. Time intervals of observation of a process can be used to classify a Markov process. Processes can be observed at discrete or restricted intervals, or continuously (15). 

In this section, classical state-space models are discussed first. They provide a versatile modeling framework that can be linear or nonlinear, continuous- or discrete-time, to describe a wide variety of processes. State variables can be defined based on physical variables, mathematical solution convenience or ordered importance of describing the process. Subspace models are discussed in the second part of this section. They order state variables according to the magnitude of their contributions in explaining the variation in data. State-space models also provide the structure for... 

To include the information about process d3mamics in the models, the data matrix can be augmented with lagged values of data vectors, or model identification techniques such as subspace state-space modeling can be used (Section 4.5). Negiz and Cinar [209] have proposed the use of state variables developed with canonical variates based realization to implement SPM to multivariable continuous processes. Another approach is based on the use of Kalman filter residuals [326]. MSPM with dynamic process models is discussed in Section 5.3. The last section (Section 5.4) of the chapter gives a brief survey of other approaches proposed for MSPM. 

The Poisson counting process of Section 2 is a continuous-time Markov chain N on the infinite state space 0, 1, 2,. . . , with generator... 

In the case of a Markov process with state space a subset of M, and with F the associated Borel a-algebra, and with a continuous index variable t e R+, the transition probabilities are defined instead for sets Ae... 

The ideal continuous plug flow reactor (CPFR) has no profile at any point of the tube in the steady state. The process, however, advances along the tube, and so shows a longitudinal concentration variation. The profile of a CPFR in space is identical to the profile of a DCSTR in time in case of a constant volume process this fact is of great importance for process design ( kinetic similarity ). 

The particles of interest to us have both internal and external coordinates. The internal coordinates of the particle provide quantitative characterization of its distinguishing traits other than its location while the external coordinates merely denote the location of the particles in physical space. Thus, a particle is distinguished by its internal and external coordinates. We shall refer to the joint space of internal and external coordinates as the particle state space. One or more of either the internal and/or external coordinates may be discrete while the others may be continuous. Thus, the external coordinates may be discrete if particles can occupy only discrete sites in a lattice. There are several ways in which the internal coordinates may be discrete. A simple example is that of particle size in a population of particles, initially all of uniform size, undergoing pure aggregation, for in this case the particle size can only vary as integral multiples of the initial size. For a more exotic example, let the particle be an emulsion droplet (a liquid) in which a precipitation process is carried out producing a discrete number of precipitate particles. Then the number of precipitate particles may serve to describe the discrete internal coordinate of the droplet, which is the main entity of population balance. 

We recall the domain A t) in particle state space considered in Section 2.6, which is initially at and continuously deforming in time and space. For the present, the particles are regarded as firmly embedded in the deforming particle state continuum described in Section 2.5. The only way in which the number of particles in A t) can change is by birth and death processes. We assume that this occurs at the net birth rate of /i(x, r, Y, t) per unit volume of particle state space so that the number conservation may be written as... 

The model is represented as a state space model with six times nine states, corresponding to the nine compositions in each six units. Luckily, the calculation of the model parameters is relatively simple because of the batch-wise function of the process The loss-in-weight silo is filled about every tenth minute from the previous silo, after which concentrate is pneumatically transported to this silo from the silos after the dryers. These two silos are continuously filled with the concentrate from the dryers and the mass in the steam dryers can be assumed to be constant. The steam dryers and silo units are modelled as first-order systems... 

The value that the process has at time t is called its state at time t. The set of possible values that the stochastic process has at any particular time is called the state space, and can be either discrete or continuous. The second thing that characterizes a stochastic process is the size and type of state space allowed. The state space may be one or more dimensions. In each dimension there may be a finite number of discrete possible values, an infinite number of discrete possible values, or all values in a continuous interval. If the possible values in each dimension are discrete, the process is equivalent to a process on a single dimensional discrete space, so the types of state space we need to consider include ... 

---