Markov process homogeneous

For many physical applications, modeled by a homogeneous Markov process in time and space, the rate of transition is time independent and depends only on the difference of the starting and arriving states. Therefore, one can see that the master equation is given by... 

The concept of infinitesimal operator is frequently used when the random evolutions are the generators of stochastic models from a mathematical point of view. This operator can be defined with the help of a homogeneous Markov process X(t) where the random change occurs with the following transition probabilities ... 

Each of the optional dynamical models mentioned above involves a homogeneous Markov process Xt = Xt teT in either continuous or discrete time on some state space X. The motion of Xt is given in terms of the stochastic... 

Consider a collection of particles that move independently of each other in three-dimensional space R. We assume that the position of a particle X(r) is a time-homogeneous Markov process with transition density p(y, r x). 

Since with a homogeneous Markov process the probabilities of transition, Poo(At), Pn(At), Poi(At) and Pio(At), do not depend on the point in time t but only on the duration of the time interval At the corresponding rates (failure and repair rate) must be constants. 

Consider a n-components dynamic system described by an irreducible homogenous Markov process = Xj, t > 0 (initial state /) with finite state space E and the transition rate matrix M. This Markov process is ergodic and a single stationary distribution exists (Ross 1996). Let a row vector jt = (tti, 7T2,. ..) be the vector of steady state probabilities (stationary distribution vector). Chapman-Kolmogorov equations at steady state can be written as ... 

Proposition 4.1 Z + is a homogeneous Markov Process. The non-zero conditional probabilities... 

Even when a system is in a steady state other than equilibrium certain physical quantities may be stationary Markov processes. An example are the current fluctuations in the circuit of fig. 7 when a battery is added, which maintains a constant potential difference and therefore a non-zero average current. Another example is a Brownian particle in a homogeneous gravitational field its vertical velocity is a stationary process, but not its position. 

These processes are non-stationary because the condition singled out a certain time t0. Yet their transition probability depends on the time interval alone as it is the same as the transition probability of the underlying stationary process. Non-stationary Markov processes whose transition probability depends on the time difference alone are called homogeneous processes. 10 They usually occur as subensembles of stationary Markov processes in the way described here. However, the Wiener process defined in 2 is an example of a homogeneous process that cannot be embedded in a stationary Markov process. 

The name refers to homogeneity in time. Unfortunately it is somewhat confusing because a process may also be homogeneous in space, i.e., invariant for a transition in the space of its states y. We shall therefore often prefer the circumlocution Markov process with stationary transition probability . 

The extraction of a homogeneous process from a stationary Markov process is a familiar procedure in the theory of linear response. As an example take a sample of a paramagnetic material placed in a constant external magnetic field B. The magnetization Y in the direction of the field is a stationary stochastic process with a macroscopic average value and small fluctuations around it. For the moment we assume that it is a Markov process. The function Px (y) is given by the canonical distribution... 

Consider a Markov process, which for convenience we take to be homogeneous, so that we may write Tx for the transition probability. The Chapman-Kolmogorov equation (IV.3.2) for Tx is a functional relation, which is not easy to handle in actual applications. The master equation is a more convenient version of the same equation it is a differential equation obtained by going to the limit of vanishing time difference t. For this purpose it is necessary first to ascertain how Tx> behaves as x tends to zero. In the previous section it was found that TX (y2 yl) for small x has the form ... 

Exercise. For Markov processes that are not stationary or homogeneous one also has a forward, or master equation and a backward equation,... 

The transition matrix P is, thus, a complete description of the Markov process. Any homogeneous Markov chain has a stochastic matrix of transition probabilities and any stochastic matrix defines a homogeneous Markov chain. 

Time homogeneity is another important distinction of Markov processes. The process is time independent or time homogeneous when the transition probabilities are constant regardless of the time of observation (12), and the distribution of the number of transitions into a state follows a homogeneous or stationary Poisson process. The Poisson distribution is defined as P N(t) = k = (AfV )/ , where A is the average number of transitions per period t (or the rate of arrivals) over k cycles (17). An exponential distribution defined by the same parameter X is used to characterize the time between transitions in a homogeneous Poisson process (18). 

The steady-state solution technique is useful for many situations. However, it is not appropriate for situations where the probability of moving from state to state is not constant (a non-homogeneous Markov model). It is also not appropriate for absorbing Markov models. This solution technique is not appropriate for safety instrumented functions where many failures are not detected until a periodic inspection and repair is performed. In the case of failures detected by a non-constant inspection and test process, the probability of repair is not constant. It is zero for most time periods. Do not use steady-state techniques to model repair processes with inspection and test. 

A Markov process describes the states of a system (enumerably many states are admitted) as a function of time. Its characteristic is that the progression of the process at any point in time t only depends on its state in t and not on states prior to t. If, in addition, it is homogeneous, as is supposed here, the probability of the transition of the state of the system at point in time t to its state at point in time t + At depends only on the duration of At and not on the point in time t. A further assumption is that the probability of more than one change of state in At can be neglected. In order to formulate the model the following quantities are needed ... 

A faster numerical procedure for solving non-homogeneous semi-Markov processes... 

Continuous time non-homogeneous semi-Markov processes (CTNHSMP) are powerful modeling tools, especially in the reliability field (as exemplified in Janssen Manca (2007)). According to Becker et al. (2000), CTNHSMP are considered as approaches to model reliability characteristics of components or small systems with complex test and maintenance strategies. 

The semi-Markov process is therefore considered as non-homogeneous so that this deterioration process may be adequately addressed. Therefore, the required data to estimate the system availability via this CTNHSMP model are the parameters pyi ) and... 

To conclude, two important limitations of this work deserve attention. Firstly discussing semi-Markov processes in general, we have the well-known and already cited difficulty in obtaining the requisite data to analyze semi-Markov processes on the non-homogeneous environment. On that, El-Gohary (2004) presents max-iminn likelihood and Bayes estimates of the parameters included in a semi-Markov reliability model of three states. 

Janssen, J. Manca, R. 2001. Numerical solution of Non Homogeneous semi Markov processes in Transient Case. Methodology and Computing in Applied Probability, 3(271-293). 

Moura, M. C. Droguett, E. L. 2009. Mathematical formulation and numerical treatment based on transition frequency densities and quadrature methods for non-homogeneous semi-Markov processes. Reliability Engineering System Safety, 94(2) 342-349. 

The first aim was to translate textual certification requirements into a stochastic model. The relevant figure turned out to be the expected number of failures, the time scale is the number of flight hours , and repairs are important. The points of time when repair is possible are the set of positive integers. This makes the continuous time process in-homogenous. So, standard tools for solving Markov Processes cannot be applied. 

A further extension of these ideas, in which multiple states that evolve in time are possible, is obtained when one models the speech signal by a hidden Markov process (HMP) [8]. An HMP is a bivariate random process of states and observations sequences. The state process S t = 1,2,... is a finite-state homogeneous Markov chain that is not directly observed. The observation process yf,t = 1,2,...) is conditionally independent given the state process. Thus, each observation depends statistically only on the state of the Markov chain at the same time and not on any other states or observations. Consider, for example, an HMP observed in an additive white noise process W),t = 1,2,...). For each t, let Zt = Yt + Wt denote the noisy signal. Let Z = Zi,..., Z,. Let / denote the number of states of the Markov chain. The causal MMSE estimator of Y, given Z is given by [6]... 

We assume that the degradation process of the centrifugal pump is modeled by a continuous-time homogeneous Markov chain with constant transition rates as shown in Figure 3. 

This is clearly a (generalized) renewal property. One can easily generalize this formula and obtain analogous formulas for S based models. We will repeatedly use this property, but mainly in a non-explicit way and we will mostly manipulate (restricted) partition functions. Homogeneous Markov and renewal processes are manageable due to their local nature, but, in inhomogeneous frameworks, they may instead display sharply nonlocal features and tools to analyze them go well beyond the tools used for homogeneous systems. 

Some restrictions are imposed when we start the application of limit theorems to the transformation of a stochastic model into its asymptotic form. The most important restriction is given by the rule where the past and future of the stochastic processes are mixed. In this rule it is considered that the probability that a fact or event C occurs will depend on the difference between the current process (P(C) = P(X(t)e A/V(X(t))) and the preceding process (P (C/e)). Indeed, if, for the values of the group (x,e), we compute = max[P (C/e) — P(C)], then we have a measure of the influence of the process history on the future of the process evolution. Here, t defines the beginning of a new random process evolution and tIt- gives the combination between the past and the future of the investigated process. If a Markov connection process is homogenous with respect to time, we have = 1 or Tt O after an exponential evolution. If Tt O when t increases, the influence of the history on the process evolution decreases rapidly and then we can apply the first type limit theorems to transform the model into an asymptotic... 

---