Time homogeneous Markov process

Consider a collection of particles that move independently of each other in three-dimensional space R. We assume that the position of a particle X(r) is a time-homogeneous Markov process with transition density p(y, r x). 

For many physical applications, modeled by a homogeneous Markov process in time and space, the rate of transition is time independent and depends only on the difference of the starting and arriving states. Therefore, one can see that the master equation is given by... 

Each of the optional dynamical models mentioned above involves a homogeneous Markov process Xt = Xt teT in either continuous or discrete time on some state space X. The motion of Xt is given in terms of the stochastic... 

Since with a homogeneous Markov process the probabilities of transition, Poo(At), Pn(At), Poi(At) and Pio(At), do not depend on the point in time t but only on the duration of the time interval At the corresponding rates (failure and repair rate) must be constants. 

We assume that the degradation process of the centrifugal pump is modeled by a continuous-time homogeneous Markov chain with constant transition rates as shown in Figure 3. 

These processes are non-stationary because the condition singled out a certain time t0. Yet their transition probability depends on the time interval alone as it is the same as the transition probability of the underlying stationary process. Non-stationary Markov processes whose transition probability depends on the time difference alone are called homogeneous processes. 10 They usually occur as subensembles of stationary Markov processes in the way described here. However, the Wiener process defined in 2 is an example of a homogeneous process that cannot be embedded in a stationary Markov process. 

The name refers to homogeneity in time. Unfortunately it is somewhat confusing because a process may also be homogeneous in space, i.e., invariant for a transition in the space of its states y. We shall therefore often prefer the circumlocution Markov process with stationary transition probability . 

Consider a Markov process, which for convenience we take to be homogeneous, so that we may write Tx for the transition probability. The Chapman-Kolmogorov equation (IV.3.2) for Tx is a functional relation, which is not easy to handle in actual applications. The master equation is a more convenient version of the same equation it is a differential equation obtained by going to the limit of vanishing time difference t. For this purpose it is necessary first to ascertain how Tx> behaves as x tends to zero. In the previous section it was found that TX (y2 yl) for small x has the form ... 

A series of probable transitions between states can be described with the Markov chain. A Markovian stochastic process is memoryless, and this is illustrated subsequently. We generate a sequence of random variables, (yo, yi, yi, ), so that each time t > 0, the next state yt+i would be sampled from a distribution P(y,+ily,), which would depend only on the current state of the chain, y,. Thus, given y, the next state y,+i would not depend additionally on the history of the chain (yo, yi, yi,---, y i). The name Markov chain is used to describe this sequence, and the transition kernel of the chain is i (.l.) does not depend on t if we assume that the chain is time homogeneous. A detail description of the Markov model is provided in Chapter 26. 

Time homogeneity is another important distinction of Markov processes. The process is time independent or time homogeneous when the transition probabilities are constant regardless of the time of observation (12), and the distribution of the number of transitions into a state follows a homogeneous or stationary Poisson process. The Poisson distribution is defined as P N(t) = k = (AfV )/ , where A is the average number of transitions per period t (or the rate of arrivals) over k cycles (17). An exponential distribution defined by the same parameter X is used to characterize the time between transitions in a homogeneous Poisson process (18). 

The steady-state solution technique is useful for many situations. However, it is not appropriate for situations where the probability of moving from state to state is not constant (a non-homogeneous Markov model). It is also not appropriate for absorbing Markov models. This solution technique is not appropriate for safety instrumented functions where many failures are not detected until a periodic inspection and repair is performed. In the case of failures detected by a non-constant inspection and test process, the probability of repair is not constant. It is zero for most time periods. Do not use steady-state techniques to model repair processes with inspection and test. 

A Markov process describes the states of a system (enumerably many states are admitted) as a function of time. Its characteristic is that the progression of the process at any point in time t only depends on its state in t and not on states prior to t. If, in addition, it is homogeneous, as is supposed here, the probability of the transition of the state of the system at point in time t to its state at point in time t + At depends only on the duration of At and not on the point in time t. A further assumption is that the probability of more than one change of state in At can be neglected. In order to formulate the model the following quantities are needed ... 

Continuous time non-homogeneous semi-Markov processes (CTNHSMP) are powerful modeling tools, especially in the reliability field (as exemplified in Janssen Manca (2007)). According to Becker et al. (2000), CTNHSMP are considered as approaches to model reliability characteristics of components or small systems with complex test and maintenance strategies. 

Moura, M. C. Droguett, E. L. in press. Numerical Approach for Assessing System Dynamic Availability Via Continuous Time Homogeneous Semi-Markov Processes. Methodol ComputAppl Probab. 

The first aim was to translate textual certification requirements into a stochastic model. The relevant figure turned out to be the expected number of failures, the time scale is the number of flight hours , and repairs are important. The points of time when repair is possible are the set of positive integers. This makes the continuous time process in-homogenous. So, standard tools for solving Markov Processes cannot be applied. 

A further extension of these ideas, in which multiple states that evolve in time are possible, is obtained when one models the speech signal by a hidden Markov process (HMP) [8]. An HMP is a bivariate random process of states and observations sequences. The state process S t = 1,2,... is a finite-state homogeneous Markov chain that is not directly observed. The observation process yf,t = 1,2,...) is conditionally independent given the state process. Thus, each observation depends statistically only on the state of the Markov chain at the same time and not on any other states or observations. Consider, for example, an HMP observed in an additive white noise process W),t = 1,2,...). For each t, let Zt = Yt + Wt denote the noisy signal. Let Z = Zi,..., Z,. Let / denote the number of states of the Markov chain. The causal MMSE estimator of Y, given Z is given by [6]... 

Keywords gas distribution networks, non time-homogeneous systems, performance evaluation, Markov regenerative processes, transient stochastic state classes. 

Some restrictions are imposed when we start the application of limit theorems to the transformation of a stochastic model into its asymptotic form. The most important restriction is given by the rule where the past and future of the stochastic processes are mixed. In this rule it is considered that the probability that a fact or event C occurs will depend on the difference between the current process (P(C) = P(X(t)e A/V(X(t))) and the preceding process (P (C/e)). Indeed, if, for the values of the group (x,e), we compute = max[P (C/e) — P(C)], then we have a measure of the influence of the process history on the future of the process evolution. Here, t defines the beginning of a new random process evolution and tIt- gives the combination between the past and the future of the investigated process. If a Markov connection process is homogenous with respect to time, we have = 1 or Tt O after an exponential evolution. If Tt O when t increases, the influence of the history on the process evolution decreases rapidly and then we can apply the first type limit theorems to transform the model into an asymptotic... 

In a CTNHSMP, transitions between two states may depend not only on such states and on the sojourn times (x) (as it occurs with the homogeneous counterpart), but also on both times of the last (v) and next (t) transitions, with x = t — t. The time variable t is also known as the most recent arrival or last entry time, and t is the calendar or process time. Thus, CTNHSMP extend other models such as homogeneous semi-Markov, (non-) homogeneous ordinary Markov and other point stochastic processes. 

---