Discrete time state space model description

The models discrete in space and continuous in time as well as those continuous in space and time, led many times to non-linear differential equations for which an analytical solution is extremely difficult or impossible. In order to solve the equations, simplifications, e.g. linearization of expressions and assumptions must be carried out. However, if this is not sufficient, one must apply numerical solutions. This led the author to a major conclusion that there are many advantages of using Markov chains which are discrete in time and space. The major reason is that physical models can be presented in a unified description via state vector and a one-step transition probability matrix. Additional reasons are detailed in Chapter 1. It will be shown later that this presentation coincides also with the fact that it yields the finite difference equations of the process under consideration on the basis of which the differential equations have been derived. 

Crew Station/equipment characteristics The crew station design module and library is a critical component in the MIDAS operation. Descriptions of discrete and continuous control operation of the equipment simulations are provided at several levels of functiontil deteiil. The system can provide discrete equipment operation in a stimulus-response (blackbox) format, a time-scripted/ event driven format, or a full discrete-space model of the transition among equipment states. Similarly, the simulated operator s knowledge of the system can be at the same varied levels of representation or can be systematically modified to simulate various states of misunderstanding the equipment function. 

In order to give a good description of the problem, we shall model it as a Markov Decision Problem (MDP). Markov Decision Processes have been studied initially by Bellmann (1957) and Howard (1960). We will first give a short description of an MDP in general. Suppose a system is observed at discrete points of time. At each time point the system may be in one of a finite number of states, labeled by 1,2,.., M. If, at time t, the system is in state i, one may choose an action a, fix>m a finite space A. This action results in a probability PJ- of finding the system in state j at time r+1. Furthermore costs qf have to be paid when in state i action a is taken. 

---