The Prediction Error Method

Based on the prediction error method, which minimizes the error between the prediction and the actual process output data, MATLAB (2005) can be used to construct models of basically any structure. For this general model the method 

The PEM command covers all cases of time series models, but also space models which will be discussed in a subsequent chapter. Computationally more efficient routines ate available for special cases  

---

This section will examine the principles and key results for modelling an open-loop process modelled using the general prediction error model given by Eq. (6.4). The foundation for such modelling is the prediction error method, which uses the fact that most models in system identification are used for predicting future values of the process. 

Although for simple models it is possible to estimate the parameters using least-squares, linear regression (see, e.g. (Question 21) in Sect. 3.8.2), for more complex models this is not possible. Instead, more complex methods are required in order to obtain them. One very popular approach is the prediction error method. Parameter estimation using the prediction error method can be summarised as follows ... 

Theorem 6.3 Open-Loop Process Identification (properties of the prediction error method). The prediction error method produces parameter estimates that are unbiased if the prediction error is a white noise signal. 

Proof This will be shown by deriving the Fisher information matrix for the prediction error method. 

Since Theorem 6.3 states that the prediction error method produces unbiased estimates, as m —> oo, the estimated parameter values will approach the true parameter values. Thus, it can be concluded that the prediction error method asymptotically approaches a minimum variance estimator. 

The second method is called direct identification, where the fact that the process is running in closed loop is ignored. In this type of identification, both the process and error structures must be simultaneously estimated. Thus, either a Box-Jenkins or a general prediction error model should be fit. Since this is one of the more common approaches to closed-loop system identification, it is necessary to examine the properties of this approach. It will be assumed that the prediction error method will be used. 

This implies that the prediction error method can be used to estimate the model parameters without taking into consideration the fact that the system is running in closed loop. Eurthermore, the model of the controller is not required nor is any... 

The prediction error method provides consistent parameter estimates. 

The identification of plant models has traditionally been done in the open-loop mode. The desire to minimize the production of the off-spec product during an open-loop identification test and to avoid the unstable open-loop dynamics of certain systems has increased the need to develop methodologies suitable for the system identification. Open-loop identification techniques are not directly applicable to closed-loop data due to correlation between process input (i.e., controller output) and unmeasured disturbances. Based on Prediction Error Method (PEM), several closed-loop identification methods have been presented Direct, Indirect, Joint Input-Output, and Two-Step Methods. 

The basis of all performance criteria are prediction errors (residuals), yt - yh obtained from an independent test set, or by CV or bootstrap, or sometimes by less reliable methods. It is crucial to document from which data set and by which strategy the prediction errors have been obtained furthermore, a large number of prediction errors is desirable. Various measures can be derived from the residuals to characterize the prediction performance of a single model or a model type. If enough values are available, visualization of the error distribution gives a comprehensive picture. In many cases, the distribution is similar to a normal distribution and has a mean of approximately zero. Such distribution can well be described by a single parameter that measures the spread. Other distributions of the errors, for instance a bimodal distribution or a skewed distribution, may occur and can for instance be characterized by a tolerance interval. 

The conclusion is that the model appears to be acceptable. This graph also provides information about how well the method will predict future samples. It is expected that the errors in prediction for component B will be 0.06. This conclu.sion is only possible because the validation set contains many samples that adequately span the calibration space (see Habit 1). A conclusion about the prediction errors for component A will be evaluated after resolving the issue with the unusual sample. 

If the objective of sampling is to provide information for warnings (threshold monitoring) autocorrelated processes can be modelled by the earlier described methods. Not only interpolation is possible but extrapolation can be applied as well. However, the uncertainty in the extrapolated estimate depends on the prediction time. As long as the predicted value, including the prediction error does not exceed the preset warning threshold, no new sample is required. Miiskens derived that the next sample should be taken at a time x after an analytical result x, according to ... 

Cross-validation is an alternative to the split-sample method of estimating prediction accuracy (5). Molinaro et al. describe and evaluate many variants of cross-validation and bootstrap re-sampling for classification problems where the number of candidate predictors vastly exceeds the number of cases (13). The cross-validated prediction error is an estimate of the prediction error associated with application of the algorithm for model building to the entire dataset. 

Savolainen et al. investigated the role of Raman spectroscopy for monitoring amorphous content and compared the performance with that of NIR spectroscopy [41], Partial least squares (PLS) models in combination with several data pre-processing methods were employed. The prediction error for an independent test set was in the range of 2-3% for both NIR and Raman spectroscopy for amorphous and crystalline a-lactose monohydrate. The authors concluded that both techniques are useful for quantifying amorphous content however, the performance depends on process unit operation. Rantanen et al. performed a similar study of anhydrate/hydrate powder mixtures of nitrofurantoin, theophyllin, caffeine and carbamazepine [42], They found that both NIR and Raman performed well and that multivariate evaluation not always improves the evaluation in the case of Raman data. Santesson et al. demonstrated in situ Raman monitoring of crystallisation in acoustically levitated nanolitre drops [43]. Indomethazine and benzamide were used as model... 

It is intuitive that the predictabihty of the dependent variables into the training data set from which a model was estimated will be optimistic, when compared to predicting into an external data set. In such a case, the prediction errors will have a downward bias. Therefore, a method that estimates predictability for external data is needed and this can be executed via the bootstrap. 

Autocorrelation in data affects the accuracy of the charts developed based on the iid assumption. One way to reduce the impact of autocorrelation is to estimate the value of the observation from a model and compute the error between the measured and estimated values. The errors, also called residuals, are assumed to have a Normal distribution with zero mean. Consequently regular SPM charts such as Shewhart or CUSUM charts could be used on the residuals to monitor process behavior. This method relies on the existence of a process model that can predict the observations at each sampling time. Various techniques for empirical model development are presented in Chapter 4. The most popular modeling technique for SPM has been time series models [1, 202] outlined in Section 4.4, because they have been used extensively in the statistics community, but in reality any dynamic model could be used to estimate the observations. If a good process model is available, the prediction errors (residual) e k) = y k)—y k) can be used to monitor the process status. If the model provides accurate predictions, the residuals have a Normal distribution and are independently distributed with mean zero and constant variance (equal to the prediction error variance). 

A more precise method that requires more computational time is cross-validation [155, 332]. It is implemented by excluding part of the data, performing PCA on the remaining data, and computing the prediction error... 

Selection of Optimal Tree. The optimal tree (most accurate tree) is the one having the highest predictive ability. Therefore, one has to evaluate the predictive error of the subtrees and choose the optimal one among them. The most common technique for estimating the predictive error is the cross-validation method, especially when the data set is small. The procedure of performing a cross validation is described earlier (see section 14.2.2.1). In practice, the optimal tree is chosen as the simplest tree with a predictive error estimate within one standard error of minimum. It means that the chosen tree is the simplest with an error estimate comparable to that of the most accurate one. 

Finding only the most important scales with respect to the prediction error does not indicate where in the wavenumber domain these features are located. Often it is observed that going from one resolution level to another has dramatic effect on the quality of the multivariate model. In such a case it is possible to perform a systematic search for where in the wavenumber region for the particular scale the important features are approximately localized. The method described here is only applied to the cluster analysis example, however it could have been used for other multivariate modelling techniques also. In cluster analysis, instead of prediction error, the interest is focused on properties such as overlap between clusters, cluster separation, cluster variance etc. In general, some interesting property H is observed to change with respect to the resolution level. 

The TPW variable selection is a more rapid method compared to both GA and GOLPE. A TPW selected 228 variables using A= 19 PLS factors. The result from the analysis is shown in Fig. 15. The prediction error on the unseen validation set is 2.1%. 

Many procedures have been developed to predict vapor pressure, and the predictive error of eleven different methods have been evaluated using a series of PCBs. A number of approaches use a set of known vapor pressures to develop a correlation with molecular properties that can be used for predictions of unknowns. For example, the free energy of vaporization has been correlated with molecular surface area to predict vapor pressures of PCB congeners. More direct approaches are based on the Clausius-Clapeyron equation, and vapor pressures can be predicted quite effectively for some series of compounds using only boiling points along with melting points for compounds that are sohds at ambient temperatures. 

If the prediction error based on the standard error is compared to the computation with only five wavelengths, for both methods, PCR (SEPcy = 0.519) and PLS (SEPcy = 0.526), Worse results are obtained. This is explained by the fact that... 

Jiang, W., and Simon, R. (2007). A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Stat. Med., 26 5320-5334. 

Cross-validation is one method to check the soundness of a statistical model (Cramer, Bunce and Patterson, 1988 Eriksson, Verhaar and Hermens, 1994). The data set is divided into groups, usually five to seven, and the model is recalculated without the data from each of the groups. Consecutively, predictions are obtained for the omitted compounds and compared to the actual data. The divergences are quantified by the prediction error sum of squares (PRESS sum of squares of predicted minus observed values), which can be transformed to a dimensionless term (Q ) by relating it to the initial sum of squares of the dependent variable (X(AT) )-... 

---