PRESS statistic

Now we come to the Standard Error of Estimate and the PRESS statistic, which show interesting behavior indeed. Compare the values of these statistics in Tables 25-IB and 25-1C. Note that the value in Table 25-1C is lower than the value in Table 25-1B. Thus, using either of these as a guide, an analyst would prefer the model of Table 25-1C to that of Table 25-1B. But we know a priori that the model in Table 25-1C is the wrong model. Therefore we come to the inescapable conclusion that in the presence of error in the X variable, the use of SEE, or even cross-validation as an indicator, is worse than useless, since it is actively misleading us as to the correct model to use to describe the data. 

Another criterion that is based on the predictive ability of PCA is the predicted sum of squares (PRESS) statistic. To compute the (cross validated) PRESS value at a certain k, we remove the ith observation from the original data set (for i = 1,. .., n), estimate the center and the k loadings of the reduced data set, and then compute the fitted value of the ith observation following Equation 6.16, now denoted as x, . Finally, we set... 

However, the PRESS, statistic is not suitable for use with contaminated data sets because it also includes the prediction error of the outliers. Even if the fitted values are based on a robust PCA algorithm, their prediction error might increase the PRESS, because they fit the model poorly. Consequently, the decision about the optimal number of components k,ip, could be wrong. 

As argued for the PRESS statistic (Equation 6.21) in PCA, this RMSECV, statistic is also not suitable for use with contaminated data sets because it includes the prediction error of the outliers. A robust RMSECV (R-RMSECV) measure can be constructed in analogy with the robust PRESS value [61], Roughly said, for each PCR model under investigation (k = 1,. .., km3X), the regression outliers are marked and then removed from the sum in Equation 6.29 or Equation 6.30. By doing this, the RMSECV, statistic is based on the same set of observations for each k. The optimal number of components is then taken as the value kopl for which RMSECV, is minimal or sufficiently small. 

Another conceptually different approach is cross-validation. In Equation (2.19), X is regarded as a model for X, and as such the model should be able to predict the values of X. This can be checked by performing a cross-validation scheme in which parts of X are left out of the calculations and kept apart, the model is built and used to predict the left out entries. The sum of squared differences between the predicted and the real entries serves as a measure of discrepancy. All data in X are left out once, and the squared differences are summed in a so called PRESS statistics (PRediction Error Sum of Squares). The model that gives the lowest PRESS is selected and the pseudo-rank of X is defined as the number of components in that model. 

Define the PRESS statistic to be PRESS = 51", (y - y i) . Here, y is the predicted value for y but object Xj was left out when estimating the parameters in the regression model. Another way of calculating the PRESS statistic is simply by using... 

A suitable criterion function for regression analysis should reflect how well the response values are predicted. In the adaptive wavelet algorithm, the criterion function considered for regression is based on the PRESS statistic and is then converted to a leave-one-out cross-validated R-squared measure as follows... 

The most conceivable difference between the AWA when applied for regression (as opposed to classification) is the criterion function which is implemented. Here, the cross-validated R-squared criterion which is based on the PRESS statistic, is the regression criterion function which is implemented... 

The prediction performance can be validated by using a cross-validation ( leave-one-out ) method. The values for the first specimen (specimen A) are omitted from the data set and the values for the remaining specimens (B-J) are used to find the regression equation of, e.g., Cj on Ay A2, etc. Then this new equation is used to obtain a predicted value of Cj for the first specimen. This procedure is repeated, leaving each specimen out in turn. Then for each specimen the difference between the actual and predicted value is calculated. The sum of the squares of these differences is called the predicted residual error sum of squares or PRESS for short the closer the value of the PRESS statistic to zero, the better the predictive power of the model. It is particularly useful for comparing the predictive powers of different models. For the model fitted here Minitab gives the value of PRESS as 0.0274584. 

As for MLR, an analysis of the residuals should be carried out. The PRESS statistic is 0.00301908, lower than it was for the MLR. Here the T values indicate that all the coefficients other than the constant term are significantly different from zero. (At this stage the possibility of fitting a model with zero intercept could be explored.)... 

Sections 8.9 to 8.11 have given a brief description of methods for making a regression model for multivariate calibration. To summarize, MLR would rarely be used because it cannot be carried out when the number of predictor variables is greater than the number of specimens. Rather than select a few of the predictor variables, it is better to reduce their number to just a few by using PCR or PLS. These methods give satisfactory results when there is correlation between the predictor variables. The preferred method in a given situation will depend on the precise nature of the data an analysis can be carried out by each method and the results evaluated in order to find which method performs better. For example, for the data in Table 8.4 PCR performed better than PLS as measured by the PRESS statistic. 

This PRESS statistic and the model total sum of squares SSj) can be combined to produce a statistic similar to the coefficient of determination (Equation [8.1]) except... 

Berinstein, P. 1999. The Statistical Handbook on Technology. Phoenix, AZ Oryx Press. Statistical information is compiled on a number of technical subjects. The book serves as a tool for watching new technologies emerge because the numbers can serve as a baseline for flourishes or declines in technology. 

There is another way to generate the prediction errors without actually having to split the data set. The idea is to set aside each data point, estimate a model using the rest of the data, and then evaluate the prediction error at the point that was removed. This concept is well known as the PRESS statistic in the statistical community (Myers, 1990) and is used as a technique for model validation of general regression models. However, to our knowledge, the system identification literature has not suggested the use of the PRESS for model structure selection. 

Chapter 3 presents the development of the PRESS statistic as a criterion for structure selection of dynamic process models which are linear-in-the-parameters. Computation of the PRESS statistic is based on the orthogonal decomposition algorithm proposed by Korenberg et al. (1988) and can be viewed as a by-product of their algorithm since very little additional computation is required. We also show how the PRESS statistic can be used as an efficient technique for noise model development directly fi-om time series data. 

Least Squares and the PRESS Statistic using Orthogonal Decomposition... 

This chapter consists of six sections. Section 3.2 introduces the orthogonal decomposition algorithm proposed by Korenberg et al. (1988). Section 3.3 describes the concept of the PRESS statistic. Section 3.4 shows how to compute the PRESS using the orthogonal decomposition algorithm. Section 3.5 applies the PRESS statistic to the problem of model structure selection for dynamic systems. Section 3.6 shows how the PRESS statistic... 

Computation of the true prediction errors e (A ), k = 1,2,is a tremendous task in dynamic system identification where we typically face a large amount of data (M) and possibly high dimensionality (n) of the parameter vector 6. It will be shown here that by using the orthogonal decomposition algorithm, the computation of the PRESS residuals is simplified to an extent that its calculation can be viewed as a byproduct of the algorithm. The following theorem presents the cornerstone for computation of the PRESS statistic. 

This section illustrates application of the PRESS statistic for process model structure selection. Ljung (1987) used data collected from a laboratory-scale Process Trainer to illustrate various identification techniques and examined the sum of squared conventional residuals and Akaike s information theoretic criterion (AIC) for model structure selection. Two sets of input-output data collected from this process are available within MATLAB. We use the entire first set of data M = 1000), called DRYER2 in MATLAB, for this study. Two different model structures are examined here, namely the ARX and FIR model structures, with the objective to find the model within a pcurtic-ular structure that produces the smallest PRESS. 

Table 3.2 presents a summary of our findings. Prom this table, it can be seen that the PRESS statistic selects a model order which is close to the best order obtained using the complete data set (M = 1000) starting with M — 400. However, the FPE criterion does not select a model order close to the best order until M = 800. In addition, it is interesting to note that with the smallest data set examined (M = 200), the PRESS statistic also manages to select the best low order model ni = 3), according to our earlier analysis, but this was not the case for the FPE criterion. These results show that, for this example, the PRESS statistic provides a consistent order estimate and is more robust than the FPE criterion in terms of sensitivity to data length effects. 

The least squares estimator and the PRESS statistic are used here as a new way to estimate the best disturbance model in Equation (3.35). This is an ideal application because the objective is to choose the most parsimonious model (smallest m) while, at the same time, achieving a good approximation in Equation (3.34). An indication that a sufficient model order has been chosen is whether the residuals associated with the model... 

---