Model validation PRESS statistic

Now we come to the Standard Error of Estimate and the PRESS statistic, which show interesting behavior indeed. Compare the values of these statistics in Tables 25-IB and 25-1C. Note that the value in Table 25-1C is lower than the value in Table 25-1B. Thus, using either of these as a guide, an analyst would prefer the model of Table 25-1C to that of Table 25-1B. But we know a priori that the model in Table 25-1C is the wrong model. Therefore we come to the inescapable conclusion that in the presence of error in the X variable, the use of SEE, or even cross-validation as an indicator, is worse than useless, since it is actively misleading us as to the correct model to use to describe the data. 

Another conceptually different approach is cross-validation. In Equation (2.19), X is regarded as a model for X, and as such the model should be able to predict the values of X. This can be checked by performing a cross-validation scheme in which parts of X are left out of the calculations and kept apart, the model is built and used to predict the left out entries. The sum of squared differences between the predicted and the real entries serves as a measure of discrepancy. All data in X are left out once, and the squared differences are summed in a so called PRESS statistics (PRediction Error Sum of Squares). The model that gives the lowest PRESS is selected and the pseudo-rank of X is defined as the number of components in that model. 

The prediction performance can be validated by using a cross-validation ( leave-one-out ) method. The values for the first specimen (specimen A) are omitted from the data set and the values for the remaining specimens (B-J) are used to find the regression equation of, e.g., Cj on Ay A2, etc. Then this new equation is used to obtain a predicted value of Cj for the first specimen. This procedure is repeated, leaving each specimen out in turn. Then for each specimen the difference between the actual and predicted value is calculated. The sum of the squares of these differences is called the predicted residual error sum of squares or PRESS for short the closer the value of the PRESS statistic to zero, the better the predictive power of the model. It is particularly useful for comparing the predictive powers of different models. For the model fitted here Minitab gives the value of PRESS as 0.0274584. 

There is another way to generate the prediction errors without actually having to split the data set. The idea is to set aside each data point, estimate a model using the rest of the data, and then evaluate the prediction error at the point that was removed. This concept is well known as the PRESS statistic in the statistical community (Myers, 1990) and is used as a technique for model validation of general regression models. However, to our knowledge, the system identification literature has not suggested the use of the PRESS for model structure selection. 

In any case, the cross-validation process is repeated a number of times and the squared prediction errors are summed. This leads to a statistic [predicted residual sum of squares (PRESS), the sum of the squared errors] that varies as a function of model dimensionality. Typically a graph (PRESS plot) is used to draw conclusions. The best number of components is the one that minimises the overall prediction error (see Figure 4.16). Sometimes it is possible (depending on the software you can handle) to visualise in detail how the samples behaved in the LOOCV process and, thus, detect if some sample can be considered an outlier (see Figure 4.16a). Although Figure 4.16b is close to an ideal situation because the first minimum is very well defined, two different situations frequently occur ... 

SEP evaluation over PRESS is the time factor. To compute a single PRESS value, as many PLS models have to be formed as there are calibration samples. Conversely, only one PLS model is needed for SEP regardless of the number of spectra considered for calibration and prediction. In order to obtain a statistically valid SEP, it is crucial that the prediction set truly represents future prediction samples. 

In any case, the simulated profiles must be compared with the measured data to check the validity of the determined data as well as the assumed model. Statistical methods to quantify the goodness of the fit are given, for example, in Lapidus (1962), Barns (1994), Korns (2000) and Press et al. (2002). 

Cross-validation is one method to check the soundness of a statistical model (Cramer, Bunce and Patterson, 1988 Eriksson, Verhaar and Hermens, 1994). The data set is divided into groups, usually five to seven, and the model is recalculated without the data from each of the groups. Consecutively, predictions are obtained for the omitted compounds and compared to the actual data. The divergences are quantified by the prediction error sum of squares (PRESS sum of squares of predicted minus observed values), which can be transformed to a dimensionless term (Q ) by relating it to the initial sum of squares of the dependent variable (X(AT) )-... 

In order to use the F statistic tables properly, it is also necessary to know the degrees of freedom in both the numerator (vj and denominator (vj) of the F ratio value. For F ratios based on PRESS values, the number of samples used to calibrate the model has been suggested as the proper value for both. Therefore, in the case of a cross-validation, the degrees of freedom would be the total number of sample in the training set minus the number left out in each group. For a validation set prediction, they would be the total number of samples in the training set. 

Applying the F test to PRESS values from a self-prediction generally does not work. This is due to the fact that the F test is primarily designed to find the statistically optimum number of factors for predicting samples that were not included when the model was built. In the self-prediction scheme, every sample is already included in the model, which gives no information on the performance of the model with true unknowns. This is merely one more reason why one of the other validation methods should be used to optimize the number of factors for the model. 

The cross-validated is also termed because it does not represent a true H from a statistical standpoint. The rationale for a squared q can be associated with PRESS and SD (defined below), but because q can also take negative values, q itself has no meaning. This index, which measures the robustness of the QSAR model,is defined as ... 

---