Predictive Error Sum of Squares PRESS

Figure 6.28) and the PC-model is calculated for the reduced data-set. Because the PC model of X is the product of t and p, the model predicts the held-out elements (the element xik is predicted as ttpk). Hence, by comparing the prediction of the held-out elements with their actual values, an estimate of the predictive power of the model is obtained. The usual estimator of the predictive power in PCA and PLS is prediction error sum of squares (PRESS), defined as ... 

Prediction error sum of squares (PRESS). The sum of the squared differences between the observed and estimated response by validation techniques [Allen, 1971, 1974] ... 

Cross-validation is one method to check the soundness of a statistical model (Cramer, Bunce and Patterson, 1988 Eriksson, Verhaar and Hermens, 1994). The data set is divided into groups, usually five to seven, and the model is recalculated without the data from each of the groups. Consecutively, predictions are obtained for the omitted compounds and compared to the actual data. The divergences are quantified by the prediction error sum of squares (PRESS sum of squares of predicted minus observed values), which can be transformed to a dimensionless term (Q ) by relating it to the initial sum of squares of the dependent variable (X(AT) )-... 

A more precise method that requires more computational time is cross-validation [155, 332]. It is implemented by excluding part of the data, performing PCA on the remaining data, and computing the prediction error sum of squares (PRESS) using the data retained (excluded from model development). The process is repeated until every observation is left out once. The order a is selected as that minimizes the overall PRESS. Two additional criteria for choosing the optimal number of PCs have also been proposed by Wold [332] and Krzanowski [155], related to cross-validation. Wold [332] proposed checking the ratio. 

PLSR method not only can avoid the collinearily problem, but also can filter off a part of noise by using predicted error sum of square (PRESS) computation. But it should be emphasized that PLSR is not very effective to solve nonlinear problems, since the addition of nonlinear terms can be only tried by trial and error method. 

Fortunately, since we also have concentration values for our samples, We have another way of deciding how many factors to keep. We can create calibrations with different numbers of basis vectors and evaluate which of these calibrations provides the best predictions of the concentrations in independent unknown samples. Recall that we do this by examing the Predicted Residual Error Sum-of Squares (PRESS) for the predicted concentrations of validation samples. 

Just as we did for PCR, we must determine the optimum number of PLS factors (rank) to use for this calibration. Since we have validation samples which were held in reserve, we can examine the Predicted Residual Error Sum of Squares (PRESS) for an independent validation set as a function of the number of PLS factors used for the prediction. Figure 54 contains plots of the PRESS values we get when we use the calibrations generated with training sets A1 and A2 to predict the concentrations in the validation set A3. We plot PRESS as a function of the rank (number of factors) used for the calibration. Using our system of nomenclature, the PRESS values obtained by using the calibrations from A1 to predict A3 are named PLSPRESS13. The PRESS values obtained by using the calibrations from A2 to predict the concentrations in A3... 

The Predicted Residual Error Sum of Squares (PRESS) is simply the sum of the squares of all the errors of all of the samples in a sample set. 

MSE is preferably used during the development and optimization of models but is less useful for practical applications because it has not the units of the predicted property. A similar widely used measure is predicted residual error sum of squares (PRESS), the sum of the squared errors it is often applied in CV. 

Several criteria can be used to select the best models, such as the F-test on regression, the adjusted correlation coefficient (R ad) and the PRESS [20] (Predictive error sum of squares). In general, even only adequate models show significant F values for regression, which means that the hypothesis that the independent variables have no influence on the dependent variables may not be accepted. The F value is less practical for further selection of the best model terms since it hardly makes any distinction between different predictive models. 

In any case, the cross-validation process is repeated a number of times and the squared prediction errors are summed. This leads to a statistic [predicted residual sum of squares (PRESS), the sum of the squared errors] that varies as a function of model dimensionality. Typically a graph (PRESS plot) is used to draw conclusions. The best number of components is the one that minimises the overall prediction error (see Figure 4.16). Sometimes it is possible (depending on the software you can handle) to visualise in detail how the samples behaved in the LOOCV process and, thus, detect if some sample can be considered an outlier (see Figure 4.16a). Although Figure 4.16b is close to an ideal situation because the first minimum is very well defined, two different situations frequently occur ... 

Sometimes the question arises whether it is possible to find an optimum regression model by a feature selection procedure. The usual way is to select the model which gives the minimum predictive residual error sum of squares, PRESS (see Section 5.7.2) from a series of calibration sets. Commonly these series are created by so-called cross-validation procedures applied to one and the same set of calibration experiments. In the same way PRESS may be calculated for a different sets of features, which enables one to find the optimum set . 

The PLS model is calculated without these values. The omitted values are predicted and then compared with the original values. This procedure is repeated until all values have been omitted once. Therefore an error of prediction, in terms of its dependence on the number of latent variables, is determined. The predicted residual error sum of squares (PRESS) is also the parameter which limits the number of latent vectors u and t ... 

D is the matrix of deviations between true and calculated. Since B is dependent on the rank A, also D will be dependent on it. A useful expression is the prediction residual error sum of squares (PRESS) ... 

For PLS solution basis sets, bulk spectra were generated as described above. Standard error of calibration values (SECV) were determined from prediction residual sum of squares (PRESS) analyses of various permutations of the amide I, II, and III bands (always including amide I) from both Ge and ZnSe spectra. After determination of the effects of different types of normalization on the results, these bands were individually normalized to an area of 100 absorbance units before PLS 1 training. 

The next step, is to leave out part of the data from the residual matrix and compute the next component from the truncated data matrix. The left-out data can then be predicted from the expanded model and the error of prediction fir - ir ir s determined. This is repeated over and over again, until all elements in E have been left out once and only once. The prediction enor sum of squares, PRESS = 22yij, is computed from the estimated errors of prediction. If it should be found that PRESS exceeds the residual sum of squares RSS calculated firom the smaller model, the new component does not improve the prediction and is considered to be insignificant. A ratio PRESS/RSS > 1 implies that the new component predicts more noise than it explains the variation. 

This procedure is then calculated [100/(y%)] times, ensuring that a given data grouping is only deleted once. The predicted residual error sum of squares (PRESS) is then... 

Root mean square error in prediction (RMSEP) (or root mean square deviation in prediction, RMSDP). Also known as standard error in prediction (SEP) or standard deviation error in prediction SDEP), is a function of the prediction residual sum of squares PRESS, defined as... 

For each reduced data set, the model is calculated and responses for the deleted objects are predicted from the model. The squared differences between the true response and the predicted response for each object left out are added to PRESS ( prediction error sum of squares). From the final PRESS, the (or R ) and RMSEP ( root mean square error in prediction) values are usually calculated [Cruciani, Baroni et al, 1992]. 

By this validation technique, the original size of the data set (n) is preserved for the training set, by the selection of n objects with repetition in this way, the training set contains some repeated objects and the evaluation set is constituted by the objects left out [Efron, 1982,1987 Wehrens, Putter et al., 2000]. The model is calculated on the training set and responses are predicted on the evaluation set. All the squared differences between the true response and the predicted response of the objects of the evaluation set are collected in PRESS ( prediction error sum of squares). This procedure of building training sets and evaluation sets is repeated thousands of times, PRESS are summed up and the average predictive power is calculated. 

Another conceptually different approach is cross-validation. In Equation (2.19), X is regarded as a model for X, and as such the model should be able to predict the values of X. This can be checked by performing a cross-validation scheme in which parts of X are left out of the calculations and kept apart, the model is built and used to predict the left out entries. The sum of squared differences between the predicted and the real entries serves as a measure of discrepancy. All data in X are left out once, and the squared differences are summed in a so called PRESS statistics (PRediction Error Sum of Squares). The model that gives the lowest PRESS is selected and the pseudo-rank of X is defined as the number of components in that model. 

Error types can be e.g. root mean square error of cross validation (RMSECV), root mean square error of prediction (RMSEP) or predictive residual sum of squares (PRESS). 

PRESS is predicted error sum of squares deviations between the observed and measured y values... 

The selection of an optimal number of factors (loadings) is a central point in PCR and PLS. In both methods the so-called prediction residual error sum of squares (PRESS) is calculated... 

---