Cross-validated residuals

Root Mean Square Error of Cross Validation for PCA Plot (Model Diagnostic) As described above, the residuals from a standard PCA calculation indicate how the PCA model fits the samples that were used to construction the PCA model. Specifically, they are the portion of the sample vectors that is not described by the model. Cross-validation residuals are computed in a different manner, A subset of samples is removed from the data set and a PCA model is constructed. Then the residuals for the left out samples are calculated (cross-validation residuals). The subset of samples is returned to the data set and the process is repeated for different subsets of samples until each sample has been excluded from the data set one time. These cross-validation residuals are the portion of the left out sample vectors that is not described by the PCA model constructed from an independent sample set. In this sense they are like prediction residuals (vs. fit). 

One advantage of the cross-validation residuals is that they are more sensitive to outliers. Because the left out samples do not influence the construaion of the PCA models, unusual samples will have inflated residuals. The cross-validation PCA models are also less prone to modeling noise in the data and therefore the resulting residuals better reflect the inherent noise in the data set. The identification and removal of outliers and better estimation of noise can provide a more realistic estimate of the inherent dimensionaliw of a data set. 

A common approach to cross-validation is called leave-one-out" cross-validation. Here one sample is left out, a PC model with given number of factors is calculated using the remaining samples, and then the residua of the sample left out is computed. This is repeated for each sample and for models with 1 to n PCs. The result is a set of cross-validation residuals for a given number of PCs. The residuals as a function of the number of PCs can be examined graphically as discussed above to determine the inherent dimensionality. In practice, the cross-validation residuals are summarized into a single number termed the Root Mean Squared Error of Cross Validation for PCA (RMSECV PCA), calculated as follows ... 

Figure 2. Diagnostics for the Wonderland approximating model (a) actual human development index (HDI) values versus their cross-validation predictions (b) standardized cross-validation residuals versus cross-validation predictions.

The two regression models have lower prediction accuracy than the random-function model when assessed using cross-validated root mean squared error of prediction . This quantity is simply the root mean of the squared cross-validated residuals y( (,)) - for i = 1,..., n in the numerator of (20). The cross-... 

Figure 7.5. The scree plot shows percentage of the total sum of squares explained by the PARAFAC model with increasing number of components. Results from fit and (expectation maximization) cross-validated residuals are shown.

For an analysis of the AD of regression models, the author has always used the Williams plot, which is now widely applied by other authors and commercial software. The Williams plot is the plot of standardized cross-validated residuals (R) versus leverage (Hat diagonal) values (h from the HAT matrix). It allows an immediate and simple graphical detection of both the response outliers i.e., compounds with cross-validated standardized residuals greater than 2-3 standard deviation units) and structurally anomalous chemicals in a model (h>h, the critical value being h = 3p /n, where p is the number of model variables plus one, and n is the number of the objects used to calculate the model).40,62,66... 

A similar problem arises with present cross-validated measures of fit [92], because they also are applied to the final clean list of restraints. Residual dipolar couplings offer an entirely different and, owing to their long-range nature, very powerful way of validating structures against experimental data [93]. Similar to cross-validation, a set of residual dipolar couplings can be excluded from the refinement, and the deviations from this set are evaluated in the refined structures. 

In a more recent variant of cross-validation one replaces PRESS(r- ) in the denominator by the residual sum of squares RSS(r ) ... 

If care is not taken about the way j is obtained, SIMCA has a tendency to exclude more objects from the training class than necessary. The 5-value should be determined by cross-validation. Each object in the training set is then predicted, using the A- -dimensional PCA model obtained, for the other (n - 1) training set objects. The (residual) scores obtained in this way for each object are used in eq. (33.14) [30]. 

To construct the reference model, the interpretation system required routine process data collected over a period of several months. Cross-validation was applied to detect and remove outliers. Only data corresponding to normal process operations (that is, when top-grade product is made) were used in the model development. As stated earlier, the system ultimately involved two analysis approaches, both reduced-order models that capture dominant directions of variability in the data. A PLS analysis using two loadings explained about 60% of the variance in the measurements. A subsequent PCA analysis on the residuals showed that five principal components explain 90% of the residual variability. 

The residuals can be calculated from a given set of calibration samples in a different way. Cross validation is an important procedure to estimate a realistic prediction error like PRESS. The data for k samples are removed from the data matrix and then predicted by the model. The residual errors of prediction of cross-validation in this case are given by... 

Infrared data in the 1575-400 cm region (1218 points/spec-trum) from LTAs from 50 coals (large data set) were used as input data to both PLS and PCR routines. This is the same spe- tral region used in the classical least-squares analysis of the small data set. Calibrations were developed for the eight ASTM ash fusion temperatures and the four major ash elements as oxides (determined by ICP-AES). The program uses PLSl models, in which only one variable at a time is modeled. Cross-validation was used to select the optimum number of factors in the model. In this technique, a subset of the data (in this case five spectra) is omitted from the calibration, but predictions are made for it. The sum-of-squares residuals are computed from those samples left out. A new subset is then omitted, the first set is included in the new calibration, and additional residual errors are tallied. This process is repeated until predictions have been made and the errors summed for all 50 samples (in this case, 10 calibrations are made). This entire set of... 

In using cross-validation it is essential to avoid, or at least minimize, bias to the free R factor itself. In the era of emerging automated procedures for modelling and refinement a frequent mistake is to set aside a fraction of reflections for minimization of the residual in reciprocal space and, at the same time, to use all data for computation of electron density and model rebuilding. Since local adjustment of the model in real space is equivalent to global phase adjustment in reciprocal space, the free reflection set becomes biased towards the current model and loses its validation credibility. 

The main parameters to define include the choice of preprocessing and, in some cases, the total number of PCs to estimate. The PCA outputs include the percent variance explained by each PC, cross validation results, scores, loadings, and residuals. 

FIGURE 5.S5. Concentration residual (known-predicted) versus predicted concentration for cotnponenl A using leave-one-cur cross-validation. 

FIGURE 5.102. Concentration residuals versus the predicted concentration for component A, corrected data, using leave-one-out cross-validation. 

FIGURE 5.115. Concentration residuals versus predicted concentrations for caustic (six PLS factors) using cross-validation. 

FIGURE 5.122, Cross-validation spectra residuals for the caustic PLS calibration. These residuals should be compared to the preprocessed calibration spectra, not the raw spectra. 

In any case, the cross-validation process is repeated a number of times and the squared prediction errors are summed. This leads to a statistic [predicted residual sum of squares (PRESS), the sum of the squared errors] that varies as a function of model dimensionality. Typically a graph (PRESS plot) is used to draw conclusions. The best number of components is the one that minimises the overall prediction error (see Figure 4.16). Sometimes it is possible (depending on the software you can handle) to visualise in detail how the samples behaved in the LOOCV process and, thus, detect if some sample can be considered an outlier (see Figure 4.16a). Although Figure 4.16b is close to an ideal situation because the first minimum is very well defined, two different situations frequently occur ... 

A simple and classical method is Wold s criterion [39], which resembles the well-known F-test, defined as the ratio between two successive values of PRESS (obtained by cross-validation). The optimum dimensionality is set as the number of factors for which the ratio does not exceeds unity (at that moment the residual error for a model containing A components becomes larger than that for a model with only A - 1 components). The adjusted Wold s criterion limits the upper ratio to 0.90 or 0.95 [35]. Figure 4.17 depicts how this criterion behaves when applied to the calibration data set of the working example developed to determine Sb in natural waters. This plot shows that the third pair (formed by the third and fourth factors) yields a PRESS ratio that is slightly lower than one, so probably the best number of factors to be included in the model would be three or four. 

PCs are ranked according to the fraction of variance of the dataset that they explain. The first PC is the most important (it explains the largest fraction of variance), and so forth. Selecting the correct number of PCs is crucial. Too few PCs will leave important information out of the model, but too many PCs will include noise, and decrease the model s robustness (if R J, the PCA is pointless). Each time you make a new PCA model, you should examine the residuals matrix E. If the residuals are structured, it means that some information is left out. You can also decide on the correct number of PCs by performing a cross-validation (see below), or by examining the percentage of the variance explained by the model. 

Sometimes the question arises whether it is possible to find an optimum regression model by a feature selection procedure. The usual way is to select the model which gives the minimum predictive residual error sum of squares, PRESS (see Section 5.7.2) from a series of calibration sets. Commonly these series are created by so-called cross-validation procedures applied to one and the same set of calibration experiments. In the same way PRESS may be calculated for a different sets of features, which enables one to find the optimum set . 

The fourth recommendation of the OECD experts is related to appropriate measuring and reporting goodness-of-fit, robustness, and predictivity of the model. The main intention was to clearly distinguish, whether a measure was derived only from the training set, from the internal validation (i.e., cross-validation, where the same chemicals are used for training and validation, but not at the same time) or from validation with use of an external set of compounds, not previously engaged in model optimization and/or calibration (external validation). A widely applied measure of fit is the squared correlation coefficient R2-1 — (RSS/ TSS), where RSS is the residual sum of squares and TSS is the total sum of squares... 

---