Leave-one-out-cross-validation

This procedure is known as "leave one out" cross-validation. This is not the only way to do cross-validation. We could apply this approach by leaving out all permutations of any number of samples from the training set. The only constraint is the size of the training set, itself. Nonetheless, whenever the term cross-validation is used, it almost always refers to "leave one out" cross-validation. 

Many people use the term PRESS to refer to the result of leave-one-out cross-validation. This usage is especially common among the community of statisticians. For this reason, the terms PRESS and cross-validation are sometimes used interchangeably. However, there is nothing inate in the definition of PRESS that need restrict it to a particular set of predictions. As a result, many in the chemometrics community use the term PRESS more generally, applying it to predictions other than just those produced during cross-validation. 

When applied to QSAR studies, the activity of molecule u is calculated simply as the average activity of the K nearest neighbors of molecule u. An optimal K value is selected by the optimization through the classification of a test set of samples or by the leave-one-out cross-validation. Many variations of the kNN method have been proposed in the past, and new and fast algorithms have continued to appear in recent years. The automated variable selection kNN QSAR technique optimizes the selection of descriptors to obtain the best models [20]. 

It is usual to have the coefficient of determination, r, and the standard deviation or RMSE, reported for such QSPR models, where the latter two are essentially identical. The value indicates how well the model fits the data. Given an r value close to 1, most of the variahon in the original data is accounted for. However, even an of 1 provides no indication of the predictive properties of the model. Therefore, leave-one-out tests of the predictivity are often reported with a QSAR, where sequentially all but one descriptor are used to generate a model and the remaining one is predicted. The analogous statistical measures resulting from such leave-one-out cross-validation often are denoted as and SpR ss- Nevertheless, care must be taken even with respect to such predictivity measures, because they can be considerably misleading if clusters of similar compounds are in the dataset. 

Fig. 36.10. Prediction error (RMSPE) as a function of model complexity (number of factors) obtained from leave-one-out cross-validation using PCR (o) and PLS ( ) regression.

Figure 4.37. RMSECV PCA from leave-one-out cross-validation for PCA Example 2. 

The preprocessed data and class membership information is submitted to the analysis software. Euclidean distance and leave-one-out cross-validation is used to determine the value for K and the cutoff for G. 

FIGURE 5.102. Concentration residuals versus the predicted concentration for component A, corrected data, using leave-one-out cross-validation. 

Figure 5.107 shows the calibration values detemiined using leave-one-out cross-validation. The F.value of 1.05 is calculated as the mean of the... 

Calibration design for components A and B three-level flill factorial Calibration design for component C natural Validation design leave-one-out cross-validation Preproctjfiing none... 

The optimal number of factors and the RMSEPs resulting from leave-one-out cross-validation analyses for all four analytes are shown in Table 5-22. The number of factors used to construct the PLS models ranges from four to six. It is not unusual to derive models with different numbers of factors for different components in a data set. The extent of overlap of the spectra and chemical interactions play major roles in dictating the optimum number of factors. 

A common approach to cross-validation is called leave-one-out" cross-validation. Here one sample is left out, a PC model with given number of factors is calculated using the remaining samples, and then the residua of the sample left out is computed. This is repeated for each sample and for models with 1 to n PCs. The result is a set of cross-validation residuals for a given number of PCs. The residuals as a function of the number of PCs can be examined graphically as discussed above to determine the inherent dimensionality. In practice, the cross-validation residuals are summarized into a single number termed the Root Mean Squared Error of Cross Validation for PCA (RMSECV PCA), calculated as follows ... 

To develop a KNN model, a distance measure is selected and tlie optimal number of nearest neighbors is determined. It is recommended that K be selected using leave-one-out cross-validation applied to a training set. The outputs of the analysis arc the predicted classes for the training set and the goodness values. 

Root Mean Square Ei ror of Cross-Validation for PCA Plot (Model Diagnostic) Figure 4.63 displays the RMSECV PCA vs. number of principal components for the class B data from a leave-one-out cross-validation calculation. The RMSECy PCA quickly drops and levels off at two principal components, consistent with the choice of a rank tv- o model. 

The leave-one-out cross-validation R.MSEP plot for this example (shown in Figure 5.93) shows a clear minimum at two factors. The shape is not ideal because the RMSEP decreases again when the founh factor is added. 

The raw data are discussed in detail in Section 5.2.2.2. In the ICLS and MIR ap plications the data are split into calibration and validation sets. For this PLS analysis, the 95 spectra from the 12 design points are all used to construct the model using a leave-one-out cross-validation procedure. 

The software requires the following information the concentration and spectral data, the preprocessing selections, the maximum number of factors to estimate, and the validation approach used to choose the optimal number of factors. The maximum rank selected is 10 for constructing the model to predict the caustic concentration. The validation technique is leave-one-out cross-validation where an entire design point is left out. Tliat is, there are 12 cross validation steps and all spectra for each standard (at various temperatures) are left out of the model building phase at each step. 

Models using 1-10 factors are constructed for the prediction of MCB. Leave-one-out cross-validation is used for model validation. 

Calibration design 22-sampIe mixture design Validation design leave-one-out cross-validation Preprocessing single-point baseline correction at 1100 nm Variable range 550 measurement variables 1100-2198 nra... 

Fig. 5.2. Leave-One-Out cross-validation results reported as predicted vs. experimental plC50 values for the four kinase chemical series. In general, model prediction of pICsois in 9°°d agreement with experimental pICso derived from percent of inhibition, with a global correlation coefficient r orr cv = 0.90 for Diaminopyrimidine (a), = 0.84 for the...

Fig. 5.3. Leave-One-Out cross-validation results for the Pyrazolopyrimidine series tested in the biochemical assays PDE2 (a) and PDE10 (b).

Schedule of the leave-one-out cross-validation scheme. Any cross-validation procedure will perform in the same way although considering more samples at each validation segment. 

---