Leave-one-out validation

To be usefully applicable, a QSAR model must be able to learn from available data and reproduce them (goodness of fit, verified by R ), be stable and robust (verified by internal cross-validations leave-one-out, leave-many-out or bootstrap) and, most importantly, by extracting the maximum information from the limited existing knowledge, it must be able to reliably predict data for new chemicals not involved in model development (external validation). 

Fig. 9. Cross-validation of models 1-3. Left panel Production vs. the number of experiment black circles data blue triangles fitted values red pluses cross-validated leave-one-out prediction. Right panel Normal probability plots of the cross-validated leave-one-out residuals.

The prediction performance can be validated by using a cross-validation ( leave-one-out ) method. The values for the first specimen (specimen A) are omitted from the data set and the values for the remaining specimens (B-J) are used to find the regression equation of, e.g., Cj on Ay A2, etc. Then this new equation is used to obtain a predicted value of Cj for the first specimen. This procedure is repeated, leaving each specimen out in turn. Then for each specimen the difference between the actual and predicted value is calculated. The sum of the squares of these differences is called the predicted residual error sum of squares or PRESS for short the closer the value of the PRESS statistic to zero, the better the predictive power of the model. It is particularly useful for comparing the predictive powers of different models. For the model fitted here Minitab gives the value of PRESS as 0.0274584. 

The maximum number of latent variables is the smaller of the number of x values or the number of molecules. However, there is an optimum number of latent variables in the model beyond which the predictive ability of the model does not increase. A number of methods have been proposed to decide how many latent variables to use. One approach is to use a cross-validation method, which involves adding successive latent variables. Both leave-one-out and the group-based methods can be applied. As the number of latent variables increases, the cross-validated will first increase and then either reach a plateau or even decrease. Another parameter that can be used to choose the appropriate number of latent variables is the standard deviation of the error of the predictions, SpREss ... 

The second task discussed is the validation of the regression models with the aid of the cross-validation (CV) procedures. The leave-one-out (LOO) as well as the leave-many-out CV methods are used to evaluate the prognostic possibilities of QSAR. In the case of noisy and/or heterogeneous data the LM method is shown to exceed sufficiently the LS one with respect to the suitability of the regression models built. The especially noticeable distinctions between the LS and LM methods are demonstrated with the use of the LOO CV criterion. 

This procedure is known as "leave one out" cross-validation. This is not the only way to do cross-validation. We could apply this approach by leaving out all permutations of any number of samples from the training set. The only constraint is the size of the training set, itself. Nonetheless, whenever the term cross-validation is used, it almost always refers to "leave one out" cross-validation. 

Many people use the term PRESS to refer to the result of leave-one-out cross-validation. This usage is especially common among the community of statisticians. For this reason, the terms PRESS and cross-validation are sometimes used interchangeably. However, there is nothing inate in the definition of PRESS that need restrict it to a particular set of predictions. As a result, many in the chemometrics community use the term PRESS more generally, applying it to predictions other than just those produced during cross-validation. 

When applied to QSAR studies, the activity of molecule u is calculated simply as the average activity of the K nearest neighbors of molecule u. An optimal K value is selected by the optimization through the classification of a test set of samples or by the leave-one-out cross-validation. Many variations of the kNN method have been proposed in the past, and new and fast algorithms have continued to appear in recent years. The automated variable selection kNN QSAR technique optimizes the selection of descriptors to obtain the best models [20]. 

In QSAR equations, n is the number of data points, r is the correlation coefficient between observed values of the dependent and the values predicted from the equation, is the square of the correlation coefficient and represents the goodness of fit, is the cross-validated (a measure of the quality of the QSAR model), and s is the standard deviation. The cross-validated (q ) is obtained by using leave-one-out (LOO) procedure [33]. Q is the quality factor (quality ratio), where Q = r/s. Chance correlation, due to the excessive number of parameters (which increases the r and s values also), can. 

It is usual to have the coefficient of determination, r, and the standard deviation or RMSE, reported for such QSPR models, where the latter two are essentially identical. The value indicates how well the model fits the data. Given an r value close to 1, most of the variahon in the original data is accounted for. However, even an of 1 provides no indication of the predictive properties of the model. Therefore, leave-one-out tests of the predictivity are often reported with a QSAR, where sequentially all but one descriptor are used to generate a model and the remaining one is predicted. The analogous statistical measures resulting from such leave-one-out cross-validation often are denoted as and SpR ss- Nevertheless, care must be taken even with respect to such predictivity measures, because they can be considerably misleading if clusters of similar compounds are in the dataset. 

Fig. 36.10. Prediction error (RMSPE) as a function of model complexity (number of factors) obtained from leave-one-out cross-validation using PCR (o) and PLS ( ) regression.

The predictive quality of the models is judged according to the cross-validated R2, known as q2, obtained using the leave-one-out (LOO) approach, which is calculated as follows ... 

For model validation we use leave-one-out CV. Other validation schemes could be used, but in this example we have a severe limitation due to the low number of objects. A plot of the prediction errors from CV versus number of PLS components is shown in Figure 4.43. The dashed lines correspond to the MSE values for the... 

Controlling the complexity of a model is called regularization. To this end, hold-out data is important. In order to benefit from a training set that is as large as possible and still to be able to measure the performance on unseen data, cross validation is used. It does multiple iterations of training and testing on different partitionings of the data. Leave-one-out is certainly the most prominent concept here [154] however, other ways to partition are in use as well. 

Most of the QSAR-modeling methods implement the leave-one-out (LOO) (or leave-some-out) cross-validation procedure. The outcome from this procedure is a... 

---