Validation set prediction

Model Training Set RMS errors Cross-validation Set Prediction Set... 

Fig. 13. A PRESS plot of a validation set prediction (solid line) of 50 samples and the cross-validation prediction (dashed line) from Fig. 10 of a training set of 50 samples of NIR diffuse reflectance spectra of wheat. The minimum in the validation set prediction is at 8...

In order to use the F statistic tables properly, it is also necessary to know the degrees of freedom in both the numerator (vj and denominator (vj) of the F ratio value. For F ratios based on PRESS values, the number of samples used to calibrate the model has been suggested as the proper value for both. Therefore, in the case of a cross-validation, the degrees of freedom would be the total number of sample in the training set minus the number left out in each group. For a validation set prediction, they would be the total number of samples in the training set. 

Figure 6.7 shows the regression plots (predicted V5. target values) for the training and validation data. In addition. Table 6.2 shows the predictions obtained for the unknowns of the validation set. Predictions are fairly good... 

A set of samples not involved in the development of the SVM model has to be predicted and its error studied. If insufficient samples are available, grid search or crossvalidation should be used on the calibration samples to select a reduced suite of candidate models and, finally, the validation set predicted to assess the best one. The objective is to avoid over-fitting, which is likely to occur if special care is not devoted to this issue. ... 

The data in the validation set are used to challenge the calibration. We treat the validation samples as if they are unknowns. We use the calibration developed with the training set to predict (or estimate) the concentrations of the components in the validation samples. We then compare these predicted concentrations to the actual concentrations as determined by an independent referee method (these are also called the expected concentrations). In this way, we can assess the expected performance of the calibration on actual unknowns. To the extent that the validation samples are a good representation of all the unknown samples we will encounter, this validation step will provide a reliable estimate of the calibration s performance on the unknowns. But if we encounter unknowns that are significantly different from the validation samples, we are likely to be surprised by the actual performance of the calibration (and such surprises are seldom pleasant). 

We will create yet another set of validation data containing samples that have an additional component that was not present in any of the calibration samples. This will allow us to observe what happens when we try to use a calibration to predict the concentrations of an unknown that contains an unexpected interferent. We will assemble 8 of these samples into a concentration matrix called C5. The concentration value for each of the components in each sample will be chosen randomly from a uniform distribution of random numbers between 0 and I. Figure 9 contains multivariate plots of the first three components of the validation sets. 

We can see, in Figure 20 that we get similar results when we use the two calibrations, Kl , and K2(al, to predict the concentrations in the validation sets. When we examine the plots for K13m and K23w die predictions for our normal validation set, A3, we see that, while the calibrations do work to a certain degree, there is a considerable amount of scatter between the expected and the predicted values. For some applications, this might be an acceptable level of performance. But, in general, we would hope to do much better. 

Figure 23 contains plots of the expected vs. predicted concentrations for all of the nonzero intercept CLS results. We can easily see that these results are much better than the results of the first calibrations. It is also apparent that when we predict the concentrations from the spectra in A5, the validation set with the... 

Now, we are ready to apply PCR to our simulated data set. For each training set absorbance matrix, A1 and A2, we will find all of the possible eigenvectors. Then, we will decide how many to keep as our basis set. Next, we will construct calibrations by using ILS in the new coordinate system defined by the basis set. Finally, we will use the calibrations to predict the concentrations for our validation sets. 

Just as we did for PCR, we must determine the optimum number of PLS factors (rank) to use for this calibration. Since we have validation samples which were held in reserve, we can examine the Predicted Residual Error Sum of Squares (PRESS) for an independent validation set as a function of the number of PLS factors used for the prediction. Figure 54 contains plots of the PRESS values we get when we use the calibrations generated with training sets A1 and A2 to predict the concentrations in the validation set A3. We plot PRESS as a function of the rank (number of factors) used for the calibration. Using our system of nomenclature, the PRESS values obtained by using the calibrations from A1 to predict A3 are named PLSPRESS13. The PRESS values obtained by using the calibrations from A2 to predict the concentrations in A3... 

As is the case for PRESS, the variance of prediction can be calculated for predictions made on independent validation sets as well as predictions made on the data set which was used to generate the calibration. 

It is often helpful to examine the regression errors for each data point in a calibration or validation set with respect to the leverage of each data point or its distance from the origin or from the centroid of the data set. In this context, errors can be considered as the difference between expected and predicted (concentration, or y-block) values for the regression, or, for PCA, PCR, or PLS, errors can instead be considered in terms of the magnitude of the spectral... 

It is usual in developing a QSPR to spUt the database into two. One part is used for training the model, while the other part is used to validate the model. This goes to the predictability of the model the model is assumed to be predictive if it can predict the solubiUty of the validation set. Since the validation set is used intimately with the training set to refine the model, it is questionable if this partitioning is warranted [50]. This partitioning is particularly questionable where the available experimental data is sparse. 

The first one we mention is the question of the validity of a test set. We all know and agree (at least, we hope that we all do) that the best way to test a calibration model, whether it is a quantitative or a qualitative model, is to have some samples in reserve, that are not included among the ones on which the calibration calculations are based, and use those samples as validation samples (sometimes called test samples or prediction samples or known samples). The question is, how can we define a proper validation set Alternatively, what criteria can we use to ascertain whether a given set of samples constitutes an adequate set for testing the calibration model at hand ... 

Cabrera et al. [50] modeled a set of 163 drugs using TOPS-MODE descriptors with a linear discriminant model to predict p-glycoprotein efflux. Model accuracy was 81% for the training set and 77.5% for a validation set of 40 molecules. A "combinatorial QSAR" approach was used by de Lima et al. [51] to test multiple model types (kNN, decision tree, binary QSAR, SVM) with multiple descriptor sets from various software packages (MolconnZ, Atom Pair, VoSurf, MOE) for the prediction of p-glycoprotein substrates for a dataset of 192 molecules. Best overall performance on a test set of 51 molecules was achieved with an SVM and AP or VolSurf descriptors (81% accuracy each). 

The methodology is very fast and completely automated. To predict the site of metabolism for drug-like substrates, the method requires few seconds per molecule. It is important to point out that the method uses neither any training set nor any statistical model or supervised technique, and it has proven to be predictive for extensively diverse validation sets preformed in different pharmaceutical companies. 

---