Cross validation error

When the GA algorithm is terminated, one is presented with one or more variable subsets, along with the cross-validation error associated with each subset. At this point, there are several ways in which these results could be used to select the final subset of variables. One could select the union or the intersection of the variables that appear in all of these subsets, or simply the subset that generated the lowest cross-vahdation error. One could also use prior knowledge regarding the stability or rehabihty of certain variables to make the final selection. However, it should be noted that the GA algorithm starts with a randomly-selected set of variable subsets, and thus will generate different results with the identical x and y data when run more than once. As a result, it is often useful to run the GA several times on the same data, in order to obtain a consensus on the selection of useful variables. 

In order to characterise wine samples into the mentioned four classes, a supervised pattern recognition method (LDA) was applied. The results obtained gave 100% correct classification for the three classes (Barbera Oltrepo, Barbera Piemonte and Barbera Alba) and only one Barbera Asti sample was not correctly classified (cross-validation error rate 1.89%). 

For acenaphthylene using PLS1, the cross-validated error is presented in Fig. 18. An immediate difference between autoprediction and cross-validation is evident. In the former case the data will always be better modelled as more components are employed in the calculation, so the error will always reduce (with occasional rare exceptions in the case of 1 Hcai). Flowever, cross-validated errors normally reach a minimum as the correct number of components are found and then increase afterwards. This is because later components really represent noise and not systematic information in the data. 

If, however, we use dataset B as the training set and dataset A as the test set, a very different story emerges as shown in Fig. 20 for acenaphthylene. The autopredictive and cross-validation errors are very similar to those obtained for dataset A the value... 

Confusion of model selection with model assessment. If one chooses the model with the lowest cross-validated error among competing models, that error is not a valid estimate of the prediction error of that model (selection bias). 

Table 4.8 Calculation of cross-validated error for sample 1 ...

The following steps are used to determine cross-validation errors for PCR. 

Decide how many PCs are in the model, which determines the size of the matrices. Normally the procedure is repeated using successively more PCs, and a cross-validation error is obtained each time. 

Perform PLS1 cross-validation on the c values for the first eight components and plot a graph of cross-validated error against component number. 

For the GA applications described in this chapter the fitness function is set to be the cross-validation error from PLS on a data set using the selected variables dictated by the binary string (i.e. the hypothesis is identical to the set of selected variables). 

However, the instability of tree-based methods implies also here a much higher error in case of cross-validation by the simple leave-one-out method, that is, the cross-validated fraction of misclassified objects for CART is with 2.25%, 10 times higher than the resubstitution error. This error can be only reduced if ensemble methods are included in the model budding step. A bagged CART model revealed a cross-validation error of only 1.0% (Figure 5.38e). The fraction of misclassifica-tions for the cross-validated models increases for QDA, SVM, and A-NN to 5.5%, 5.0%, and 4.75%, respectively. The cross-validated classifications by LDA reveal 58.8% of misclassified objects as expected from the type of data. 

Fig. 13.1 Plot of the mean absolute cross-validation error (calculated for compounds outside AD) vs coverage for the thrombin dataset...

Figure 8.3. Mean absolute errors and mean squared errors of approximations computed for materials from the seventh generation of the genetic algorithm by MLPs with one-hidden-layer architectures fulfilling 7 S bh S 14, trained with all the data considered during the architecture search. For comparison, average cross-validation error values of the involved architectures are recalled from Figure 8.1.

We have used both these methods to check the impression given by the above figures. The results are listed in Table 8.2. They clearly confirm that the order of errors of approximations computed with the trained multilayer perceptions for catalytic materials from the seventh generation and the order of the mean cross-validation errors of their architectures correlate. That correlation is substantially more significant if the errors of approximations are measured in the same way as during the architecture search, i.e., using MSB. 

Table 8.2. Results of quantitatively checking whether the order of errors of approximations computed by the trained mnltilayer perceptrons for catal3dic materials from the seventh generation of the genetic algorithm correlates with the order of the mean cross-validation errors of their architectures.

The model was built with one LVs (Y explained variance in cross validation, LOO, 84.38%). Analogously to the unfold-PLS case, the robustness and the predictive capability of the model were tested by leaving one producer out (RMSEP-LOP) procedure. In all the six cases, the models built with one LV gave the best results in terms of lowest root mean squares cross-validation error. Considering the RMSEP-LOP values for each sensory parameter for the different models (data not reported), they have the same trend of the respective unfolding analysis with numerical values smaller than the previous one. 

---