Cross validations

Validation results obtained from factor analysis of Table 31.2, containing the retention times of 23 chalcones in 8 chromatographic methods, after log double-centering and global normalization. The results are used in the Malinowski s f-test and in cross-validation by PRESS. 

the number of structural eigenvectors is the largest r for which Malinowski s F-ratio is still significant at a predefined level of probability a (say 0.05)  

In Table 31.9 we represent the results of Malinowski s F-test as computed from the eigenvalues of the transformed retention times in Table 31.2. The second eigenvector produces an F-value which still exceeds the critical F-statistic (6.6 with 1 and 5 degrees of freedom) at the 0.05 level of probability. Hence, from this evidence we conclude again that there are two, possibly three, structural eigenvectors in this data set. 

The method of cross-validation is based on internal validation, which means that one predicts each element in the data set from the results of an analysis of the remaining ones. This can be done by leaving out each element in turn, which in the case of an nxp table would require nxp analyses. Wold [44] has implemented a scheme for leaving out groups of elements at the same time, which reduces the 

As more structural eigenvectors are included we expect PRESS to decrease up to a point when the structural information is exhausted. From this point on we expect PRESS to increase again as increasingly more error eigenvectors are included. In order to determine the transition point r one can compare PRESS(r -i-l) with the previously obtained PRESS(r ). The number of structural eigenvectors r is reached when the ratio  

In order to appropriately evaluate the performance of a gene set from each method, cross-vahdation (CV) is widely used. CV utilizes resampling without replacement of the entire data to repeatedly develop classifiers on a training set and evaluate classifiers on a separate test set and then averages the procedure over the resampling. 

If data are large, a subset of the data can be set aside for validation of the model to assess the performance of the classification algorithms. However, in general, a dataset is often insufficient to satisfy the need, and it is usually not possible to obtain such a dataset. A popular method to remedy this difficulty is CV. 

In V-fold CV, the data are randomly divided into V roughly equal subsets (see Table 6.2). In each CV mn, a set of genes is selected by each method fi om a training set of data (V — 1 subsets together). Using a set of selected genes, each classifier is built and a separate test set of data (the remaining subset) is applied to the model. This process is repeated V times and the V estimates are combined to evaluate the performance of the model. This completes one trial of the V-fold CV. 

With V = N (the total number of samples), commonly referred to as leave-one-out cross-validation (LOOCV), each CV run uses N I samples for training and the remaining one sample for testing. Since the N training samples are so similar in each run, CV is approximately unbiased for the tme prediction error. However, the variance of the true prediction error would be large and it is also computationally expensive. In practice, the preference of the number of folds depends on the size of the dataset. Tenfold CV has become generally accepted. 

Another vahdation approach is the holdout vahdation. Holdout validation is not considered a CV because the data are not crossed over. The data are randomly divided into a training set and a test set. Typically less than one-third of the data is chosen as the test set. 

The quality of a theoretical semivariogram model can be assessed by means of point kriging and cross-validation [MYERS, 1991]. Each point of the data set is deleted one after another and then newly estimated according to Eqs. 4-33 to 4-35 by means of its neighbors. Additionally, the kriging variance is calculated for each estimated point (Eq. 4-36). 

These values are squared and divided by the corresponding kriging variance  

The mean of these quotients Az2rei should be situated near zero and their standard deviation near unity, i.e. it should be a standardized normal distribution if the kriging estimation is distortion-free. 

The goodness of fit for experimental data to a theoretical semivariogram model can be tested by means of cross-validation. The best model is that with the smallest deviation from the mean zero and with the smallest standard deviation. 

Often the number of samples for calibration is limited and it is not possible to split the data into a calibration set and a validation set containing representative samples that are representative enough for calibration and for validation. As we want a satisfactory model that predicts future samples well, we should include as many different samples in the calibration set as possible. This leads us to the severe problem that we do not have samples for the validation set. Such a problem could be solved if we were able to perform calibration with the whole set of samples and validation as well (without predicting the same samples that we have used to calculate the model). There are different options but, roughly speaking, most of them can be classified under the generic term cross-validation . More advanced discussions can be found elsewhere [31-33]. 

Schedule of the leave-one-out cross-validation scheme. Any cross-validation procedure will perform in the same way although considering more samples at each validation segment. 

A plateau is obtained instead of a minimum. In this case, we should be careful and spend time testing the model (e.g. different cross-validation schemes) and, more importantly, validating it with new samples (parsimony is key here). 

Neither a minimum nor a plateau is seen, but a random behaviour. This suggests problems with either the model or the data (too different samples, etcl [22]. 

There are many other options to select the dimensionality of PLS models. Unfortunately, we should not expect them to yield the same results and, therefore, some consensus should be reached. The problem can be more difficult whenever calibration sets cannot be deployed according to an experimental design (e.g. in industrial applications see [38] for a worked example). 

A significant component should describe systematic variation above the noise level. The principal components model can be used to predict the values of the descriptors. Since a model can never tell the whole story, there will always be a deviation between the observed and the predicted values. We can then determine 

Assume that a principal components model with A components (/ j, / 2 — Pa) has been determined. The model can be used in two ways (1) For a new compound r the corresponding score values can be determined by projecting the descriptors of r down to the hyperplane spanned by the components. (2) It is then possible to predict the original descriptors of r from the scores and the loading vectors. If the model is good, the predicted value, ijj, of a descriptor should be close to the observed value, X ,.. The difference, Xj, - is the prediction error. (The letter/will be used to 

The score values of r are computed by multiplication of the loading matrix P by the centred and scaled descriptors of r . 

The predicted values of the descriptors are obtained by multiplication of the transposed loading matrix P by the scores. 

The NIPALS algorithm can tolerate missing data. It is therefore possible to compute a principal components model if data are left out from the data matrix during the modelling process. This can be used to determine whether or not a new component is significant by examining how well the expanded model with the new component can predict left-out data, as compared to the model without the new component. If the new component does not improve the predictions, it is considered not to be significant. The cross validation procedure can be summarized as follows  

---

An observation of the results of cross-validation revealed that all but one of the compounds in the dataset had been modeled pretty well. The last (31st) compound behaved weirdly. When we looked at its chemical structure, we saw that it was the only compound in the dataset which contained a fluorine atom. What would happen if we removed the compound from the dataset The quahty ofleaming became essentially improved. It is sufficient to say that the cross-vahdation coefficient in-CTeased from 0.82 to 0.92, while the error decreased from 0.65 to 0.44. Another learning method, the Kohonen s Self-Organizing Map, also failed to classify this 31st compound correctly. Hence, we had to conclude that the compound containing a fluorine atom was an obvious outlier of the dataset. 

Another method of detection of overfitting/overtraining is cross-validation. Here, test sets are compiled at run-time, i.e., some predefined number, n, of the compounds is removed, the rest are used to build a model, and the objects that have been removed serve as a test set. Usually, the procedure is repeated several times. The number of iterations, m, is also predefined. The most popular values set for n and m are, respectively, 1 and N, where N is the number of the objects in the primary dataset. This is called one-leave-out cross-validation. 

Oui recommendation is that one should use n-leave-out cross-validation, rather than one-leave-out. Nevertheless, there is a possibility that test sets derived thus would be incompatible with the training sets with respect to information content, i.e., the test sets could well be outside the modeling space [8]. 

A crucial decision in PLS is the choice of the number of principal components used for the regression. A good approach to solve this problem is the application of cross-validation (see Section 4.4). 

The predictive power of the CPG neural network was tested with Icavc-one-out cross-validation. The overall percentage of correct classifications was low, with only 33% correct classifications, so it is clear that there are some major problems regarding the predictive power of this model. First of all one has to remember that the data set is extremely small with only 11 5 compounds, and has a extremely high number of classes with nine different MOAs into which compounds have to be classified. The second task is to compare the cross-validated classifications of each MOA with the impression we already had from looking at the output layers. 

The maximum number of latent variables is the smaller of the number of x values or the number of molecules. However, there is an optimum number of latent variables in the model beyond which the predictive ability of the model does not increase. A number of methods have been proposed to decide how many latent variables to use. One approach is to use a cross-validation method, which involves adding successive latent variables. Both leave-one-out and the group-based methods can be applied. As the number of latent variables increases, the cross-validated will first increase and then either reach a plateau or even decrease. Another parameter that can be used to choose the appropriate number of latent variables is the standard deviation of the error of the predictions, SpREss ... 

The second task discussed is the validation of the regression models with the aid of the cross-validation (CV) procedures. The leave-one-out (LOO) as well as the leave-many-out CV methods are used to evaluate the prognostic possibilities of QSAR. In the case of noisy and/or heterogeneous data the LM method is shown to exceed sufficiently the LS one with respect to the suitability of the regression models built. The especially noticeable distinctions between the LS and LM methods are demonstrated with the use of the LOO CV criterion. 

The most serious problem with ensemble average approaches is that they introduce many more parameters into the calculation, making the parameter-to-observable ratio worse. The effective number of parameters has to be restrained. This can be achieved by using only a few confonners in the ensemble and by determining the optimum number of confonners by cross-validation [83]. A more indirect way of restraining the effective number of parameters is to restrict the conformational space that the molecule can search... 

A similar problem arises with present cross-validated measures of fit [92], because they also are applied to the final clean list of restraints. Residual dipolar couplings offer an entirely different and, owing to their long-range nature, very powerful way of validating structures against experimental data [93]. Similar to cross-validation, a set of residual dipolar couplings can be excluded from the refinement, and the deviations from this set are evaluated in the refined structures. 

SJ Cho, A Tropsha. Cross-validated R2-guided region selection for comparative molecular held analysis A simple method to achieve consistent results. J Med Chem 38 1060-1066, 1995. 

During the selection of the number of hidden layer neurons, the desired tolerance should also be considered. In general, a tight tolerance requires that the selected network be trained with fewer hidden neurons. As mentioned earlier, cross-validation during training can be used to monitor the error progression, which subsequently serves as a guideline in the selection of the hidden layer neurons. 

Sometimes it is just not feasible to assemble any validation samples. In such cases there are still other tests, such as cross-validation, which can help us do a certain amount of validation of a calibration. However, these tests do not provide the level of information nor the level of confidence that we should have before placing a calibration into service. More about this later. 

Indicator functions have the advantage that they can be used on data sets for which no concentration values (y-data) are available. But cross-validation and, especially PRESS, can often provide more reliable guidance. 

Cross-validation. We don t always have a sufficient set of independent validation samples with which to calculate PRESS. In such instances, we can use the original training set to simulate a validation set. This approach is called cross-validation. The most commom form of cross-validation is performed as follows ... 

This procedure is known as "leave one out" cross-validation. This is not the only way to do cross-validation. We could apply this approach by leaving out all permutations of any number of samples from the training set. The only constraint is the size of the training set, itself. Nonetheless, whenever the term cross-validation is used, it almost always refers to "leave one out" cross-validation. 

If we did not have a validation set available to us, we could use cross-validation for the same purposes. Figure 55 contains plots of the results of cross validation of the two training sets, A1 and A2. Since no separate validation data set is involved, we name the results PCRCROSS1 and PCRCROSS2, respectively. 

Figure 55. Logarithmic plots of the cross-validation results as a function of the number of factors (rank) used to construct the calibration.

So, cross-validation and PRESS both indicate that we should use 5 factors for our calibrations. This indication is sufficiently consistent with the F-test on the REV" and with our "eyeball" inspection of the EV s and REV s, themselves. It can also be worthwhile to look at the eigenvectors themselves. 

Many people use the term PRESS to refer to the result of leave-one-out cross-validation. This usage is especially common among the community of statisticians. For this reason, the terms PRESS and cross-validation are sometimes used interchangeably. However, there is nothing inate in the definition of PRESS that need restrict it to a particular set of predictions. As a result, many in the chemometrics community use the term PRESS more generally, applying it to predictions other than just those produced during cross-validation. 

In this book, the term PRESS is used only for the case where the calibration was generated with one data set and the predictions were made on an independent data. The term CROSS is used to denote the PRESS computed during cross-validation. This was done to in an attempt to distinguish cross-validation from other means of validation. 

Cross-validation of PCR calibration from Al/Cl for A3 Cross-validation of PCR calibration from A2/C2 for A3... 

Correlation coefficient, 60 Cross-validation, 106 Cynicism, 5, 23 Data... 

Cross-Validation methods make use of the fact that it is possible to estimate missing measurements when the solution of an inverse problem is obtained. 

---