Test-set and cross-validation

The first treated is test-set and cross-validation which are general tools useful for many purposes in validation. For example, cross-validation is often used in two-way analysis for establishing the number of components in PCA or PLS models. In essence, test-set and cross-validation simply provide methods for obtaining more realistic residuals than those obtained by ordinary residuals from fitted models. Thus, test-set and cross-validation are of general interest when residuals are used for reporting explained variance, assessing outliers etc. 

Test-set validation is performed by fitting the current model to new data. How this is done is explained in Chapter 6 for the different models. The residuals are simply obtained by subtracting the model of the new data from the actual data. Test-set validation is natural because it specifically simulates the practical use of the model on new data in the future. If the test-set is made independent of the calibration data (future samples measured by future technicians or in a future location etc.) then the test-set validation is the definitive validation because the new data are, in fact, made as a drawing from the future total population of all possible measurements. Small test-sets may provide uncertain results merely because of the small sample size. 

Multi-way Analysis With Applications in the Chemical Sciences 

In component models, one practical problem associated with the test-set validation method is that the residuals obtained are not independent of the data used for fitting the model. Consider a two-way PCA model as a simple example of a component model. If the loadings P (/ x R) are found from a calibration set and a PCA model is to be tested on either a test-set or on left-out samples during cross-validation, then the residuals are found by first calculating the scores for the new data Xtest as 

As can be clearly seen, the model values (Xt PP7) are not independent of the data (Xtest) and thus overfitting can be a problem even for a test set. Thus, to compare different models on the basis of residual sum of squares calculated in this way, some correction for degrees of freedom is necessary [Martens Naes 1989], Such degrees of freedom are not easily defined, especially for multi-way models and hence can, at most, be considered as reasonable approximations. For example, the approximate degrees of freedom can be set to the number of elements minus the number of parameters or minus the number of parameters corrected for scaling and rotational indeterminacy (Chapter 5 or [Bro 1998, Durell et al. 1990, Liu Sidiropoulos 2001, Louwerse et al. 1999]). 

---

Validation or Test Set A set of samples used to validate a prediction or classification model. These samples are not part of the calibration set that is used to construct the model. (See also Calibration or Training Set and Cross-Validation.)... 

In a multivariate calibration, where a set of NIR spectra (Xnxk, N samples and K variables) is regressed onto a continuous variable (yivxi) such as the fat or moisture content, the statistical errors, the accuracy, are most often used as a quality measure of the calibration. The absolutely most common quality measure of a multivariate calibration is the prediction error, expressed either as root mean square error of prediction (RMSEP) or standard error of performance (SEP). Both are calculated and are the result of a validation process, such as test set or cross-validation. These prediction errors are defined as ... 

The method has currently 22 parameters, about 8 of which need to be optimized for each database and analyte. This is done by trial and error using a test set or cross-validation. With the current software, this involves quite a bit of work. A recent paper [19] describes experiences with three large databases and suggests a strategy for the optimization. Once the optinoization has been done, the program should be used for prediction without any alteration of critical parameters. 

When not enough examples are available to make an independent monitoring set, the cross-validation procedure can be applied (see Chapter 10). The data set is split into C different parts and each part is used once as monitoring set. The network is trained and tested C times. The results of the C test sessions give an indication on the performance of the network. It is strongly advised to validate the network that has been trained by the above procedure with a second independent test set (see Section 44.5.10). 

On the other hand, the prediction by content approach is applicable regardless of the variety of sorting pathways. It may be applied to partial sequences, which are now massively produced day by day. In addition, this approach allows a simple and unified treatment, which is convenient for objective testing (e.g., cross validation). However, there is no guarantee that the amino acid composition of proteins in each localization site is well conserved. Even when a clear tendency is observed for a known set of proteins, it can be an artifact resulting from the deviation of data because the size of known proteins for each site is often insufficient to perform reliable statistical analyses. It is also evident that this approach cannot handle the differences among isoforms with different localization (see Section III,K,3). 

The essential characteristic of a proper test set is that it represents a new drawing from the population , realized as a new, independent [X,Y] data set specifically not used in the modeling. It is evident that any A -object data set constitute but only one specific realization of an iV-tuple of individual TSE materializations. It takes a completely new ensemble of objects, the test set, to secure a second manifestation. All new measurements, for example when a PAT model is used for the purpose of automated prediction, constitute precisely such a new drawing/sampling. All new measurement situations are therefore to be likened to a test set - and this is exactly what is missing in all forms of cross-validation. 

Independent (external) test-set validation and cross-validation are the most current methods of estimating prediction error. External test-set validation is based on testing the model on a subset of available samples, which will not be involved in... 

We should, however, remark that in reality the data analyst will look for more samples, will probably try several cross-validation procedures, will test the classification functions using independent test sets, and so on. 

If, however, we use dataset B as the training set and dataset A as the test set, a very different story emerges as shown in Fig. 20 for acenaphthylene. The autopredictive and cross-validation errors are very similar to those obtained for dataset A the value... 

Comparative analysis of the performance of various algorithms has been carried out in the past (Kabsh and Sander, 1983). However, this task can be deceptive if factors such as the selection of proteins for the testing set and the choice of the scoring index are not carried out properly. The present work alms to provide an updated evaluation of several predictive methods with a testing set size that permits to obtain more accurate statistics, which in turn can possibly measure the usefulness of the information gathered by those methods and also identify trends that characterize the behavior of individual algorithms. Further, we present a uniform testing of these methods, vis-a-vis the size of the datasets, the measure of accuracy and proper cross-validation procedures. 

A common practice for defining a test set in external validation is to randomly select a portion of chemicals from a dataset. From this perspective, cross validation provides a similar measure of model performance for a given and fixed set of chemicals. In cross validation a fraction of chemicals in the training set are excluded, and then predicted by the model generated from the remaining chemicals. As each chemical is excluded one at a time, and the... 

Most, if not all, QSAR methods require selection of relevant or informative descriptors before modeling is actually performed. This is necessary because the method could otherwise be more susceptible to the effects of noise. The a priori selection of descriptors, however, carries with it the additional risk of selection bias [73], when the descriptors are selected before the dataset is divided into the training and test sets (Figure 6.6A). Because of selection bias, both external validation and cross validation could significantly overstate pre-... 

Validation without an independent test set. Each application of the adaptive wavelet algorithm has been applied to a training set and validated using an independent test set. If there are too few observations to allow for an independent testing and training data set, then cross validation could be used to assess the prediction performance of the statistical method. Should this be the situation, it is necessary to mention that it would be an extremely computational exercise to implement a full cross-validation routine for the AWA. That is. it would be too time consuming to leave out one observation, build the AWA model, predict the deleted observation, and then repeat this leave-one-out procedure separately. In the absence of an independent test set, a more realistic approach would be to perform cross-validation using the wavelet produced at termination of the AWA, but it is important to mention that this would not be a full validation. 

---