Cross-validation purposes

If we did not have a validation set available to us, we could use cross-validation for the same purposes. Figure 55 contains plots of the results of cross validation of the two training sets, A1 and A2. Since no separate validation data set is involved, we name the results PCRCROSS1 and PCRCROSS2, respectively. 

A pharmaceutical specialty is produced in three dosage strengths (major component A) A and a second component B are controlled by HPLC for batch release purposes. It is decided to replace the manual injection of the sample solution by an automatic one. It is expected the means will remain the same but the standard deviations will be smaller for the automatic injection. Cross-validation of the methods is effected by running both methods on each of 10 samples. The mean and the standard deviation for each series of 10 measurements is given in Table 4.19. 

The number of latent variables (PLS components) must be determined by some sort of validation technique, e.g., cross-validation [42], The PLS solution will coincide with the corresponding MLR solution when the number of latent variables becomes equal to the number of descriptors used in the analysis. The validation technique, at the same time, also serves the purpose to avoid overfitting of the model. 

The essential characteristic of a proper test set is that it represents a new drawing from the population , realized as a new, independent [X,Y] data set specifically not used in the modeling. It is evident that any A -object data set constitute but only one specific realization of an iV-tuple of individual TSE materializations. It takes a completely new ensemble of objects, the test set, to secure a second manifestation. All new measurements, for example when a PAT model is used for the purpose of automated prediction, constitute precisely such a new drawing/sampling. All new measurement situations are therefore to be likened to a test set - and this is exactly what is missing in all forms of cross-validation. 

An important aspect of variable selection that is often overlooked is the hazard brought about through the use of cross-validation for two quite different purposes namely (1) as an optimization criterion for variable selection and other model optimization tasks (including selection of the optimal number of PLS LVs or PCR PCs) and (2) as an assessment of the quality of the final model built using all samples. In this case, one can get highly optimistic estimates of a model s performance, because the same criterion is used to both optimize and evaluate the model. As a result, when doing variable selection, especially with a limited number of calibration samples, it is advisable to do an additional outer loop cross-validation across the entire model... 

The first treated is test-set and cross-validation which are general tools useful for many purposes in validation. For example, cross-validation is often used in two-way analysis for establishing the number of components in PCA or PLS models. In essence, test-set and cross-validation simply provide methods for obtaining more realistic residuals than those obtained by ordinary residuals from fitted models. Thus, test-set and cross-validation are of general interest when residuals are used for reporting explained variance, assessing outliers etc. 

In multi-way analysis, a distinction is made between cross-validation of regression models and cross-validation of component models. In regression models, the purpose is to predict a y and cross-validation can then be used to find the best regression model to achieve this. In component models cross-validation can be used to find the most parsimonious but adequate... 

If the filter coefficients are to be used for discriminatory purposes, then the criterion function should strive to reflect differences among classes. In this section three suitable discriminant criterion functions are described. These discriminant criterion functions are Wilk s lambda (3a), entropy (3e), and the cross-validated quadratic probability measure (3cvqpm)-... 

During method development (Chapter 9) and validation (Chapter 10), QCs are used for several purposes including checks on precision and accuracy, lower limit of quantitation (LLOQ), recovery and method robustness and ruggedness (Section 9.8.4), as well as stability studies of various kinds (Sections 10.2.7 and 10.2.8), studies of inter-day validation within a specified laboratory and cross-validations in inter-laboratory method transfer (Section 10.2.11). QC samples are also used during method development to assess the final method prior to validation experimental runs that use QCs for this purpose are often referred to as assay prequalifications or pre-study assay evaluations (PSAE). 

To discuss the prediction error, one must validate the calibration model [2]. There are two sorts of validation. One method is based on a new set of objects (external prediction). It requires a large and representative set of objects which have to be kept apart from the calibration for testing purposes only. The other validation method is based on the calibration data themselves (internal validation). In most cases, internal validation methods such as cross-validation and leverage correction [2] give sensible results with valuable information about the prediction ability. Cross-validation seeks to validate the calibration model with independent test data, but contrary to external validation it does not use data for testing only. The cross-validation is performed a number of times, each time with the use of only a few calibration samples as a test set. From the validation set it is possible to compare the prediction ability for the models, expressed by the estimated prediction mean square error. 

The use of a new data set or data splitting for the purpose of cross validation may not always be applicable or desirable. In addition, the results depend on the location of the split. An alternative is to define the prediction... 

---