Cross-validation methods

The maximum number of latent variables is the smaller of the number of x values or the number of molecules. However, there is an optimum number of latent variables in the model beyond which the predictive ability of the model does not increase. A number of methods have been proposed to decide how many latent variables to use. One approach is to use a cross-validation method, which involves adding successive latent variables. Both leave-one-out and the group-based methods can be applied. As the number of latent variables increases, the cross-validated will first increase and then either reach a plateau or even decrease. Another parameter that can be used to choose the appropriate number of latent variables is the standard deviation of the error of the predictions, SpREss ... 

Cross-Validation methods make use of the fact that it is possible to estimate missing measurements when the solution of an inverse problem is obtained. 

Both the determination of the effective number of scatterers and the associated rescaling of variances are still in progress within BUSTER. The value of n at the moment is fixed by the user at input preparation time for charge density studies, variances are also kept fixed and set equal to the observational c2. An approximate optimal n can be determined empirically by means of several test runs on synthetic data, monitoring the rms deviation of the final density from the reference model density (see below). This is of course only feasible when using synthetic data, for which the perfect answer is known. We plan to overcome this limitation in the future by means of cross-validation methods. 

Unlike test set validation methods, cross-validation methods attempt to validate a model using the calibration data only, without requiring the preparation and analysis of an additional test set of samples. This involves the execution of one or more internal validation procedures (hereby called subvalidations), where each subvalidation involves three steps ... 

Cross-validation methods differ in how the sample subsets are selected for the subvalidation experiments. Several methods that are typically encountered in chemometrics software packages are listed below... 

The factors that influence the optimal cross-validation method, as well as the parameters for that method, are the number of calibration samples (AO, the arrangement order of the samples in the calibration data set, whether the samples arise from a design of experiments (DOE, Section 12.2.6), the presence or absence of replicate samples, and the specific objective of the cross-validation experiment. In addition, there are two traps that one needs to be aware of when setting up a cross-validation experiment. 

Zhang et al. [78] analysed the metal contents of serum samples by ICP-AES (Fe, Ca, Mg, Cr, Cu, P, Zn and Sr) to diagnose cancer. BAM was compared with multi-layer feed-forward neural networks (error back-propagation). The BAM method was validated with independent prediction samples using the cross-validation method. The best results were obtained using BAM networks. 

Regression statistics showed that 97.8% of the variation iixlfog polar solutes in isopropyl alcohol was accounted for by Equation 3.40. The quality of Equation 3.40 was also evaluated by a cross-validation method (Myers, 1990 Tripos, 1992). In this study, the dataset was divided into... 

Although all of these cross-validation methods can be used effectively, there could be an optimal method for a given application. The factors that most often influence the optimal cross-validation method are the design of the calibration experiment, the order of the samples in the calibration data set, and the total number of calibration samples (N). 

The selected subset cross-validation method is probably the closest internal validation method to external validation in that a single validation procedure is executed using a single split of subset calibration and validation data. Properly implemented, it can provide the least optimistic assessment of a model s prediction error. Its disadvantages are that it can be rather difficult and cumbersome to set it up so that it is properly implemented, and it is difficult to use effectively for a small number of calibration samples. It requires very careful selection of the validation samples such that not only are they sufficiently representative of the samples to be applied to the model during implementation, but also the remaining samples used for subset calibration are sufficiently representative as well. This is the case because there is only one chance given to test a model that is built from the data. 

To circumvent this issue, cross-validation methods have been proposed to evaluate the internal predictivity of the model by discarding one or several com-... 

As with univariate and multivariate calibration, three-way calibration assumes linear additivity of signals. When the sample matrix influences the spectral profiles or sensitivities, either care must be taken to match the standard matrix to those of the unknown samples, or the method of standard additions must be employed for calibration. Employing the standard addition method with three-way analysis is straightforward only standard additions of known analyte quantity are needed [42], When the standard addition method is applied to nonbilinear data, the lowest predicted analyte concentration that is stable with respect to the leave-one-out cross-validation method is unique to the analyte. 

Partial least squares (PLS) is similar to MLR in that it also assumes a linear relationship between a vector x and a target property y. However, it avoids the problems of collinear descriptors by calculating the principal components for the molecular descriptors and target property separately. The scores for the molecular descriptors are used as the feature vector x and are also used to predict the scores for the target property, which can in turn be used to predict y. An important consideration in PLS is the appropriate number of principal components to be used for the QSAR model. This is usually determined by using cross-validation methods like fivefold cross validation and leave-one-out. PLS has been applied to the prediction of carcinogenicity [19], fathead minnow toxicity [20], Tetrahymena pyriformis toxicity [21], mammalian toxicity [22], and Daphnia magna toxicity [23],... 

Selection of Optimal Tree. The optimal tree (most accurate tree) is the one having the highest predictive ability. Therefore, one has to evaluate the predictive error of the subtrees and choose the optimal one among them. The most common technique for estimating the predictive error is the cross-validation method, especially when the data set is small. The procedure of performing a cross validation is described earlier (see section 14.2.2.1). In practice, the optimal tree is chosen as the simplest tree with a predictive error estimate within one standard error of minimum. It means that the chosen tree is the simplest with an error estimate comparable to that of the most accurate one. 

For the sake of comparison, the results of the mechanistic model are also given in Table 14.2. To assess the robustness of the models, a 10-fold cross-validation method was used on all data sets (29). The consistency of the results of cross validation for all groups proved the stability and robustness of the models. Figure 14.6 demonstrates the plot of the CART-ANFIS calculated values for the acid mobilities against the experimental values. The high value of R = 0.970 for this plot indicates that the CART-ANFIS model can be considered as a powerful tool for the prediction of the electrophoretic mobihty of organic and sulfonic... 

There is yet only limited experience with these three-way cross-validation methods. The expectation maximization cross-validation method can be implemented in software without many difficulties. The leave-bar-out cross-validation method is more complex to implement. In general, the expectation maximization method outperforms the leave-bar-out method. In most cases the PRESS plots show a more distinct minimum and it is thus easier to identify the best model. However, the expectation maximization method requires much longer computation time [Louwerse et al. 1999] and this can be a problem for large arrays. 

One of the most useful and widely accepted cross validation methods is the use of Rfree (Brunger, 1992). A subset of reflections are withheld from the minimization process and only checked periodically for their agreement to Fc. If Rfree drops during the refinement, the crystallographer obtains an unbiased indication that the refinement has gone well and no gross errors have been introduced in the model. 

With their natural logarithm, the data in Table 2 and Table 3 were input to the Eq. (1) to build early-warning models in R environment (Wold, et al., 2001, Mevik Wehrens, 2007), respectively. The model parameters in Chongqing City and Ningbo City were shown in Table 5. One was called Chongqing Model, the other was named Ningbo model. All the models were assessed by Leave-One-Out Cross Validation method (LOOCV), and the maximum model error was less than 15%. 

Two cross-validation methods provide a better approach than that just described but generally not as good as the validation method. The first involves splitting of the available data into two subsets and the second involves removal of one datum at a time from the data set prior to building models. 

The optimum number of PLS factors used in the models was determined by a cross-validation method. In cross-validation, five samples were temporarily removed from the calibration set to be used for validation. With the remaining samples, a PLS model was developed and applied to predict the respective milk component for each sample in the group of five. Results were compared with the respective reference values. This procedure was repeated several times until prediction for all samples had been obtained. Performance statistics were accumulated for each group of removed samples. The validation errors were combined into a standard error of cross-validation. The optimum number of PLS factors in each equation was defined to be that which corresponded to the lowest standard error of cross-validation. The performance of each regression was tested with independent validation set of samples. 

---