Model overfitting

In practice, the choice of an optimal number of PCs to retain in the PCA model (A) is a rather snbjective process, which balances the need to explain as much of the original data as possible with the need to avoid incorporating too much noise into the PCA model (overfitting). The issne of overfitting is discnssed later in Section 12.4. 

Another potential disadvantage of PLS over PCR is that there is a higher potential to overfit the model through the use of too many PLS factors, especially if the Y-data are rather noisy. In such situations, one could easily run into a situation where the addition of a PLS factor helps to explain noise in the Y-data, thus improving the model fit without an improvement in real predictive ability. As for all other quantitative regression methods, the use of validation techniques is critical to avoid model overfitting (see Section 8.3.7). 

How small it can legitimately be, however, depends on the relative size of the random errors affecting the response values, else there would be model overfitting. We will return to this point later. 

Another method of detection of overfitting/overtraining is cross-validation. Here, test sets are compiled at run-time, i.e., some predefined number, n, of the compounds is removed, the rest are used to build a model, and the objects that have been removed serve as a test set. Usually, the procedure is repeated several times. The number of iterations, m, is also predefined. The most popular values set for n and m are, respectively, 1 and N, where N is the number of the objects in the primary dataset. This is called one-leave-out cross-validation. 

Evaluating the model in tenns of how well the model fits the data, including the use of posterior predictive simulations to determine whether data predicted from the posterior distribution resemble the data that generated them and look physically reasonable. Overfitting the data will produce unrealistic posterior predictive distributions. 

In all modeling techniques, and neural networks in particular, care must be taken not to overtrain or overfit the model. 

We chose the number of PCs in the PCR calibration model rather casually. It is, however, one of the most consequential decisions to be made during modelling. One should take great care not to overfit, i.e. using too many PCs. When all PCs are used one can fit exactly all measured X-contents in the calibration set. Perfect as it may look, it is disastrous for future prediction. All random errors in the calibration set and all interfering phenomena have been described exactly for the calibration set and have become part of the predictive model. However, all one needs is a description of the systematic variation in the calibration data, not the... 

Leaving out one object at a time represents only a small perturbation to the data when the number (n) of observations is not too low. The popular LOO procedure has a tendency to lead to overfitting, giving models that have too many factors and a RMSPE that is optimistically biased. Another approach is k-fold cross-validation where one applies k calibration steps (5 < k < 15), each time setting a different subset of (approximately) n/k samples aside. For example, with a total of 58 samples one may form 8 subsets (2 subsets of 8 samples and 6 of 7), each subset tested with a model derived from the remaining 49 or 50 samples. In principle, one may repeat this / -fold cross-validation a number of times using a different splitting [20]. 

In practice, the choice of parameters to be refined in the structural models requires a delicate balance between the risk of overfitting and the imposition of unnecessary bias from a rigidly constrained model. When the amount of experimental data is limited, and the model too flexible, high correlations between parameters arise during the least-squares fit, as is often the case with monopole populations and atomic displacement parameters [6], or with exponents for the various radial deformation functions [7]. 

The number of latent variables (PLS components) must be determined by some sort of validation technique, e.g., cross-validation [42], The PLS solution will coincide with the corresponding MLR solution when the number of latent variables becomes equal to the number of descriptors used in the analysis. The validation technique, at the same time, also serves the purpose to avoid overfitting of the model. 

All regression methods aim at the minimization of residuals, for instance minimization of the sum of the squared residuals. It is essential to focus on minimal prediction errors for new cases—the test set—but not (only) for the calibration set from which the model has been created. It is relatively easy to create a model— especially with many variables and eventually nonlinear features—that very well fits the calibration data however, it may be useless for new cases. This effect of overfitting is a crucial topic in model creation. Definition of appropriate criteria for the performance of regression models is not trivial. About a dozen different criteria— sometimes under different names—are used in chemometrics, and some others are waiting in the statistical literature for being detected by chemometricians a basic treatment of the criteria and the methods how to estimate them is given in Section 4.2. 

The complexity of the model can be controlled by the number of components and thus overfitting can be avoided and maximum prediction performance for test set data can be approached. 

In variable selection models with a different number of variables have to be compared, and the applied performance criteria must consider the number of variables in the compared models. The model should not contain a too small number of variables because this leads to poor prediction performance. On the other hand, it should also not contain a too large number of variables because this results in overfitting and thus again poor prediction performance (see Section 4.2.2). In the following, m denotes the number of regressor variables (including the intercept if used) that are selected for a model. 

The number of segments in the outer and inner loop (. 0ut and sin, respectively) may be different. Each loop of the outer CV results in an optimum complexity (for instance, optimum number of PLS components, Aopt)- In general, these Sout values are different for a final model the median of these values or the most frequent value can be chosen (a smaller complexity would avoid overfitting, a larger complexity would result in a more detailed model but with the risk of overfitting). A final model can be created from all n objects applying the final optimum complexity the prediction performance of this model has been estimated already by double CV. This strategy is especially useful for PLS and PCR. 

Use of all variables will produce a better fit of the model for the training data because the residuals become smaller and thus the R2 measure increases (see Section 4.2). However, we are usually not interested in maximizing the fit for the training data but in maximizing the prediction performance for the test data. Thus a reduction of the regressor variables can avoid the effects of overfitting and lead to an improved prediction performance. 

Nonlinear models often tend to overfitting and should thus be used very carefully and only after having shown that linear models are not satisfactory. 

However, there are prices to pay for the advantages above. Most empirical modeling techniques need to be fed large amounts of good data. Furthermore, empirical models can only be safely applied to conditions that were represented in the data used to build the model (i.e., extrapolation of such models is very dangerous). In addition, the availability of multiple response variables for building a model results in the temptation to overfit models, in order to obtain artificially optimistic results. Finally, multivariate models are usually much more difficult to explain to others, especially those not well versed in math and statistics. 

Like MLR, however, one must be careful to avoid the temptation of overfitting the PCR model. In this case, overfitting can occur through the use of too many principal components, thus adding unwanted noise to the model and making the model more sensitive to unforeseen disturbances. Model validation techniques (discussed in Section 12.4) can be used to avoid overfitting of PCR models. 

Unfortunately, the ANN method is probably the most susceptible to overfitting of the methods discussed thus far. For similar N and M, ANNs reqnire many more parameters to be estimated in order to define the model. In addition, cross validation can be very time-consuming, as models with varying complexity (nnmber of hidden nodes) mnst be trained individually before testing. Also, the execntion of an ANN model is considerably more elaborate than a simple dot product, as it is for MLR, CLS, PCR and PLS (Eqnations 12.34, 12.37, 12.43 and 12.46). Finally, there is very little, or no, interpretive value in the parameters of an ANN model, which eliminates one nseful means for improving the confidence of a predictive model. 

Like ANNs, SVMs can be useful in cases where the x-y relationships are highly nonlinear and poorly nnderstood. There are several optimization parameters that need to be optimized, including the severity of the cost penalty , the threshold fit error, and the nature of the nonlinear kernel. However, if one takes care to optimize these parameters by cross-validation (Section 12.4.3) or similar methods, the susceptibility to overfitting is not as great as for ANNs. Furthermore, the deployment of SVMs is relatively simpler than for other nonlinear modeling alternatives (such as local regression, ANNs, nonlinear variants of PLS) because the model can be expressed completely in terms of a relatively low number of support vectors. More details regarding SVMs can be obtained from several references [70-74]. 

The principal component space does not have the redundancy issue discussed above, because the PCs are orthogonal to one another. In addition, because each PC explains the most remaining variance in the x data, it is often the case that fewer PCs than original x variables are needed to capture the relevant information in the x data. This leads to simpler classification models, less susceptibility to overfitting through the use of too many dimensions in the model space, and less noise in the model. 

However, the PLS-DA method requires sufficient calibration samples for each class to enable effective determination of discriminant thresholds, and one must be very careful to avoid overfitting of a PLS-DA model through the use of too many PLS factors. Also, the PLS-DA method does not explicitly account for response variations within classes. Although such variations in the calibration data can be useful information for assigning and determining uncertainty in class assignments during prediction, it will be treated essentially... 

---