Underfitting

The ultimate goal of multivariate calibration is the indirect determination of a property of interest (y) by measuring predictor variables (X) only. Therefore, an adequate description of the calibration data is not sufficient the model should be generalizable to future observations. The optimum extent to which this is possible has to be assessed carefully when the calibration model chosen is too simple (underfitting) systematic errors are introduced, when it is too complex (oveifitting) large random errors may result (c/. Section 10.3.4). 

For k 1 (1 -NN), a new object would always get the same class membership as its next neighbor. Thus, for small values of k, it is easily possible that classes do no longer form connected regions in the data space, but they can consist of isolated clouds. The classification of new objects can thus be poor if k is chosen too small or too large. In the former case, we are concerned with overfitting, and in the latter case with underfitting. 

A less tempting alternative, which is equally dangerous, is to underfit a model. In this case, the model is not sufficiently complex to account for interfering effects in the analyzer data. As a result, the model can provide inaccurate results even in cases where it is applied to conditions that were used to build it ... 

Figure 12.25 provides a graphical explanation of the phenomena of over and underfitting [1], It shows that the overall prediction error of a model has contributions from two sources (1) the interference error and (2) the estimation error. The interference error continually decreases as the complexity of the calibration model increases, as the added complexity enables the model to explain more interferences in the analyzer data. At the same time, however, the estimation error of the model increases with the model complexity, because there are more independent model parameters that need to be estimated from the same limited set of data. These competing forces result in a conceptual minimum in the overall prediction error of a model, where the combination of interference error and estimation error are minimized. It should be noted that this explanation of a model s prediction error assumes that the calibration data are sufQciently representative of the data that will be obtained when the model is applied. 

Under tting Underfitting occurs when the model used to describe a data set is too simple. An example of this in regression analysis is the use of a straight line to describe the relationship between two variables when the true relationship is quadratic. (5ce also Overfitting.)... 

Figure 4.14 Taguchi s loss function to show the trade-off between bias (underfitting) and variance (overfitting) see Ref. [30] for more mathematical details. Here, k would be the optimum dimensionality of the PLS model. 

The optimal model is determined by finding the minimum error between the extracted concentrations and the reference concentrations. Cross-validation is also used to determine the optimal number of model parameters, for example, the number of factors in PLS or principal components in PCR, and to prevent over- or underfitting. Technically, because the data sets used for calibration and validation are independent for each iteration, the validation is performed without bias. When a statistically sufficient number of spectra are used for calibration and validation, the chosen model and its outcome, the b vector, should be representative of the data. 

Overfitting is the commonest problem in multivariate statistical procedures when the number of variables is greater than objects (samples) one can fit an elephant with enough variables. Tabachnick and Fidell (1983) have suggested minimum requirements for some multivariate procedures to avoid the overfitting or underfitting that can occur in a somewhat unpredictable manner, regardless of the multivariate procedure chosen. 

As the number of parameters in a model increases, the closeness of the predicted values to the observed values increases, but at the expense of estimating the model parameters. In other words, the residual sum of squares decreases as more parameters are added into a model, but the ability to precisely estimate those model parameters also decreases. When too many parameters are included in a model the model is said to be overfitted or overparameterized, whereas when too few parameters are included, the model is said to be underfitted. Overfitting produces parameter estimates that have larger variances than the simpler model, both in the parameter estimates and in predicted values. Underfitting results in biased parameter estimates and biased prediction estimates. As model complexity increases,... 

Akaike s criterion and its derivations has been called by some [see Verbeke and Molenberghs (2000) for example] as a minimization function plus a penalty term for the number of parameters being estimated. As more model parameters are added to a model, 2LL tends to decrease but 2 p increases. Hence, AIC may decrease to a point as more parameters are added to a model but eventually the penalty term dominates the equation and AIC begins to increase. Conceptually this fits into the concept of bias-variance trade-off or the trade-off between overfitting and underfitting. 

Model development in drug development is usually empirical or exploratory in nature. Models are developed using experimental data and then refined until a reasonable balance is obtained between overfitting and underfitting. This iterative process in model selection results in models that have overly optimistic inferential properties because the uncertainty in the model is not taken into account. No universally accepted solution to this problem has been found. 

Conversely, if we are too cautious and stop improving the model too soon, we will underfit the calibration data by not taking relevant effects into consideration. In this case, when we come to using the model for prediction, the unconsidered effects may change and push the model away from correct answers. 

So, how can we tell the difference between an underfitted, good, or over-fitted model when we have no external reference points This problem is the basis of model validation. 

Cross-validation is a good method for estimating the optimal number of components in latent variable methods because in these cases we expect the model to move from underfitted to optimal to overfitted through addition of components. A graph of MSEP versus number of components will therefore show a minimum at the point where the model is optimally fit (Wold, 1978). 

Models that include noise vectors or more vectors than are actually necessary to predict the constituent concentrations are called overfit. Models that do not have enough factors in them are known as underfit. 

In Fig. 10, notice that from 0 to 7 factors the prediction error (PRESS) decreases as each new factor is added to the model. This indicates that the model is underfit and there are not enough factors to completely account for the constituents of interest. 

To avoid building a model that is either overfit or underfit, the number of factors at which the PRESS plot reaches a minimum would be the obvious... 

Simply looking at a plot of the eigenvalues might lead to a model that has too few factors. Empirically, from examination of the eigenvalue plot in Fig. 9, it appears that a model with four factors would probably work fine because the values seem to be very small from this factor on up. However, in actuality, this would build a model that is significantly underfit for this data set. 

Take the AR(2) process data in Sect. A5.2 and fit an AR(1) process to it. Analyse the residuals and fit. Comment on the results. Repeat, but using an AR(3) model. Compare the two models with the accurate AR(2) model (see Example 5.8 for the model). What happens when a model is over- or underfit ... 

Underfitting increases modei bias (fitting error)... 

---