Overfitting and Model Validation

Throughout the above discussion of quantitative modeling tools, a recurrent theme is the danger of overfitting a model, through the use of too many variables in MLR, too many estimated pure components in CLS, too many factors in PCR and PLS, or too many hidden nodes in ANN. This danger cannot be understated, not only because it is so 

As a result, when the model is used in practice, it is much more sensitive to any condition that deviates from the conditions used to build the model. In process analytical applications, where there is significant error in the analyzer and reference data anyway, the second disadvantage is usually the most visible one. 

A less tempting, but nonetheless dangerous alternative, is to under-fit a model. In this case, the model is not sufficiently complex to account for interfering effects in the analyzer data. As a result, the model can provide inaccurate results even in cases where it is applied to conditions that were used to build it Under-fitting can occur if one is particularly concerned about overfitting, and zealously avoids any added complexity to the model, even if it results in the model explaining more useful information. 

If overfitting and under-fitting are such big problems, then how can one avoid them The most commonly used tools for combating them are called model validation techniques. There are several tools that fall under this category, but they all operate with the same objective attempt to assess the performance of the model when it is applied to data that were not used to build it  

In external validation, a model is tested using data that were not used to build the model. This type of validation is the most intuitively straightforward of the validation techniques. If the external samples are sufficiently representative of the samples that will be applied to the model during its operation, then this technique can be used to provide a reasonable assessment of the model s prediction performance on future samples, as well as to provide a good assessment of the optimal complexity of the model. 

---