Bias versus Variance Tradeoff

As the number of parameters in a model increases, the closeness of the predicted values to the observed values increases, but at the expense of estimating the model parameters. In other words, the residual sum of squares decreases as more parameters are added into a model, but the ability to precisely estimate those model parameters also decreases. When too many parameters are included in a model the model is said to be overfitted or overparameterized, whereas when too few parameters are included, the model is said to be underfitted. Overfitting produces parameter estimates that have larger variances than the simpler model, both in the parameter estimates and in predicted values. Underfitting results in biased parameter estimates and biased prediction estimates. As model complexity increases, 

To further illustrate the bias-variance trade-off, consider the concentration-time (C, t) data in Fig. 1.12. The data were simulated using a two-exponent model 

The two-exponent model significantly decreased the SSE and also resulted in precise parameter estimates. The goodness of fit of the three-exponent model looked 

---

Figure 1.12 Scatter plot of data used to illustrate bias versus variance tradeoff. The dashed-dotted line is the fit to a one-exponent model. The dashed line is the fit to a two-exponent model and the solid line is the line of fit to a three-exponent model. Data were fit using nonlinear linear regression with weights equal to inverse concentration.

---