Lack Of Fitting

For each experiment, the true values of the measured variables are related by one or more constraints. Because the number of data points exceeds the number of parameters to be estimated, all constraint equations are not exactly satisfied for all experimental measurements. Exact agreement between theory and experiment is not achieved due to random and systematic errors in the data and to "lack of fit" of the model to the data. Optimum parameters and true values corresponding to the experimental measurements must be found by satisfaction of an appropriate statistical criterion. 

In the maximum-likelihood method used here, the "true" value of each measured variable is also found in the course of parameter estimation. The differences between these "true" values and the corresponding experimentally measured values are the residuals (also called deviations). When there are many data points, the residuals can be analyzed by standard statistical methods (Draper and Smith, 1966). If, however, there are only a few data points, examination of the residuals for trends, when plotted versus other system variables, may provide valuable information. Often these plots can indicate at a glance excessive experimental error, systematic error, or "lack of fit." Data points which are obviously bad can also be readily detected. If the model is suitable and if there are no systematic errors, such a plot shows the residuals randomly distributed with zero means. This behavior is shown in Figure 3 for the ethyl-acetate-n-propanol data of Murti and Van Winkle (1958), fitted with the van Laar equation. 

The last example serves to show that in some cases the exponentiated polynomial function used to estimate the true parameter distribution can show serious lack of fit. Therefore other estimating functions are required. 

A number of replications under at least one set of operating conditions must be carried out to test the model adequacy (or lack of fit of the model). An estimate of the pure error variance is then calculated from ... 

An F-test for lack of fit is based on the ratio of the lack of fit sum to the pure error sum of squares divided by their corresponding degrees of freedom ... 

The unknown model parameters will be obtained by minimizing a suitable objective function. The objective function is a measure of the discrepancy or the departure of the data from the model i.e., the lack of fit (Bard, 1974 Seinfeld and Lapidus, 1974). Thus, our problem can also be viewed as an optimization problem and one can in principle employ a variety of solution methods available for such problems (Edgar and Himmelblau, 1988 Gill et al. 1981 Reklaitis, 1983 Scales, 1985). Finally it should be noted that engineers use the term parameter estimation whereas statisticians use such terms as nonlinear or linear regression analysis to describe the subject presented in this book. 

If the nonlinear estimation procedure is carefully applied, a minimum in the sums-of-squares surface can usually be achieved. However, because of the fitting flexibility generally obtainable with these nonlinear models, it is seldom advantageous to fit a large number of models to a set of data and to try to eliminate inadequate models on the basis of lack of fit (see Section IV). For example, thirty models were fitted to the alcohol dehydration data just discussed (K2). As is evident from the residual mean squares of Table II, approximately two-thirds of the models exhibit an acceptable fit of the data... 

Here, y is the average of all of the replicated data points. If the residual sum of squares is the amount of variation in the data as seen by the model, and the pure-error of squares is the true measure of error in the data, then the inability of the model to fit the data is given by the difference between these two quantities. That is, the lack-of-fit sum of squares is given by... 

If there are n replications at q different settings of the independent variables, then the pure-error sum of squares is said to possess (n — 1) degrees of freedom (1 degree of freedom being used to estimate y) while the lack-of-fit sum of squares is said to possess N — p — q(n — 1) degrees of freedom, i.e., the difference between the degrees of freedom of the residual sum of squares and the pure-error sum of squares. 

The sums of squares of the individual items discussed above divided by its degrees of freedom are termed mean squares. Regardless of the validity of the model, a pure-error mean square is a measure of the experimental error variance. A test of whether a model is grossly adequate, then, can be made by acertaining the ratio of the lack-of-fit mean square to the pure-error mean square if this ratio is very large, it suggests that the model inadequately fits the data. Since an F statistic is defined as the ratio of sum of squares of independent normal deviates, the test of inadequacy can frequently be stated... 

In some cases when estimates of the pure-error mean square are unavailable owing to lack of replicated data, more approximate methods of testing lack of fit may be used. Here, quadratic terms would be added to the models of Eqs. (32) and (33), the complete model would be fitted to the data, and a residual mean square calculated. Assuming this quadratic model will adequately fit the data (lack of fit unimportant), this quadratic residual mean square may be used in Eq. (68) in place of the pure-error mean square. The lack-of-fit mean square in this equation would be the difference between the linear residual mean square [i.e., using Eqs. (32) and (33)] and the quadratic residual mean square. A model should be rejected only if the ratio is very much greater than the F statistic, however, since these two mean squares are no longer independent. 

This is, then, the regression sum of squares due to the first-order terms of Eq. (69). Then, we calculate the regression sum of squares using the complete second-order model of Eq. (69). The difference between these two sums of squares is the extra regression sum of squares due to the second-order terms. The residual sum of squares is calculated as before using the second-order model of Eq. (69) the lack-of-fit and pure-error sums of squares are thus the same as in Table IV. The ratio contained in Eq. (68) still tests the adequacy of Eq. (69). Since the ratio of lack-of-fit to pure-error mean squares in Table VII is smaller than the F statistic, there is no evidence of lack of fit hence, the residual mean square can be considered to be an estimate of the experimental error variance. The ratio... 

By using only simple hand calculations, the single-site model has been rejected and the dual-site model has been shown to represent adequately both the initial-rate and the high-conversion data. No replicate runs were available to allow a lack-of-fit test. In fact this entire analysis has been conducted using only 18 conversion-space-time points. Additional discussion of the method and parameter estimates for the proposed dual-site model are presented elsewhere (K5). Note that we have obtained the same result as available through the use of nonintrinsic parameters. 

However, this particular experimental design only covered values of x3 up to 1.68 consequently, the saddle point is only predicted by the model and not exhibited by the data. This is the reason the lack-of-fit tests of Section IV indicated neither model 3 nor model 4 of Table XVI could be rejected as inadequately representing the data. As is apparent, additional data must be taken in the vicinity of the stationary point to confirm this predicted nature of the surface and hence to allow rejection of certain models. This region of experimentation (or beyond) is also required by the parameter estimation and model discrimination designs of Section VII. 

Figure 30 portrays the grid of values of the independent variables over which values of D were calculated to choose experimental points after the initial nine. The additional five points chosen are also shown in Fig. 30. Note that points at high hydrogen and low propylene partial pressures are required. Figure 31 shows the posterior probabilities associated with each model. The acceptability of model 2 declines rapidly as data are taken according to the model-discrimination design. If, in addition, model 2 cannot pass standard lack-of-fit tests, residual plots, and other tests of model adequacy, then it should be rejected. Similarly, model 1 should be shown to remain adequate after these tests. Many more data points than these 14 have shown less conclusive results, when this procedure is not used for this experimental system.

Finally, a measure of lack of fit using a PCs can be defined using the sum of the squared errors (SSE) from the test set, flSSETEST = Latest 2 (prediction sum of squares). Here, 2 stands for the sum of squared matrix elements. This measure can be related to the overall sum of squares of the data from the test set, SStest = -Xtest 2- The quotient of both measures is between 0 and 1. Subtraction from 1 gives a measure of the quality of fit or explained variance for a fixed number of a PCs ... 

For example, a) in (radioactivity) counting experiments a non-Poisson random error component, equal in magnitude (variance) to the Poisson component, will not be detected until there are 46 degrees of freedom ( ), and b) it was necessary for a minor component in a mixed Y-ray spectrum to exceed its detection limit by -50 , before its absence was detected by lack-of-fit (x, model error) (7). 

This is perhaps the "best solution for the given data set, and it is certainly the most interesting. It is not offered as a rigorous solution, however, for the lack of fit (x /df -[9.64]2) implies additional sources of error, which may be due to additional scatter about the calibration curve (oy -"between" component), residual error in the analytic model for the calibration function, or errors in the "standard" x-values. (We believe the last source of error to be the most likely for this data set.) For these reasons, and because we wish to avoid complications introduced by non-linear least squares fitting, we take the model y=B+Axl 12 and the relation Oy = 0.028 + 0.49x to be exact and then apply linear WLS for the estimation of B and A and their standard errors. 

The amount data corresponding to the response values in 1 above were transformed by the same general family of power transformations until linearity was obtained. The F-test statistic that relates lack of fit and pure error was used as the criterion for linearity. 

Amount Transformation. Step 2. The amount transformation was performed in a way similar to that of response by use of a power series but for a different reason. In this case linearity was desired in order to use a simple linear regression model. This transformation therefore required a test for satisfactory conformity. One can use a variety of criteria including the correlation coefficient or visual examination of the plot of rgsiduals verses amount. We chose the F test for lack of fit,... 

F =MSLF/MSPE, based on the ratio mean square for lack of fit (MSLF) over the mean square for pure error (MSPE) ( 31 ). F follows the F distribujfion with (r-2) and (N-r) degrees of freedom. A value of F regression equation. Since the data were manipulated by transforming the amount values jfo obtain linearity, i. e., to achieve the smallest lack of fit F statistic, the significance level of this test is not reliable. 

At any transformation level if the minimum F statistic were less than or equal to the critical F value, our work was done and the confidence band calculations began. Otherwise we either accepted a lack of fit (and would note it in published results), segmented the graph to shorter lengths, or sought a non-linear or higher order model. 

Wegscheider fitted a cubic spline function to the logarithmically transformed sample means of each level. This method obviates any lack of fit, and so it is not possible to calculate a confidence band about the fitted curve. Instead, the variance in response was estimated from the deviations of the calibration standards from their means at an Ot of 0.05. The intersection of this response interval with the fitted calibration line determined the estimated amount interval. 

The relationship takes into account that the day-to-day samples determined are subject to error from several sources random error, instrument error, observer error, preparation error, etc. This view is the basis of the process of fitting data to a model, which results in confidence intervals based on the intrinsic lack of fit and the random variation in the data. 

First, amount error estimations in Wegscheider s work were the result of only the response uncertainty with no regression (confidence band) uncertainty about the spline. His spline function knots were found from the means of the individual values at each level. Hence the spline exactly followed the points and there was no lack of fit in this method. Confidence intervals around spline functions have not been calculated in the past but are currently being explored ( 5 ). 

Assessing linearity is an important aspect in calibration work since lack-of-fit will usually lead to biased results. When a simple linear regression model is chosen, the more general test of goodness-of-fit becomes a test of linearity. 

---