Lack-of-fit test

By using only simple hand calculations, the single-site model has been rejected and the dual-site model has been shown to represent adequately both the initial-rate and the high-conversion data. No replicate runs were available to allow a lack-of-fit test. In fact this entire analysis has been conducted using only 18 conversion-space-time points. Additional discussion of the method and parameter estimates for the proposed dual-site model are presented elsewhere (K5). Note that we have obtained the same result as available through the use of nonintrinsic parameters. 

However, this particular experimental design only covered values of x3 up to 1.68 consequently, the saddle point is only predicted by the model and not exhibited by the data. This is the reason the lack-of-fit tests of Section IV indicated neither model 3 nor model 4 of Table XVI could be rejected as inadequately representing the data. As is apparent, additional data must be taken in the vicinity of the stationary point to confirm this predicted nature of the surface and hence to allow rejection of certain models. This region of experimentation (or beyond) is also required by the parameter estimation and model discrimination designs of Section VII. 

Figure 30 portrays the grid of values of the independent variables over which values of D were calculated to choose experimental points after the initial nine. The additional five points chosen are also shown in Fig. 30. Note that points at high hydrogen and low propylene partial pressures are required. Figure 31 shows the posterior probabilities associated with each model. The acceptability of model 2 declines rapidly as data are taken according to the model-discrimination design. If, in addition, model 2 cannot pass standard lack-of-fit tests, residual plots, and other tests of model adequacy, then it should be rejected. Similarly, model 1 should be shown to remain adequate after these tests. Many more data points than these 14 have shown less conclusive results, when this procedure is not used for this experimental system.

Formal tests are also available. The ANOVA lack-of-fit test ° capitalizes on the decomposition of the residual sum of squares (RSS) into the sum of squares due to pure error SSs and the sum of squares due to lack of fit SSiof. Replicate measurements at the design points must be available to calculate the statistic. First, the means of the replicates (4=1,. .., m = number of different design points) at all design points are calculated. Next, the squared deviations of all replicates U — j number of replicates] from their respective mean... 

Here, we want to emphasize that one is able to calculate the fraction of the experimental error only if replicate measurements (at least at one point x ) have been taken. It is then possible to compare model and experimental errors and to test the sources of residual errors. Then, in addition to the GOF test one can perform the test of lack of fit, LOF, and the test of adequacy, ADE, (commonly used in experimental design). In the lack of fit test the model error is tested against the experimental error and in the adequacy test the residual error is compared with the experimental error. 

The suitability of the regression model should be proven by a special statistical lack-of-fit-test, which is based on an analysis of variance (ANOVA). Here the residual sum of squares of regression is separated into two components the sum of squares from lack-of-fit (LOF) and the pure error sum of squares (PE, pure errors)... 

The linearity of (a part of) the range should be evaluated to check the appropriateness of the straight-line model. This can be achieved by a graphical evaluation of the residual plots or by using statistical tests. It is strongly recommended to use the residual plots in addition to the statistical tests. Mostly, the lack-of-fit test and Mandel s fitting test are used to evaluate the linearity of the regression line [8, 10]. The ISO 8466 describes in detail the statistical evaluation of the linear calibration function [11]. 

Lack-of-Fit Test The best-known statistical test to evaluate the appropriateness of the chosen regression model is the lack-of-fit test [6]. A prerequisite for this test is the availability of replicate measurements. An analysis of variance (ANOVA) approach is used in this test. The total sum of squares (SS V) can be written as follows ... 

The idea of the lack-of-fit test is to compare the pure error of the regression line with the error due to the use of an inappropriate regression model. The MSLOf, which is a measure for the spread of the mean response per concentration from the regression line, is divided by the MSpE, which is a measure for the spread of the instrument response due to experimental variation. The obtained F-value (F = MS[ oii/MS i) is compared with the F-distribution with k 2 and n k d.f. 

Table 13.13 ANOVA table for the regression with the lack of fit test ...

After outliers have been purged from the data and a model has been evaluated visually and/or by, e.g. residual plots, the model fit should also be tested by appropriate statistical methods [2, 6, 9, 10, 14], The fit of unweighted regression models (homoscedastic data) can be tested by the ANOVA lack-of-fit test [6, 9]. A detailed discussion of alternative statistical tests for both unweighted and weighted calibration models can be found in Ref. [16]. The widespread practice to evaluate a calibration model via its coefficients of correlation or determination is not acceptable from a statistical point of view [9]. 

The three extra experiments at the center point (xj = X2 = 0) were included to allow an estimate of the lack of fit of the model. We will not perform any statistical analysis of this particular model. The lack-of-fit tests will be discussed in the context of the composite designs. 

A couple of additional checks are performed for the adequacy of the model. The coefficient of determination, or value, is 96.8%, which is very good. A lack-of-fit test is performed comparing the error in the estimated values at each data point with the estimated noise obtained from the repeated trials. The lack-of-fit test passes, indicating that the model s fit to the data is within the accuracy expected based on the data s noise. 

The significant effects for the burn are given in italic in Table 6. The arcsine of the fraction defective is analyzed. Only HB and DT have been identified as critical inputs and were included in the model. The above model has an R-squared value of 79.6% and passes the lack of fit test. Removing the two quadratic terms, which both tested non-significant, causes the lack of fit test to fail. Therefore, it was decided to leave the two quadratic terms in the fitted equation ... 

To evaluate the fit of the calculated model, usually ANOVA is applied (1, 7,17,116). ANOVA will evaluate the data set variation. Often a test for the significance of regression and a lack-of-fit test are performed (7,17,116). A model is then considered adequate and well fitted to the data when both a significant regression and a nonsignificant lack-of-fit occur. 

Analysis of variance for each dependent variable showed that in almost all cases, R2 coefficients higher than 0.83 were obtained (Table 2), which means that models were able to explain more than 83% of the observed responses. For the rate of gelation, thermal hysteresis and hardness, the lack of fit test was not significant. For Tge, and Tm, the lack of fit was significant, which means that the model may not have included all appropiate function of independent variables. According to Box and Draper,13 we considered the high coefficients R2 as evidence of the applicability of the model. 

Let us now perform the lack-of-fit test with MiniTab using the original data as shown in Table 2.11. 

For data that resemble Pattern C, the researcher needs to up the power scale of X (x, r , etc.) or down the power scale of y y/y, log y, etc.) to linearize the data. For reasons previously discussed, it is recommended to transform the y values only, leaving the x values in the original form. In addition, once the data have been reexpressed, plot them to help determine visually if the reexpression adequately linearized them. If not, the next lower power transformation should be used, on the y value in this case. Once the data are reasonably linear, as determined visually, the F test for lack of fit can be used. Again, the smaller the Fc value, the better. If, say, the data are not quite linearized by y/y but are slightly curved in the opposite direction with the log y transformation, pick the reexpression with the smaller F value in the lack-of-fit test. 

Recall that the lack-of-fit test partitions the sum of squares error (SSe) into two components pure error, the actual random error component and lack of fit, a nonrandom component that detects discrepancies in the model. The lack-of-fit computation is a measure of the degree to which the model does not fit or represent the actual data. 

Classical statistical tests can be applied mainly to validate regression models that are linear with respect to the model parameters. The most common empirical models used in EXDE are linear models (main effect models), linear plus interactions models, and quadratic models. They all are linear with respect to the p>arameters. The most useful of these (in DOE context) are 1) t-tests for testing the significance of the individual terms of the model, 2) the lack-of-fit test for testing the model adequacy, and 3) outlier tests based on so-called externally studentized residuals, see e.g. (Neter et. al., 1996). 

The lack-of fit test can be apvphed only if the design contains replicate experiments which permit estimation of the so-called pure error, i.e. an error term that is free from modelling errors. Assuming that the replicate experiments are included in regression, the calculations are carried out according to the following equations. First calculate the pure error sum of squares SS ... 

According to Table 22 only the intercept and the quadratic effect of are significant. The p-value of the lack-of-fit test based on the 3 replicates is ca. 0.28. Thus the lack-of-fit is not significant. The apparent reason for the low significance is the rather poor repeatabiUty of the experiments. The standard deviation of the recoveries of the replicate experiments is ca. 1.68 which is relatively high compared to the overall variation in the recoveries. 

The lack-of-fit test is based on the comparison of the MSS due to the model and the experimental error ... 

For very small experimental error, the F value in the lack-of-fit test might be extraordinarily large and the test becomes significant. It is to be decided then whether this result is also practically significant. 

The importance of applying the lack-of-fit test was demonstrated by Loco and coworkers by showing that a straight-line model, with a high correlation coefficient, but with a lack of fit, yields significantly less accurate results than its curvilinear alternative. 

The lack-of-fit test has gained popularity however, it requires the measurement of true replicates for the calibration solutions. This is not always possible and an alternative to compare the linear fit to a non-linear one was proposed in 1964 by statistician and chemist John Mandel, while working for the then US National Bureau of Standards." This test, despite being simple and quite straightforward, has not been used broadly although its use has increased steadily. A reason for its lack of popularity might be reluctance of analysts to fit a set of data to a polynomial model. Nevertheless, this is... 

Another interesting application of Mandel s test is to decide on the linear calibration range. It consists of the deletion of the highest concentration levels of the standardization and application of Mandel s test repeatedly. This will yield a concentration range where it can be proved statistically that the function is linear and that quadratic terms are not required. This can also be done by the successive application of the lack of fit test. ... 

Finally, the model can be validated in a next step. Usually an ANOVA is applied to evaluate the dataset and determine the significance of the terms included in the model. The model can be recalculated with the least nonsignificant terms deleted. Also, tests for the significance of the regression coefficients, the lack of fit test and a residual analysis are often performed. 

Box and Draper warn, however, that the investigator should not resort immediately to the joint analysis of responses. Rather he should. .. consider the consistency in the minima of individual and combined (the minors of V) responses. Yet a formal lack of fit test has not been proposed. 

---