Lack of fit and pure error

So far we have based the evaluation of our models on the appearance of the residual graph. If the distribution of the residuals does not show any systematic structure, that is, if it seems random, we consider the model satisfactory. This approach is no doubt quite subjective but we should not 

Source of variation Sum of squares Degree of freedom Mean square  

Reaction yield as a function of temperature, in the 30-70°C range, with catalyst A. The runs are in duplicate 

If our experiments furnish replicated response values, we can use them to obtain an estimate of random error. With this estimate, we will have a quantitative criterion to judge whether the chosen model is a good representation of the observations, or if we need to improve it. To show how this is done, we will present a numerical example, based on duplicates of the runs performed in the 30-70 °C range. 

Suppose that the runs in Table 5.4 have been performed in duplicate, and that our data are the 18 yields given in Table 5.7. For each X value there are now two different values of y. It is evident that no model will be able to pass through these two points at the same time. There wiU always be residuals no matter what model is chosen. We can attribute them, at least in part, to random errors. 

---

This is, then, the regression sum of squares due to the first-order terms of Eq. (69). Then, we calculate the regression sum of squares using the complete second-order model of Eq. (69). The difference between these two sums of squares is the extra regression sum of squares due to the second-order terms. The residual sum of squares is calculated as before using the second-order model of Eq. (69) the lack-of-fit and pure-error sums of squares are thus the same as in Table IV. The ratio contained in Eq. (68) still tests the adequacy of Eq. (69). Since the ratio of lack-of-fit to pure-error mean squares in Table VII is smaller than the F statistic, there is no evidence of lack of fit hence, the residual mean square can be considered to be an estimate of the experimental error variance. The ratio... 

The amount data corresponding to the response values in 1 above were transformed by the same general family of power transformations until linearity was obtained. The F-test statistic that relates lack of fit and pure error was used as the criterion for linearity. 

In anticipation of this kind of analysis, it is often useful to include the lack-of-fit and pure error within the basic ANOVA Table (Table 2.9). Note that the computation of lack-of-fit and pure error are a decomposition of SSe-... 

As one can see, the ANOVA consists of the regression and residual error (SSe) term. The regression is highly significant, with an of 390.00. The residual error (SSe) is broken into lack-of-fit and pure error. Moreover, the researcher sees that the lack-of-fit component is significant. That is, the linear model is not a precise fit, even though, from a practical perspective, the linear regression model may be adequate. 

With the partitioning of the residual sum of squares into contributions from lack of fit and pure error, the ANOVA table gains two new lines and becomes the complete version (Table 5.8). The pure error mean square. 

The first two lines represent the regression model and the residual, where the residual can be divided into two parts lack of fit and pure error. We start with the regression and residual used to test if the model is significant. The sum of squares due to regression, SSReg, can be calculated according to... 

Since certain experiments have been replicated (in this case all of the experiments, but the treatment is the same if only some of the experiments are repeated) the residual sum of squares may be divided further, into two parts pure error, and lack-of-fit. The pure error sum of squares is given by ... 

Of the 14 formulations listed in Table 1, six experimental runs were required to fit the quadratic mixture model, four additional distinct runs were used to check for the lack of fit, and finally four runs were replicated to provide an estimate of pure error. Design-Expert used the vertices, the edge centers, the overall centroid, and one point located halfway between the overall centroid and one of the edge centers as candidate points. Additionally, four vertices of the design region were used as check points [106],... 

The ratio = MS QpIMSp p = 1.69 is not significantly high, showing that there is no significant lack-of-fit, and the mean squares for the lack-of-fit and the pure error are comparable. Thus, the residual mean square MS psiD can be used as our estimate for the experimental variance. Taking its square root, the experimental standard deviation is estimated as 0.69, with 10 degrees of freedom. 

An F test to compare the variances associated with the lack of fit and the pure error allows us to decide whether the straight line model is eompatible with the experimental data. If the experimental value is greater than the tabulated one i.e. we can reject the null hypothesis) we should conclude that the model is inadequate." In this case a visual inspection of the residuals will help identify the cause of the problem. 

Here, y is the average of all of the replicated data points. If the residual sum of squares is the amount of variation in the data as seen by the model, and the pure-error of squares is the true measure of error in the data, then the inability of the model to fit the data is given by the difference between these two quantities. That is, the lack-of-fit sum of squares is given by... 

If there are n replications at q different settings of the independent variables, then the pure-error sum of squares is said to possess (n — 1) degrees of freedom (1 degree of freedom being used to estimate y) while the lack-of-fit sum of squares is said to possess N — p — q(n — 1) degrees of freedom, i.e., the difference between the degrees of freedom of the residual sum of squares and the pure-error sum of squares. 

In some cases when estimates of the pure-error mean square are unavailable owing to lack of replicated data, more approximate methods of testing lack of fit may be used. Here, quadratic terms would be added to the models of Eqs. (32) and (33), the complete model would be fitted to the data, and a residual mean square calculated. Assuming this quadratic model will adequately fit the data (lack of fit unimportant), this quadratic residual mean square may be used in Eq. (68) in place of the pure-error mean square. The lack-of-fit mean square in this equation would be the difference between the linear residual mean square [i.e., using Eqs. (32) and (33)] and the quadratic residual mean square. A model should be rejected only if the ratio is very much greater than the F statistic, however, since these two mean squares are no longer independent. 

F =MSLF/MSPE, based on the ratio mean square for lack of fit (MSLF) over the mean square for pure error (MSPE) ( 31 ). F follows the F distribujfion with (r-2) and (N-r) degrees of freedom. A value of F regression equation. Since the data were manipulated by transforming the amount values jfo obtain linearity, i. e., to achieve the smallest lack of fit F statistic, the significance level of this test is not reliable. 

Formal tests are also available. The ANOVA lack-of-fit test ° capitalizes on the decomposition of the residual sum of squares (RSS) into the sum of squares due to pure error SSs and the sum of squares due to lack of fit SSiof. Replicate measurements at the design points must be available to calculate the statistic. First, the means of the replicates (4=1,. .., m = number of different design points) at all design points are calculated. Next, the squared deviations of all replicates U — j number of replicates] from their respective mean... 

The suitability of the regression model should be proven by a special statistical lack-of-fit-test, which is based on an analysis of variance (ANOVA). Here the residual sum of squares of regression is separated into two components the sum of squares from lack-of-fit (LOF) and the pure error sum of squares (PE, pure errors)... 

The idea of the lack-of-fit test is to compare the pure error of the regression line with the error due to the use of an inappropriate regression model. The MSLOf, which is a measure for the spread of the mean response per concentration from the regression line, is divided by the MSpE, which is a measure for the spread of the instrument response due to experimental variation. The obtained F-value (F = MS[ oii/MS i) is compared with the F-distribution with k 2 and n k d.f. 

The residual sum of squares (SSR) contains contributions of a pure error PE due to pure experimental errors, and a lack-of-fit LF due to the inadequacy of the model. The pure error sum of squares PE can be obtained by, for example nc replicated experiments at a number of, at least one, experimental settings. The relationships that hold are given in eq 52 ... 

In conclusion, despite the indication of the test point 7, going from a quadratic to a reduced cubic model does not improve the model. There is a substantial and statistically significant lack of fit of the model to the data. The probability that the lack of fit is due to random error is less than 0.1 %. Values of the F ratio are therefore calculated using the pure error mean square. 

One reason for the significant lack of fit is the considerable variation (more than 3 orders of magnitude) of the solubility over the domain, while the experimental standard deviation is only 8%. It is not surprising that such a simple relationship as a reduced cubic model is insufficient. Examination of the predicted and experimental data shows that almost all the lack of fit is concentrated in the 3 test points. Since they contribute least to the estimations of the model coefficients, it is these points that would normally show up any deviation from the model. In particular, it is seen that the solubility at point 8, with 66.7% water content, is overestimated by a factor of 2.4. For the other test points the error is 33% or less. This is still very high compared to the pure error. 

However, let us revisit the plot of e, vs. x,- (Figure 3.3). There is reason to suspect that the linear regression model y = bo + biXi is not exact. Recall from Chapter 2 that we discussed both pure error and lack of fit in regression. Most statistical software programs have routines to compute these, or the computations can be done easily with the aid of a hand-held calculator. 

Recall that the sum of squares term (SSe) consists of two components if the test is significant (1) sum of squares pure error (SSpe) and (2) sum of squares lack of fit (SSu). 

Recall that the lack-of-fit test partitions the sum of squares error (SSe) into two components pure error, the actual random error component and lack of fit, a nonrandom component that detects discrepancies in the model. The lack-of-fit computation is a measure of the degree to which the model does not fit or represent the actual data. 

Since there is no lack of fit, both MSiof and MS estimate a. We can take advantage of this fact to obtain a variance estimate with a larger number of degrees of freedom, summing SSiof and SS and dividing the total by (viof + Vpe). With this operation, we again calculate the residual mean square, which now becomes a legitimate estimate of the pure error. 

Various sums of square are used to test models. Figure 2.6.3.1-1 shows the partitioning of the total sum of square into its components. The model adequacy can be tested when the lack of fit sum of squares and the pure error sum of squares are available. The latter can be calculated when replicated experiments have been performed. An estimate of the pure error variance is obtained from... 

The model quality was adequate for all models except for the span model, which did not accomplish the required quality values as mentioned in Table 14.6 and Sect. 2.2.5 The particle size and BET surface area models exhibit an artificial lack of fit of the single measurements due to high reproducibility of experiments dried at the same conditions (centre point results). The resulting small pure error (single measurements) exceeds the model error (centre point results) and reduces the model validity (marked with a in Table 14.6). All models besides the span model give good 2 and measures, which qualifies this model to make valid predictions for further experiments. 

Dividing the sum of squares by the number of degrees of freedom for the respective case yields the, values for the mean squares, MS eg, MSloF, MSpg, MS,ot,corr- The pure error mean square, MSpg, is an estimate of that is valid both when there is a lack of fit in the model and when there isn t. 

Lack of perfection is apparent even in the results for the 1 1 salts. Although the parameters show significant regularity, they do not satisfy additivity relations within the fitting error. Letting q stand for the characteristic ion size or hydration number of an electrolyte, relations such as q(KBr) = q(KCl) -I- q(NaBr) - q(NaCl) should be satisfied. For KBr, we predict values of 7.27 and 1.98 for the ion size and hydration number the fitted values are 7.14 and 1.53. Although this is not too bad, if we attempt to predict the same parameters for CsBr from those of CsCl, NaBr, and NaCl, we obtain 5.25 and 1.75, compared to fitted values of 4.20 and 1.14. One could go further and test the ability of the models for pure aqueous electrolytes to be extrapolated to predict the properties of mixtures, such as the system NaCl-KCl-H20, as no additional fit parameters are required by the model. However, because the additivity relations are not precisely satisfied, there seems little point in doing so. 

---