Errors and Goodness of Fit

It is often the case in chemical analysis that the independent variable, standard solution concentrations in the above example, is said to be fixed and free of error. The concentration values for the calibration solutions are chosen by the analyst and assumed to be accurately known. The errors associated with x, therefore, are considered negligible compared with the uncertainty in y due to fluctuations and noise in the instrumental measurement. 

To use Equations 6.6 and 6.7 to determine the characteristics of the fitted line, and employ this information for prediction, it is necessary to estimate the uncertainty in the calculated values for the slope, b, and intercept, a. Each of the absorbance values, has been used in the determination of a and b and each has contributed its own uncertainty or error to the final result. 

Estimates of error in the fitted line and estimates of confidence intervals may be made if three assumptions are valid  

The deviation or residual for each of the absorbance values in the nickel data is given by yj — i.e. the observed values minus the calculated or predicted values according to the linear model. The sum of the squares of these deviations. Table 6.2, is the residual sum of squares, and is denoted as SS, . The least squares estimate of the line can be shown to provide the best possible fit and no other line can be fitted that will produce a smaller sum of squares. 

The variance associated with these deviations will be given by this sum of squares divided by the number of degrees of freedom, 

---

Table 2 Errors and goodness of fit calculations associated with the linear regression model for nickel AAS data from Table 1...

The Michaehs-Menten model was fitted to the experimental data using standard nonlinear regression techniques to obtain estimates of K (Fig. 4.2). Best-fit values of and K, corresponding standard errors of the estimates plus the number of values used in the calculation of the standard error, and goodness-of-fit statistic are reported in Table 4.5. 

Different tests for estimation the accuracy of fit and prediction capability of the retention models were investigated in this work. Distribution of the residuals with taking into account their statistical weights chai acterizes the goodness of fit. For the application of statistical weights the scedastic functions of retention factor were constmcted. Was established that random errors of the retention factor k ai e distributed normally that permits to use the statistical criteria for prediction capability and goodness of fit correctly. 

Critical validation of the PLS models is essential. Explained variance (R X) and goodness of fit (R Y) are important parameters but not sufficient. Goodness of prediction (root mean square error [RMSE] or Q ) obtained after cross-validation [22] is essential to avoid overfit and correlations that are simply due to chance. A final validation should be performed by distinguishing between a training set and a test set. The training set is used to create a PLS model, which is subsequently used to predict values obtained from an independent test set. Furthermore, repeatability and reproducibility should be evaluated using repeated analyses and parallel samples. 

A FfTEQL modification allowing for pointwise introduction of error estimates, which has been tested on some of the datasets given in Figs. 6, 7, and 8, for which extensive experimental work yielded reliable error estimates, numerical goodness-of-fit estimates were found to be appropriate [36]. In general, such extensive data are not available and goodness of fit must be checked graphically. 

Equation 7 provides the formal error bars on estimates of the relative travel time and, through Eq. 4, velocity changes. However, in addition to formal error bars, goodness-of-fit criteria must be taken into accotmt as well (Press et al. 1986). In the implementation of CWI at Okmok, two goodness-of-fit criteria are required to be met in order to accept an estimate of relative travel time, irrespective of the error estimate in Eq. 7. It must be that either (a) the standard deviation in Eq. 6 is less than one time sample or (b) the Pearson linear correlation coefficient given by... 

In many process-design calculations it is not necessary to fit the data to within the experimental uncertainty. Here, economics dictates that a minimum number of adjustable parameters be fitted to scarce data with the best accuracy possible. This compromise between "goodness of fit" and number of parameters requires some method of discriminating between models. One way is to compare the uncertainties in the calculated parameters. An alternative method consists of examination of the residuals for trends and excessive errors when plotted versus other system variables (Draper and Smith, 1966). A more useful quantity for comparison is obtained from the sum of the weighted squared residuals given by Equation (1). 

This gives some information about the errors (i.e., the variance and standard deviation of each data point), although the goodness of fit, P, cannot be calculated. 

A good model is consistent with physical phenomena (i.e., 01 has a physically plausible form) and reduces crresidual to experimental error using as few adjustable parameters as possible. There is a philosophical principle known as Occam s razor that is particularly appropriate to statistical data analysis when two theories can explain the data, the simpler theory is preferred. In complex reactions, particularly heterogeneous reactions, several models may fit the data equally well. As seen in Section 5.1 on the various forms of Arrhenius temperature dependence, it is usually impossible to distinguish between mechanisms based on goodness of fit. The choice of the simplest form of Arrhenius behavior (m = 0) is based on Occam s razor. 

The weighting model with which the goodness-of-fit or figure-of-merit (GOF = E(m,)) is arrived at can take any of a number of forms. These continuous functions can be further modified to restrict the individual contributions M, to a certain range, for instance r, is minimally equal to the expected experimental error, and all residuals larger than a given number r ax are set equal to rmax- The transformed residuals are then weighted and summed over all points to obtain the GOF. (See Table 3.5.)... 

Frequency domain performance has been analyzed with goodness-of-fit tests such as the Chi-square, Kolmogorov-Smirnov, and Wilcoxon Rank Sum tests. The studies by Young and Alward (14) and Hartigan et. al. (J 3) demonstrate the use of these tests for pesticide runoff and large-scale river basin modeling efforts, respectively, in conjunction with the paired-data tests. James and Burges ( 1 6 ) discuss the use of the above statistics and some additional tests in both the calibration and verification phases of model validation. They also discuss methods of data analysis for detection of errors this last topic needs additional research in order to consider uncertainties in the data which provide both the model input and the output to which model predictions are compared. 

The attached worksheet from MathCad ( 1986-2001 MathSoft Engineering Education, Inc., 101 Main Street Cambridge, MA 02142-1521) is used for computing the statistical parameters and graphics discussed in Chapters 58 through 61, in references [b-l-b-4]. It is recommended that the statistics incorporated into this series of Worksheets be used for evaluations of goodness of fit statistics such as the correlation coefficient, the coefficient of determination, the standard error of estimate and the useful range of calibration standards used in method development. If you would like this Worksheet sent to you, please request this by e-mail from the authors. 

Based on previous recommendations [31], a combination of graphical techniques and error index statistics was used for evaluating the goodness-of-fit between the simulated and observed streamflow values, both during the calibration and validation period. The used statistics were the mean error (ME), the percent bias (PBIAS, [32]) and the Nash-Sutcliffe efficiency (NSeff, [33]) ... 

Figure 1. Top portion shows a plot of the observed Fenvalerate response vs. the mass (ng). Lower plot gives ordered, normalized residuals from the fit of model-3 to the data (Table IV) using the weights given in Table III. (Symbols indicate the five replicates, and the plotted residuals are normalized by the standard deviations for these individual replicates. The "goodness of fit residuals of the model to the means of the replicates are larger by 1/5T because they are normalized by the standard errors at each concentration.)...

In Eq. 13.15, the squared standard deviations (variances) act as weights of the squared residuals. The standard deviations of the measurements are usually not known, and therefore an arbitrary choice is necessary. It should be stressed that this choice may have a large influence of the final best set of parameters. The scheme for appropriate weighting and, if appropriate, transformation of data (for example logarithmic transformation to fulfil the requirement of homoscedastic variance) should be based on reasonable assumptions with respect to the error distribution in the data, for example as obtained during validation of the plasma concentration assay. The choice should be checked afterwards, according to the procedures for the evaluation of goodness-of-fit (Section 13.2.8.5). 

The central-limit theorem (Section III.B) suggests that when a measurement is subject to many simultaneous error processes, the composite error is often additive and Gaussian distributed with zero mean. In this case, the least-squares criterion is an appropriate measure of goodness of fit. The least-squares criterion is even appropriate in many cases where the error is not Gaussian distributed (Kendall and Stuart, 1961). We may thus construct an objective function that can be minimized to obtain a best estimate. Suppose that our data i(x) represent the measurements of a spectral segment containing spectral-line components that are specified by the N parameters... 

---