Confidence intervals, predictive model

The Gaussian Process could be also used to estimate prediction accuracy based on the variance of different models derived within this approach. The usefulness of this approach for confidence intervals prediction of aqueous solubility and lipophilicity was shown. ... 

Inference is the act of drawing conclusions from a model, be it making a prediction about a concentration at a particular time, such as the maximal concentration at the end of an infusion, or the average of some pharmacokinetic parameter, like clearance. These inferences are referred to as point estimates because they are estimates of the true value. Since these estimates are not known with certainty they have some error associated with them. For this reason confidence intervals, prediction intervals, or simply the error of the point estimate are included to show what the degree of precision was in the estimation. With models that are developed iteratively until some final optimal model is developed, the estimate of the error associated with inference is conditional on the final model. When inferences from a model are drawn, modelers typically act as though this were the true model. However, because the final model is uncertain (there may be other equally valid models, just this particular one was chosen) all point estimates error predicated on the final model will be underestimated (Chatfield, 1995). As such, the confidence interval or prediction interval around some estimate will be overly optimistic, as will the standard error of all parameters in a model. 

Figure 22.3 The drug dose-response model was augmented by nsing data for the comparator drug. Because the mechanism of the drugs was the same, this comprised additional data for the model. This enhanced the predictive power of the model, in a better estimate for central tendency (solid line compared with dotted line) bnt also in smaller confidence intervals. This is especially prononnced at the higher doses— precisely where data on the drug were sparse. See color plate.

A valuable inference that can be made to infer the quality of the model predictions is the (l-a)I00% confidence interval of the predicted mean response at x0. It should be noted that the predicted mean response of the linear regression model at x0 is y0 = F(x0)k or simply y0 = X0k. Although the error term e0 is not included, there is some uncertainty in the predicted mean response due to the uncertainty in k. Under the usual assumptions of normality and independence, the covariance matrix of the predicted mean response is given by... 

Fig. 6.3. Three-dimensional model of calibration, analytical evaluation and recovery spatial model (A) the three relevant planes are given separately in (B) as the calibration function with confidence interval, in (C) as the recovery function with confidence interval, and in (C) as the evaluation function with prediction interval (D)... 

Precision. The precision of the calibration is characterized by the confidence interval cnffyf of the estimated y values at position x, according to Eq. (6.30). In contrast, the precision of analysis is expressed by the prediction intervals prd(y ) and prd(x,), respectively, according to Eqs. (6.32) and (6.33). The precision of analytical results on the basis of experimental calibration is closely related to the adequacy of the calibration model. 

Furthermore, there are two other aspects to the extrapolation problem one structural and one statistical. An illustrative example of these various cases can be found in a dataset of benzamides (S16.1). that one of the present authors (U.N.) published some time ago [44]. If one develops a PLS model based on the same descriptors and the same, experimental design-based, training set (compounds 1-16) augmented by compound 17 (Table 16.8) in order to prove the points raised above [the prediction limit (1.502) set to two times the overall RSD of the model (0.751) which roughly gives 95% confidence interval], one can observe the following with respect to predictions on the remaining test set compounds ... 

All in vivo data, including the human and rat absorption data used by both Egan and Zhao et al., have considerable variability. Zhao et al. comment that measurements of percent absorbed for the same molecule may vary by 30%, and that the 95% confidence interval for a prediction is approximately 30% given a model RMSE of 15%. This is approximately the same as the normal experimental error for absorption values. This means that models predicting percent absorbed have to be carefully interpreted, i.e., a prediction of 30% absorbed really means the molecule is predicted to have absorption from 15 to 45%, and that classification models should work nearly as well as regression models. 

Calculate the 99% confidence interval for predicting a single new value of response at pH = 7.0 for the data of Equation 11.16 and the second-order model of Equation 11.39. Calculate the 99% confidence interval for predicting the mean of seven new values of response for these conditions. Calculate the 99% confidence interval for predicting the true mean for these conditions. What confidence interval would be used if it were necessary to predict the true mean at several points in factor space ... 

The parameter estimation approach is important in judging the reliability and accuracy of the model. If the confidence intervals for a set of estimated parameters are given and their magnitude is equal to that of the parameters, the reliability one would place in the model s prediction would be low. However, if the parameters are identified with high precision (i.e., small confidence intervals) one would tend to trust the model s predictions. The nonlinear optimization approach to parameter estimation allows the confidence interval for the estimated parameter to be approximated. It is thereby possible to evaluate if a parameter is identifiable from a particular set of measurements and with how much reliability. 

The performance of the model was satisfactory when tested by comparing predictions with observed concentrations of a range of organochlorine chemicals in organisms of the Lake Ontario food-chain (Gobas, 1993). The 95% confidence intervals of the ratio of observed and predicted concentrations of persistent organic chemicals is a factor of 2 to 3. 

All predictions should lie within a 95% confidence interval of the model, meaning all estimates of LCj0 should fall within 0.704 log units of the regression line (0.704 = 1.95 X s.e.). 

Prediction of an unknown sample is the primary motivation for developing a calibration model and is easily accomplished by use of Equation 5.13. Often, statisticians refer to this as forecasting. The 100%(1 - a) confidence interval for prediction at x0 is given by... 

In this section we will look at the confidence interval of the predicted value, y. As we know, the goal of regression analysis is to build a model that minimizes... 

The BMC approach can provide a more refined assessment of the prediction of the empirical NOAEL. It must be emphasized that even the empirical NOAEL may represent a response level that is not detected. When 5 to 10 animals are used in an experiment, a 10-20% response can be missed (Leisenring and Ryan 1992) and even a BMCio is similar to a LOAEL with dichotomized data (Gaylor 1996). It is expected that the BMC is less than the empirical LOAEL. In the Fowles et al. (1999) analysis of the data, the BMCqs and BMCoi values were always below the empirical LOAEL for the studies analyzed. The probit analysis of the data by Fowles et al. (1999) provided a better fit with the data as measured by the chi-squared goodness-of-fit test, mean width of confidence intervals, and number of data sets amenable to analysis by the model. ... 

The coefficients A, B, and C are fitted using a least-squares approach. Results from this approach are given in Table 6. Each data point for the occluded moisture at a given time is a mean value of at least six assays from samples sterilized and dried in different locations within the autoclave. The model predicts mean moisture values within the 95% confidence interval of the measured mean moisture values. Using the data in Table 6 we selected 12 h as the time required to adequately dry the stoppers, since 16 h did not offer a significant decrease in moisture over the 12-h results. 

In the second part of the validation we used the predicted values from the model at 12 h for the acceptance criterion for three reproducibility runs. The model predictions were used since they take into account all of the data generated in the first part of the validation. The acceptance criterion for the mean level of stopper moisture was the 99% confidence interval determined from the model. The calculated 99% confidence interval for the mean value after 12 h of drying was 963 213 ig/stopper. Results from validation runs in Table 7 show that the modeling approach accurately predicted the mean moisture content of the stoppers. The moisture content of the stoppers was consistent from run to run as can be seen in the overlapping 95% confidence intervals for the mean values. 

Use a Monte Carlo simulation to identify the frequency-dependent confidence interval for the model prediction. 

---