Confidence intervals, predictive model comparisons

In this final section, we will consider the comparison of two predictive models. The cheminformatics literature is replete with papers comparing predictive models. When developing a new method, it is always important to examine how the method compares with the current state of the art. However, when making comparisons, one must remember that correlations have an associated error. This error is a function of both the correlation coefficient and the number of data points used to obtain the correlation coefficient. When comparing correlation coefficients, we must not only consider the value of the correlation coefficient, but also the confidence intervals around the correlation coefficient. When we have a larger number of data points or a higher correlation coefficient, we are more confident in the correlation and our confidence interval is relatively narrow. When we have a smaller number of data points or our correlation coefficient is lower, the confidence interval around the correlation is larger. If the confidence intervals of two correlations overlap, we cannot claim that one predictive model is superior to another. 

We will now consider the case of two predictive models. Model A and Model B, for aqueous solubility. We will use R to compare the performance of these models when tested on 25, 50, and 100 compounds. Listing 9 provides an example of how this comparison can be performed in R. In this listing, we first calculate the Pearson r and the upper and lower 95% confidence intervals for the Pearson r. Table 1.6 and Figure 1.5 show the correlations and associated bar plots. The bar plots show the value of Pearson r for each subset and the associated whiskers show the upper and lower limits of the 95% confidence interval. 

The auto- and cross-correlation plots are shown in Fig. 6.12. A comparison between the predicted and actual levels is shown in Fig. 6.13. Both figures use the validation data set for testing the model. From Fig. 6.12, it is clear that the residuals are not uncorrelated with each other or the inputs. Therefore, the initial model needs to be improved. Since there is a suggestion that the process model is incorrectly specified, it will first be changed. The best approach is to increase the order of the numerator and denominator (of the B- and F-polynomials) until either the cross-correlation plot shows the desired behaviour or the confidence intervals for the parameters cover zero. If the second case is reached, then this could be a suggestion that a linear model is insufficient/inappropriate for the given data set. Furthermore, the fit between the predicted and measured levels is not great (55.4%). 

For an upper dwell temperature of 1020°C and upper dwell time of 16 h, the predicted time to spall is 95.1 h with a 95% confidence interval of 61 to 129 h. The experimentally determined values for Alloy 800 at 1000°C and 1050°C were 96 and 32 for 8 h upper dwell time and 160 and 40 for 20 h upper dwell time for comparison. Figure 18.11 shows the observed times to spall plotted against the times predicted by the model. Figure 18.11 shows a good fit between the observed values and the model because the points fall approximately on a straight line. The points for trials 4 and 9 are slightly off the line but there is no obvious connection between trials 4 and 9 to suggest a trend. 

---