Predictive Accuracies of Models

H. van der Voet, Comparing the predictive accuracy of models using a simple randomization test. Chemom. Intell. Lab. Syst., 25 (1994) 313-323. 

Analysis of in-house data in pharma companies frequently demonstrates a low prediction ability of current models for both lipophilicity and aqueous solubility. The calculated errors of these models are often around or higher than 1 log-unit, which is not sufficient for screening purposes. Thus, despite relatively large amounts of data for physicochemical properties and their simplicity compared to more complex ADME/T properties, the accuracy of prediction remains low. Let us consider factors that limit the prediction accuracy of models. 

However, there is still a strong need to develop new methods that will be able to quantitatively or at least qualitatively estimate the prediction accuracy of log D models. Such models will allow the computational chemist to distinguish reliable versus nonreliable predictions and to decide whether the available model is sufficiently accurate or whether experimental measurements should be provided. For example, when applying ALOGPS in the LIB RARY model it was possible to predict more than 50% and 30% compounds with an accuracy of MAE <0.35 for Pfizer and AstraZeneca collections, respectively [117]. This precision approximately corresponds to the experimental accuracy, s=0.4, of potentiometric lipophilicity determinations [15], Thus, depending on the required precision, one could skip experimental measurements for some of the accurately predicted compounds. 

A low accuracy of models for prediction of log D at any pH would not encourage the use of these models for practical applications in industry. Thus, it is likely that the methods for log D prediction at fixed pH that are developed in house by pharmaceutical companies will dominate in industry. However, log D measurements... 

Only a few models applicable to paddy field conditions have been developed. RICEWQ by Williams, PADDY by Inao and Kitamura," and PCPF-1 by Watanabe and Takagi are useful for paddy fields. EXAMS2 by the United States Environmental Protection Agency (USEPA), a surface water model, can also be used to simulate paddy fields with an appropriate model scenario and has been used for the prediction of sulfonylurea herbicide behavior in paddy fields. The prediction accuracy of PADDY and PCPF-1 is high, although these models require less parameter... 

A prediction accuracy of 10 to 30% is claimed. D. Models of Kumar, Kuloor, and Co-workers... 

Although junk science has no rigorous definition, it is characterized by one or both of two properties (1) data that do not meet the normal criteria for being unbiased and objective, and (2) inappropriate or incomplete representations of tests of the predictive accuracy of of models that create a false impression of reliability. 

Using a computer-aided model for the prediction of pseu-doallergic reactions from prospective data collected from 581 patients in a controlled clinical trial with an outdated formulation of polygeline, accurate prediction of 86% of the patients who had a systemic reaction was possible (9). The data were handled by multivariate analysis using the independence Bayes model. The predictive accuracy of other reactions was poor. A history of allergy was recorded in 26% of the patients who had systemic reactions and in 12 and 13% of the patients with no systemic or skin reactions. However, these differences were not statistically significant. 

Applicability Domain for DT-Based Models We describe applicability domain for QSAR models as being determined by two parameters (1) prediction confidence, or the certainty of a prediction for an unknown chemical, and (2) domain extrapolation, or the prediction accuracy of an unknown chemical that lies beyond the chemical space of the training set [60]. Both parameters can be quantitatively estimated in the consensus tree approaches, where individual models are constructed as DTs. Taken together, prediction confidence and domain extrapolation assess the applicability domain of a model for each prediction. 

Several methods considered in this article, such as the quantum-chemical approaches developed by Clark et al. [51,113,114] or the ALOGPS program [99], provide an estimation of the accuracy of model predictions. Other approaches to estimate accuracy of model predictions were reviewed elsewhere [99,141],... 

---