Activity prediction accuracy

The first benchmark of a QSAR model is usually to determine the accuracy of the fit to the training data. However, because QSAR models are often used for predicting the activity of compounds that have not yet been synthesized, the most important statistical measures are those giving an indication of their prediction accuracy. Common methods to test QSAR predictivity are listed below. 

If new scaffolds are to be found, a second important feature of such models is their completeness. It allows different structural solutions to fulfil the interaction pattern required by the pharmacophore. Accuracy in the activity prediction, on the other hand, is not paramount. 

Al. Abrass, C, Nies, K., Louis, J., Broder, W. A., and Glassock, R. J., Correlation and predictive accuracy of circulating immune complexes with disease activity in patients with systemic lupus erythematosus. Arthritis Rheum. 23, 273-282 (1980). 

At Lilly we have focused on predictive accuracy in most of our project work. Predictive accuracy and interpretability tend to be inversely proportional. An active area of research at Lilly is an investigation of the question of ways in which the model can help us design a better molecule. This may involve interpretation, and there are excellent tools that can be used for this, such as partial dependence plots. It can also be approached through virtual screening—a scientist proposes a scaffold or series and the model provides an evaluation of the prospects of that idea. 

In this chapter we overview some probabilistic methods used for biological activity prediction, paying particular attention to the problems of creation of the training and evaluation sets, validation of (Q)SAR models, estimation of prediction accuracy, interpretation of the prediction results and their application in virtual screening. 

When a classifier that provides the estimation of P(C) is constructed, its performance must be estimated. The most important estimation is of the prediction accuracy. To do this, an evaluation set (test set or validation set - see Section 6.3.2) must be used. The evaluation set (ES) must be relevant and include both type of examples - positive and negative ( active and inactive compounds). For all compounds CeES, estimations P(C) are calculated, and obtained values are analyzed using knowledge about the true classification of compounds in ES. Figure 6.4 shows the main features of this task. 

The latest version of PASS (2007) predicts 3300 kinds of biological activity with a mean prediction accuracy of about 95%. PASS could predict about 1000 kinds of biological activity in 2004, only 541 activities in 1998, and 114 activities in 1996. ... 

The estimations Equations (6.17a, b) of probabilities P Ak), P Ak D not only increase the algorithm s prediction accuracy but also open up new possibilities. For example, function fn A in the range [0,1] can be considered as a measure of molecule n belonging to a fuzzy set of molecules that reveal activity Ak- The descriptor weight gn Dd can be considered in the same manner, and then the molecule structure descriptors can be of arbitrary nature, e.g., such as in the refs. 51 and 52. 

Leave one out cross-validation for 3300 kinds of biological activity and 117 332 substances provides the estimate of PASS prediction accuracy during the training procedure. The average accuracy of prediction is about 94.7% according to the LOO CV estimation, while that for particular kinds of activity varies from 65% [System lupus erythematosus treatment, Immunomodulator (HIV)] to 99.9% (Allergic rhinitis treatment, histone acetylation inducer). The estimated accuracy of prediction for all kinds of biological activity predicted by PASS is presented at the web site. " ... 

Advances in the understanding of the immunobiology of skin sensitization have led to the establishment of predictive in vivo tests which not only identify sensitizing hazards but also characterize their potency. Recently, appreciation of the underlying biology has also resulted in the development of mechanistically based in vitro alternatives which offer the prospect of the replacement of current in vivo methods. Assays under active validation include the Direct Peptide Reactivity Assay (DPRA), the human Cell Line Activation Test (h-CLAT), and KeratinoSens. None of the methods have a sufficient level of accuracy or freedom from applicability domain limitations to allow them to act as a standalone replacement. Consequently, it will be necessary to consider how to deploy these assays, perhaps in combination and/or in a structured assessment of skin sensitization hazard, to ensure at least the same level of predictive accuracy as the in vivo methods. However, a challenge remains the capacity of these methods to provide potency information on skin-sensitizing chemicals has yet to be assessed. This is an essential requirement for future risk assessment without use of animal models if we are to retain the same level of human health protection that is currently delivered. 

Zhu, H., Rusyn, I., Richard, A., and Tropsha, A. (2008). Use of cell viabihty assay data improves the prediction accuracy of conventional quantitative structure-activity relationship models of animal cardnogenicity. Environ Health Perspect 116, 506-513. 

CNS activity is a complex process, and remains far from fuUy understood. It is difficult to interpret why certain fragments appear to influence CNS activity. Several compounds with a protonated tertiary amine can pass the blood-brain barrier, although they are not CNS-active. Tertiary amines have a pKa around 7 and can easily be protonated. A comparison of five different approaches, including Bayesian neural network and several other methods described here, revealed that substructural analysis gained so far the best prediction accuracy [38]. 

Molina, C.A., et al.. Improving the predictive accuracy of recanalization on stroke outcome in patients treated with tissue plasminogen activator. Stroke, 2004. 35(1) p. 151-6. 

Downsizing, whereby the majority class from the training set is sampled to produce a balanced training set of actives and inactives, was found to worsen the predictive accuracy as it reduces the effective sample size considerably when... 

Probabilistic QSAR models, logistic PLS and local similarity assessment, SVM and random forest N = 907 compounds from the literature CYP3A4 IC50. <40 j,M = active IC50 > 60 pM = inactive. PubChem data on N = 11,060 molecules. Fragment descriptors were used. As reliability index increases, prediction accuracy increases but fewer compounds belong to the applicability domain. Prediction accuracy may be 93% but only 41 % of the PubChem dataset is used. 206... 

A so-called confusion or contingency matrix is generally calculated for categorical classifications, where one models presence or absence of activity (see Box 23.2). This provides a measure of the prediction accuracy for known active compounds (sensitivity), non-active compounds (specificity), and overall predictivity (concordance or accuracy). To... 

Structure-activity considerations. Predictions of the adverse effects that a substance may have can sometimes be made on the basis of a knowledge of the chemical structure of the substance and an association of various features of that structure with particular types of toxicity. It must be emphasised that this is a field for the expert and uninformed structure-activity predictions can be dangerously misleading. There are some areas, however, in which experts may make such predictions with a fair degree of accuracy. Perhaps the simplest example would be the prediction of irritant or corrosive properties for a molecule likely to give rise to acidic conditions in contact with water. Predictions of mutagenic activity may also in some cases be made by experts on the basis of the potential of the molecule to interact with DNA. 

Upon generalization of the potential approach for adsorption equilibria of gas mixtures, Rudzinski et al. [39] pointed out that the potential theory approach corresponds to a special case of lAST for adsorption on a heterogeneous surface and therefore it works better on strongly heterogeneous adsorbents such as activated carbon. Based on thermodynamic analysis, Rudzinski et al. also suggested that an extra coalescing factor be introduced in Eq. (35) and in this way the predictive accuracy of the potential theory approach could be further improved. 

---