ROC curve

The Role of Receiver Operating Characteristic (ROC) Curves in the Evaluation... 

A new tumor marker is evaluated using the same criteria used for many diagnostic tests (i.e., sensitivity, specificity, and accuracy). The diagnostic sensitivity and specificity are best represented by a receiver operating characteristic (ROC) curve. The ROC curve is constructed with the true-positive rate versus false-positive rate at various decision levels. As a test improves in its diagnostic performance, it shifts upward and to the left as the true-positive rate increases and the false-positive rate decreases. 

An ROC curve plots the true positives against the false positives for different classifications of the same set of objects this corresponds to plotting a against n - a using the notation of Table 1, and thus the shape of an ROC curve tends to the shape of a cumulative recall plot when n a. An example of the use of ROC plots in chemoinformatics is provided by the work of Cuissart et al. on similarity-based methods for the prediction of biodegradability (24). 

Figure 11.8 illustrates the relationship between linearity plots of intensity (and intensity variation) vs. concentration to ROC plots or what are more often called ROC curves. Basically, what is desired is for the distribution of signal level at a required concentration to be completely separated from the distribution of signal level for the background. ROC curves can be generated from a series of measurements at one concentration or a series of concentrations as shown in Figure 11.8. 

The ability to discriminate target signal from background constitutes specificity. Figure 11.9 illustrates by way of ROC curves the characteristics of Pd vs. Pfp for prototypical high- and low-specificity detectors. 

Figure 11.8 Basis for taking signal data at different concentrations to generate ROC curves for showing the Pq and Ppp dependence.

Figure 11.9 Illustrative example of ROC curves for a screening system with high and low specificity.

Fig. 7.2 Event-free survival according to levels of circulating CD34 KDR -EPCs defined by ROC curve analysis. Reprinted from [46]...

Heagerty PJ, Lumley T, Pepe MS. Time-dependent ROC curves for eensored survival data and a diagnostie marker. Biometrics 2000 56 337-344. 

Heagerty PJ, Zheng Y. Survival model predietive accuracy and ROC curves. Biometrics 2005 61 92-105. 

Fig. 8.12. A Conventional H E stained images and their classified counterpart (right) can be compared pixel by pixel for accuracy. B Selected regions showing correspondence between H E and class images. C Accuracy is best assessed by ROC curves shown for individual classes with colors as per legend in (A). D The number of spectral features and classification accuracy for training data (red) and validation data (blue) show that high accuracy is possible. The effects of over-training with a feature set larger than 45 can be seen...

Fig. 11.8. A binormal receiver operating characteristic (ROC) curve (solid line) plotting the sensitivity versus 1-specificity using the 850 nm OCT attenuation coefficient to distinguish between eariy caries and heaithy enamel. The dashed line represents the ROC curve for a nondiscriminatory test. The markers on the curve represent setting an OCT attenuation coefficient threshold between caries and healthy enamel at 0.9 rnrri 1 (diamond) and at 1.08 rrirri 1 (star)...

Enrichment based methods. These focus on recovering active molecules from a test database in which a small number of known actives have been hidden in a large database of randomly selected compounds. Database mining and the utilization of receiver operating characteristic (ROC) curves [43] can be included in this category. 

This maximizes Se (Se l). However, specificity will be minimized as most of the inactive molecules will be in the selection. The situation is reversed in the case where n is very small (n 0). Consequently, one cannot optimize both Se and Sp at the same time and a trade-off is to be determined. If one can choose to rely on one of the above metrics to find an optimum, we have recently advocated the use of a simpler graphical technique which has been adopted as a gold standard in many other research areas the receiver operating characteristic (ROC) curve method [53]. 

Fig. 15.4 Performance assessment with ROC curves. The theoretical distributions for active (red curve) and inactive compounds (blue) as a function of their fit score on the pharmacophore (left). In most cases, these distributions overlap, leading to false predictions (colored areas). Upon threshold modification, proportions of such erroneous classifications change dramatically. Hence to any selection threshold S corresponds a unique...

Fig. 15.5 Decision making from ROC curves. Different selection thresholds (S1-S3) correspond to different points on the ROC curve and allow one to tune S according to different strategies in drug discovery and different stages of R. D.

Apart from expressing SAR, there is no validation method that is particularly recommended for this use. Of course, the selectivity of the pharmacophore will definitely facilitate library design and a possible way to assess it is to screen a database of molecules flagged as active and inactive. In this respect, this is rather similar to the virtual screening usage and the ROC curve approach could be used... 

Fig. 15.6 (b) Activities of the mGlu4 agonists reported with L-glutamate (l-GIu) as a reference. Molecules used in the training set for Catalyst-HypoGen are flagged with an x . Those taken to plot the ROC curves are flagged either as actives (A) and inactives (I). 

In addition to this pharmacophore hypothesis, although it met only three of the four criteria, model 1 from run 6 was retained. Surprisingly, despite criterion number 2 not being satisfied (RMS= 1.62, r=0.79), this model exhibits a remarkable ability to discriminate between active and inactive compounds as assessed by the ROC curve, AUC=0.95. In contrast, model 1 from run 8 has good statistics (RMS=0.76, r= 0.96) but a lower AUC of 0.87. This illustrates that a good model for activity prediction may not be the best for virtual screening applications. Let us analyze these two pharmacophore hypotheses further. 

In a second row, when the AUCs of the two retained pharmacophore models are compared, hypothesis 1 appears to perform better than hypothesis 2. Not only is its AUC larger when compared with hypothesis 2, but also its ROC curve shape is more interesting. Hence, if sensitivity is to be maximized to reduce the false negative rate and increase the chances of finding novel leads (Se=l), it is possible to increase the specificity to 0.73 with hypothesis 1 (point Fig. 15.11), whereas the highest possible specificity value for Se=l is only 0.36 with hypothesis 2 (point S2, Fig. 15.11). In other words, if we analyze the... 

Fig. 15.11 Statistical performances (above) and ROC curves (below) of hypothesis 1 (left) and hypothesis 2 (right). The 17 compounds used for the statistical assessment are those from the training set whereas the 21 compounds used to plot the ROC curves...

In conclusion, hypothesis 1 was selected because it satisfies three of our four validation criteria and seems particularly appropriate for virtual screening (ROC curve validation). The selection threshold was set according to point Sx (Fig. 15.11), corresponding to an activity threshold of 24 (i.e. estimated activity below 24 times the activity of l-G1u). 

---