Area under the ROC curve

If tens or hundreds of models have to be validated, a simple metric providing a value can be more comfortable for enrichment comparison than the visual analysis of tens or hundreds of ROC curves. For this purpose, a metric also derived from the area under the ROC curve (AUC) exists. The AUC value can be obtained by simply adding the areas of rectangles formed by Se and 1-Sp for the ith active molecule (5.3) [65]. 

A pharmacophore model can obtain an AUC value between 0 (ranking all inactive molecules first) and 1 (ranking all actives first). An AUC value of 0.5 refers to a 

On that account, the ROC curve is more useful than the AUC if the best model for addressing the early recognition problem has to be selected. 

To sum up, a variety of enrichment metrics for evaluating VS methods are available. Among these enrichment metrics, the EF and the AUC value, as well as visualizations of ROC curves, represent popular methods for assessing the enrichment of active molecules by pharmacophore models [83-85]. The two disadvantages of many of these enrichment metrics are the lack of information about early enrichment of active molecules and the missing comparability between validation 

---

In any case, this approach uses several additional assumptions. For this reason in the last time in ML the recognition accuracy criterion of the Area Under the ROC Curve AUC), which is free of additional assumptions, becomes very popular. Mathematically, AUC equals the... 

It has been advocated that the area under the ROC curve is a relative measure of a tesfs performance. A Wilcoxon statistic (or equivalently the Mann-Whitney U-Test) statists cally determines which ROC curve has more area under it. Less computationally intensive alternatives, which are no longer necessary, have been described. These methods are particularly helpful when the curves do not intersect. When the ROC curves of two laboratory tests for the same disease intersect, they may offer quite different performances even though the areas under their curves are identical. The performance depends on the region of the curve (i.e., high sensitivity versus high specificity) chosen. Details on how to compare statistically individual points on two curves have been developed elsewhere. ... 

Figure 5.4 The AUC value represents the area under the ROC curve and is limited by 0 and 1. Since pharmacophore-based screening mostly results in the selection of a limited number of database molecules that are subjected to...

Fig. 6 (a) A theoretical distribution of compounds in a virtual screen based upon the docking score. The overlap between active and inactive compounds indicates that the scoring threshold used to identify a hit by virtual screening is critical, (b) A ROC curve is used to evaluate the enrichment of a virtual screen and select a scoring threshold. A ROC curve that approaches Se = 1 and 1-Sp = 0 represents perfect enrichment. The area under the ROC curve (AUC) represents the probability that a true active is identified. (Reprinted with permission from [131], copyright 2008 by Springer)... 

A given study provides an estimate of the ROC curve for that test and patient population. The confidence limits around the ROC curve can be calculated (8,9). Furthermore, the area under the ROC curve can be calculated for each test so as to derive a quantitative index of the test s individual accuracy and its relation to the other tests being evaluated (8,9). 

Fig. 6.6. The graph on the left shows two ROC curves that have equal area under the curve (AUC), but the curves cross. One system is superior if high sensitivity is desirable, while the other system would be preferred for high specificity. On the right is an illustration of partial area under the ROC (pAUC), where it is assumed that high sensitivity is desirable. The shaded area divided by the area above the 0.9 value line is djAUC, the area under the ROC curve above a sensitivtty of 90%. When high sensitivity is desired, ,AUC shows the advantage for the blue curve, when compared with using AUC for the whole curve...

For the entire cohort, area under the ROC curve (AUC) using the 7-point scale was 0.78 for digital and 0.74 for film. This ference was not statistically significant Sensitivity, based on the 5-point Bl-RADS scale, was 0.70 for digital mammography and 0.66 for screen-film, also not a statistically significant difference. Figure 9.2a shows the ROC curves for the entire cohort Specificity was the same for each modality at 0.92 positive predictive value for each was 0.05. 

Separate case sets and groups of readers were used for each machine type. A study was also performed using the Lorad/Hologic machine, but the results were not reported due to an insufficient number of cases. There was a nonsignificant trend toward superiority of screen-film in area under the ROC curve (AUC), sensitivity, and specificity (not shown) for all three manufacturers reported (Hendrick et al. 2008)... 

When injnry scores are dichotomized into positive and nonpositive, sensitivity and specificity are usually estimated and displayed in an ROC curve (Pepe, 2003). The area under the ROC curve (AUROC) is a common summary performance metric. Some advantages of this approach are that ROC curve estimation generally requires few assumptions, a plot of the ROC curve gives an easily interpreted visual display of results, and the AUROC is a global metric that does not require the specification of a threshold. Since both sensitivity and specificity can be greatly influenced by characteristics that are study specific such as severity or type of injury, it is advisable to compare candidate marker performance to that of standard markers. In rodent nephrotoxicity studies, candidate markers are commonly compared to sCr and BUN. 

Pencina MS, D Agostino RB Sr., D Agostino RB Jr., Vasan RS (2008). Evaluating the added predictive ability of a new marker from area under the ROC curve to reclassification and beyond. Statistics in Medicine, Th 157-172. 

The result gives a set of optimized parameters, performance measure in terms of accuracy, precision, recall and area under the roc curve (AUC). 

TABLE 5.3 Enrichments (at 1%) and AUC (Area Under the ROC Curve) for the Muchmore-Martin Set of Lead-Hops... 

Urinary tract In a retrospective study of a random cohort of 171 patients, of whom 53 developed acute renal insufficiency and 118 did not, logistic multivariate regression cmalysis showed that the cumulative dose of torasemide was a susceptibihty factor (OR = 1.02 95% CI = 1.002, 1.03 area under the ROC curve = 0.632) [18 ]. 

Fig. 27.6. Example of the gain in the human reader performance obtained by use of CAD (Okamura et al. 2004). Regardless of the different levels of reading skill, the detection performance, measured by the area under the ROC curve (vertical axis), increased for all of the readers when they used CAD. Among the four observers, the increase in performance was the largest for the gastroenterologist...

When these coefficients were applied to MLP for classification, the results showed an increase of the classification rate from 87% as in [9] to 96%. The average false positive, true positive and area under the ROC curve was 0.2%, 96% and 99.84% respectively which indicates that MLP is a classifier of high precision. The results are summarised as in Table 1. 

---