False positives/negatives validation

Figure 18.2 Representative receiver operator curves to demonstrate the leave n out validation of K-PLS classification models (metabolite formed or not formed) derived with approximately 300 molecules and over 60 descriptors. The diagonal line represents random. The horizontal axis represents the percentage of false positives and the vertical axis the percentage of false negatives in each case. a. Al-dealkylation. b. O-dealkylation. c. Aromatic hydroxylation. d. Aliphatic hydroxylation. e. O-glucuronidation. f. O-sulfation. Data generated in collaboration with Dr. Mark Embrechts (Rensselaer Polytechnic Institute).

Assessment and definition of sensitivity are often described for quantitative analysis but are of equal importance for qualitative devices of the dip-stick type that are very popular for farm- or field-based screening assays. Because of the somewhat subjective nature of visually assessed assays, the assay s sensitivity must be validated using a number of observers to determine at what level a test is deemed positive. The number of false positives and false negatives must be carefully determined in order to balance consumer safety and potential economic loss to animal producers. 

Nevertheless, caution must be taken to avoid any false IHC negative or false positive results after AR treatment. It should be emphasized that additional biochemical assays other than IHC need to be adopted in order to validate the AR-IHC results whenever necessary. Otherwise, as pointed out by Wick and Mills, there is a real risk that artifacts may become facts. 25... 

The pioneers of bioavailability modeling can be traced back to year 2000. Andrews and coworkers [57] developed a regression model to predict bioavailability for 591 compounds. Compared to the Lipinski s "Rule of Five," the false negative predictions were reduced from 5% to 3%, while the false positive predictions decreased from 78% to 53%. The model could achieve a relatively good correlation (r2 = 0.71) for the training set. But when 80/20 cross-validation was applied, the correlation was decreased to q1 — 0.58. 

Once the assay is validated by these two sets of compounds, it can be used to test a larger group of marketed compounds, which will reveal the performance of the assay with an unbiased set of diverse, already well characterized compounds. False positive and false negative rates can be often defined with this validation. 

Limit of detection (LOD) sounds like a term that is easily defined and measured. It presumably is the smallest concentration of analyte that can be determined to be actually present, even if the quantification has large uncertainty. The problem is the need to balance false positives (concluding the analyte is present, when it is not) and false negatives (concluding the analyte is absent, when it is really present). The International Union of Pure and Applied Chemistry (IUPAC) and ISO both shy away from the words limit of detection, arguing that this term implies a clearly defined cutoff above which the analyte is measured and below which it is not. The IUPAC and ISO prefer minimum detectable (true) value and minimum detectable value of the net state variable, which in analytical chemistry would become minimum detectable net concentration. Note that the LOD will depend on the matrix and therefore must be validated for any matrices likely to be encountered in the use of the method. These will, of course, be described in the method validation document. 

As mentioned before, LEAP1 and LEAP2 are the results of conscious choices between accuracy and practical execution performance. Therefore it is important to conduct a set of controlled validation studies to assess the accuracy in terms of rates of false positives and false negatives in their search results and performance in terms of end-to-end search turnaround time. 

Finally we hope to see that more validation studies are conducted to compare any new search method with the reference exhaustive search (of course on a smaller validation virtual space of 104-106). Only through this type of rigorous validation studies, one can truly probe the rates of false positives and false negatives as well as the fold increase in search speed. This in turn allows end users to make informed decisions on which search method will be a best match for their specific tasks. 

In general, there is a tendency to accept a positive response as more valid than a negative one when comparing tests, because we know more ways in which a test can fail than in which it can give a falsely positive result. 

We compared the assay results for 80 common chemicals from both the Nishihara et al. and NCTR data sets inconsistent assay results were observed for 12 chemicals. Specifically, of 30 active chemicals in the Nishihara et al. data set, one chemical was found inactive in the NCTR data set of 50 inactive chemicals in the Nishihara et al. data set, 11 chemicals were found active in the NCTR data set. These observations show that even using the experimental data from the ER binding assay (the NCTR data set) to predict the experimental results from the yeast two-hybrid assay (the Nishihara et al. data set), there may be about a 15% (12/80) discrepancy, or 3.3% (1/30) false negative rate and 22% (11/50) false positive rate. Care should be taken in interpreting the QSAR validation results using this data set (Hong et al., 2002). 

A noninvasive and simple H2 breath test has been suggested for measurement of pancreatic function based on the principle that undigested starch will pass into the colon, be metabolized by the colonic flora, and thus lead to an increase in breath H2 exhalation. However, numerous mechanisms for false-positive (e.g., bacterial overgrowth) and -negative (e.g., insufficient H2 production by colonic bacteria) limit its validity. 

Gad and co-workers (1986,1887) presented validation studies of 72 compounds and reported a false-negative rate of only 2 % and no false-positives when comparing MEST to GPMT data on the same materials. The incidence of sensitization in MEST was consistently lower than that produced by GPMT. Similar findings were reported for comparisons between the Buehler assay and MEST in the same paper. More recent publications have confirmed that the incidence of positive response in MEST is consistently lower than that in GMPT, and weak and moderate sensitizers are not identified correctly (Comacoff et al. 1988 and Dunn et al. 1990). 

It is our experience that the resnlts obtained by using fluorescent techniques that do not use ratiometric measurement and/or time resolution should be carried out and validated with great care in order to avoid false positive or negative results. 

---