Accuracy estimates statistical validation

The basic criterion for successful validation was that a method should come within 25% of the "true value" at the 95% confidence level. To meet this criterion, the protocol for experimental testing and method validation was established with a firm statistical basis. A statistical protocol provided methods of data analysis that allowed the accuracy criterion to be evaluated with statistical parameters estimated from the laboratory test data. It also gave a means to evaluate precision and bias, independently and in combination, to determine the accuracy of sampling and analytical methods. The substances studied in the second phase of the study are summarized in Table I. 

Validation is one of the most difficult aspects of environmental QSAR development due to the comparatively small size of the database. Cross-validation has been useful in validating the effectiveness of the model. In this method, one compound is removed from the database, the equation is recalculated, and the toxicity of the omitted compound is estimated. The process is repeated for all compounds in the dataset and the results are tabulated. In this manner, a calculation of the accuracy of prediction of continuous data and the rate of misclassification for categorial data can be compiled. A more useful estimate of the validity of the QSAR model is its ability to predict the toxicity of new compounds. Generally, this is difficult to accomplish in a statistically significant way due to the slow accumulation of new data that meet the criteria used in the modeling process and the associated expense. 

Unfortunately, there is no simple and widely applicable method for determining the reliability of data with absolute certainty. Often, estimating the quality of experimented results reejuires as much effort as collecting the data. Reliability can be assessed in several wa vs. Experiments designed to reveal the presence of errors can be peiformed. Standards of known composition can be analyzed and the results compared with the known composition. A few minutes in the library to consult the chemical literature can be profitable. Calibrating equipment usually enhances the quality of data. Finally, statistical tests can be applied to the data. Because none of these options is perfect, we must ultimately make judgments as to the probable accuracy of our results. These judgments tend to become harsher and less optimistic with experience. The quality assurance of analytical methods and the ways to validate and report results are further discussed in Section 8D-3. 

Accuracy is a measure of how close to truth a method is in its measurement of a product parameter. In statistical terms, accuracy measures the bias of the method relative to a standard. As accuracy is a relative measurement, we need a definition of true or expected value. Often, there is no gold standard or independent measurement of the product parameter. Then, it may be appropriate to use a historical measurement of the same sample or a within-method control for comparison. This must be accounted for in the design of experiments to be conducted for the validation and spelled out in the protocol. Accuracy is measured by the observed value of the method relative to an expected value for that observation. Accuracy in percent can be calculated as ratio of observed to expected results or as a bias of the ratio of the difference between observed and expected to the expected result. For example, suppose that a standard one-pound brick of gold is measured on a scale 10 times and the average of these 10 weights is 9.99 lbs. Then calculating accuracy as a ratio, the accuracy of the scale can be estimated at (9.99/10) x 100% = 99.90%. Calculating the accuracy as a bias then [(9.99 - 10)/10] X 100% =-0.10% is the estimated bias. In the first approach ideal accuracy is 100%, and in the second calculation ideal bias is 0%. 

In this section, three categories of experimental design are considered for method validation experiments. An important quality of the design to be used is balance. Balance occurs when the levels each factor (either a fixed effects factor or a random effects variance component) are assigned the same number of experimental trials. Lack of balance can lead to erroneous statistical estimates of accuracy, precision, and linearity. Balance of design is one of the most important considerations in setting up the experimental trials. From a heuristic view this makes sense, we want an equivalent amount of information from each level of the factors. 

In many cases and circumstances of the daily quality control of analytical work RMs and CRMs are helpful tools. Very often RMs are sufficient, in particular for statistical control actions. Where a rough estimate of accuracy or even precision is sufficient, a simple RM or calibration material is also largely adequate. However, for the establishment of the accuracy in the procedure of method development and validation, for revalidation of modified methods or whenever the analyst needs to demonstrate accuracy, e.g. measurements for court cases, CRMs should be employed as they have the advantage of being certified. It will be up to the operator and the laboratory s quality management to determine when, where, and how RM or preferably CRMs shall be used. 

Precision and Accuracy. Precision and accuracy are assay performance characteristics that describe the random (statistical) errors and systematic errors (bias) associated with repeated measurements of the same sample under specified conditions [3-5]. Precision is typically estimated by the percent coefficient of variation (% CV, also referred to as relative standard deviation or RSD) but may certainly also be reported as standard deviations. Method accuracy is expressed as the percent relative error (% RE) and is determined by the percent deviation of the weighted samples mean from samples with nominal reference values. A collection of validation sample statistics can be found in References 9,11, and 25. 

A pair of cross-validated Waters Acuity UPLC instruments (Waters Corp., Milford MA) are used at Vitalea Science for metabolic analyses and have very high reproducibility. Figure 16.3 shows AMS quantitation for a single metabolite peak from urine separated on different days. AMS provides an uncertainty estimate for each measurement based on the number of recorded " C atoms (CV = 100/v ), and multiple measures (>3) are made for each fraction providing a normal standard deviation that often equals the counting statistics. There should be two of the six data replicates that do not overlap at the Ict uncertainties but there is only one, and that one occurs at the sharply rising start of the peak. Thus, these UPLCs appear reproducible with AMS quantitation to better than statistical accuracy. The LLOQ of chromatographic fractions is discussed in detail below. 

The results of this study indicated a significant decrease in both UCS and BTS, in the water saturated samples than those in dry condition. The non-linear relationships among UCS and BTS in the dry and saturated conditions are expressed by equations derived from statistical analyses with good determination coefficients. Equations were validated by the t-test and show that UCS can be estimated using BTS with good accuracy. 

---