Protocol testing accuracy

A toxicity test is carried out following standard protocols. The accuracy of the test is determined by its sensitivity and specificity. The test s sensitivity determines how many should-be positives are scored as positive, whereas the specificity tells how many inactive substances also score negative in the test. In the example below we suppose that the test has 95% sensitivity and 95% specificity. Suppose that the truth is that 2% of 1000 substances are carcinogenic. However, in spite of the good accuracy of the test, it may be too coarse for screening. Let us do some calculation to illustrate this ... 

An article that has been published In a peer-reviewed industrial hygiene journal describing the protocol and explaining how test data support the protocol s accuracy and reliability. 

Accuracy is the term used to describe the degree of deviation (bias) between the (often unknown) true value and what is found by means of a given analytical method. Accuracy cannot be determined by statistical means the test protocol must be devised to include the necessary comparisons (blanks, other methods). 

The establishment of performance criteria for a given tumor marker test is not a simple process because accuracy and precision are unique for each type of analyte and its application. Establishing methodological limits for accuracy, precision, sensitivity, and specificity often requires standard reference materials, quality control materials, comparative studies, and actual clinical specimens. Accuracy and precision must be measured over the analyte reportable range for which the device is intended to be used. Sensitivity and specificity must be considered with respect to the intended clinical use of the device. Also, the indications for use should be carefully considered in the design of the study protocol. The indications for class II should be to monitor residual tumor after surgery (or radiation), the recurrence of tumor, or response to therapy. A 510(k) must provide clear evidence that the device is accurate, safe, effective, and substantially equivalent to a device legally marketed in the United States. 

Combining both heating and nonheating protocols employed in a sequential order were evaluated, but without any advantage (Fig. 3.4). RT-PCR was performed by standard methods, RNA extracted from fresh MDA cells and human tissue of breast cancer with known tested genes was used as positive control, and pure water was used to replace template (cDNA) as negative control for every experiment of PCR. To assure the accuracy of PCR tests, all reactions were performed in triplicate. 

Re-qualification (RQ) is a combination of OQ and PQ. This can be written as a separate protocol or can be included as part of the PQ protocol. This protocol can be executed anytime to demonstrate that maintenance has been performed, or periodically to demonstrate that the system is within tolerances. The RQ protocol should be able to be executed as needed without getting new signatures. Suppose the power supply has been corrected. An RQ protocol could include the high voltage accuracy and stability tests along with system suitability for one of the methods run on the system. This would demonstrate that the system is performing appropriately after the maintenance, as the manufacturer and you intend. 

While methods validation and accuracy testing considerations presented here have been frequently discussed in the literature, they have been included here to emphasize their importance in the design of a total quality control protocol. The Youden two sample quality control scheme has been adapted for continuous analytical performance surveillance. Methods for graphical display of systematic and random error patterns have been presented with simulated performance data. Daily examination of the T, D, and Q quality control plots may be used to assess analytical performance. Once identified, patterns in the quality control plots can be used to assist in the diagnosis of a problem. Patterns of behavior in the systematic error contribution are more frequent and easy to diagnose. However, pattern complications in both error domains are observed and simultaneous events in both T and D plots can help to isolate the problems. Point-by-point comparisons of T and D plots should be made daily (immediately after the data are generated). Early detection of abnormal behavior reduces the possibility that large numbers of samples will require reanalysis. 

The basic criterion for successful validation was that a method should come within 25% of the "true value" at the 95% confidence level. To meet this criterion, the protocol for experimental testing and method validation was established with a firm statistical basis. A statistical protocol provided methods of data analysis that allowed the accuracy criterion to be evaluated with statistical parameters estimated from the laboratory test data. It also gave a means to evaluate precision and bias, independently and in combination, to determine the accuracy of sampling and analytical methods. The substances studied in the second phase of the study are summarized in Table I. 

We have presented a statistical experimental design and a protocol to use in evaluating laboratory data to determine whether the sampling and analytical method tested meets a defined accuracy criterion. The accuracy is defined relative to a single measurement from the test method rather than for a mean of several replicate test results. Accuracy here is the difference between the test result and the "true value, and thus, must combine the two sources of measurement error ... 

The process of providing an answer to a particular analytical problem is presented in Figure 2. The analytical system—which is a defined method protocol, applicable to a specified type of test material and to a defined concentration rate of the analyte —must be fit for a particular analytical purpose [4]. This analytical purpose reflects the achievement of analytical results with an acceptable standard of accuracy. Without a statement of uncertainty, a result cannot be interpreted and, as such, has no value [8]. A result must be expressed with its expanded uncertainty, which in general represents a 95% confidence interval around the result. The probability that the mean measurement value is included in the expanded uncertainty is 95%, provided that it is an unbiased value which is made traceable to an internationally recognized reference or standard. In this way, the establishment of trace-ability and the calculation of MU are linked to each other. Before MU is estimated, it must be demonstrated that the result is traceable to a reference or standard which is assumed to represent the truth [9,10]. 

The key elements of an inspection are to ensure that the facility is capable of fulfilling the application commitments to manufacture, process, control, package, and label a drug product following GMP the adequacy and accuracy of analytical methods submitted, to ensure that these methods are proper for the testing proposed correlation between the manufacturing process for clinical trial material, bioavailability study material, and stability studies and submitted process that the scientific data support full-scale production procedures and controls that only factual data have been submitted and that the protocols are in place to validate the manufacturing process. 

The ideal validated method would be the one that has progressed fully through a collaborative study in accordance with international protocols for the design, conduct, and interpretation of method performance studies. A typical study of a determinative method conducted in accordance with the internationally harmonized International Organization for Standardization (ISO)/International Union for Pure and Applied Chemistry (IUPAC)/AOAC International (AOAC) protocol would require a minimum of up to five test materials including blind replicates or split-level samples to assess within-laboratory repeatability parameters, and eight participating laboratories (15). Included with the intended use should be recommended performance criteria for accuracy, precision and recovery. 

Basic tests, such as hardness, alkalinity, chloride, silica, and phosphate, can generally be undertaken with low-priced simple equipment. The individual methods normally incorporate suitable protocols to minimize interferences. Thus, almost any equipment and any test method can be selected that meets the rule of fitness-for-purpose. (Typically this is a mix of portability, ease of testing, and speed of handling multiple water samples, combined with accuracy, suitable detection limits, price of equipment and replacement reagents, and service company testing protocol uniformity.)... 

Quantitation limit is defined in the ICH Q2A guideline as the lowest amount of analyte in a sample which can be quantitatively determined with suitable precision and accuracy [2]. As with the detection limit, ICH guideline Q2B expanded upon this statement and listed the same three approaches to the testing methodology [3]. Again the method for the determination must be listed in the validation protocol and supported by the validation experiments. 

---