Interlaboratory reproducibility

Reproducibility (interlaboratory precision) is the precision that measures variability among the results obtained for the same sample at different laboratories (EPA, 1998a). It is usually statistically expressed as variance or the standard deviation. The low variance or the standard deviation value indicates a high degree of reproducibility. 

Reproducibility in the context of Directive 96/46/EC is defined as a validation of the repeatability of recovery, from representative matrices at representative levels, by at least one laboratory, which is independent of the laboratory which initially validated the study. This independent laboratory may be within the same company, but may not be involved in the development of the method. This concept of independent laboratory validation (ILV) substitutes the conduct of interlaboratory trials (e.g., according to ISO 5725) because the resources are not available taking into consideration the high number of a.i., matrix types and concentration levels which must be validated in the registration procedure. 

Wang et al. also addressed the mass spectral reproducibility. They conducted a carefully controlled interlaboratory experiment where the effects of a number of parameters were systematically investigated.22 They demonstrated that nearly identical spectra could be obtained in carefully controlled experiments. Minor variations in the sample/matrix preparation procedures for MALDI and in the experimental conditions used for bacterial protein extraction or analysis were shown to result in changes in the resulting spectra. They also noted that a subset of peaks was less sensitive to experimental variables. These ions appeared to be conserved in spectra obtained even under different experimental conditions so long as they were obtained using genetically identical bacteria. The existence of these conserved peaks helped explain... 

On the other hand, reproducibility is the closeness of the agreement between the results of measurements of the same measurand carried out under changed conditions of measurement . The changed conditions include principle of measurement, method of measurement, observer, measuring instrument, reference standards, location, conditions of use, and time. Such variable conditions are typical for interlaboratory studies (laboratory intercomparisons). 

It is a well-known fact that the precision in trace analysis decreases with diminishing concentration in a similar way as it does with decreasing sample weight (Sect. 2.1). The dependency of the repeatability and reproducibility standard deviation on the concentration of analytes has been investigated systematically at first by Horwitz et al. [1980] on the basis of thousands of pieces of interlaboratory data (mostly from food analysis). The result of the study has been represented in form of the well-known Horwitz trumpet which is represented in Fig. 7.3. 

Rhodes A, Jasani B, Balaton AJ, et al. Study of interlaboratory reliability and reproducibility of estrogen and progesterone receptor assays in Europe documentation of poor reliability and identification of insufficient microwave antigen retrieval time as a major contributory element of unreliable assays. Am. J. Clin. Pathol. 2001 115 44-58. 

Figure 4.6 Interlaboratory coefficient of variation as a function of concentration (note that the filled circles are values calculated by using equation (4.4), not experimental points) [10]. Reproduced by permission of AOAC International, from Horwitz, W., J. Assoc. Off. Anal. Chem., 66, 1295-1301 (1983).

To calculate the RSD value expected for the interlaboratory reproducibility a starting point is to apply the Horwitz function. 

Most of the methods employ an internal standard, and all are reported as accurate and reproducible. However, Schmidt, et. al. (40) recently reported great interlaboratory variation in the analysis of such biological samples. 

The last type of precision study is reproducibility, which is determined by testing homogeneons samples in mnltiple laboratories, often as part of interlaboratory crossover stndies. The evalnation of reprodncibility results often focuses more on measnring bias in resnlts than on determining differences in precision alone. Statistical eqnivalence is often nsed as a measnre of acceptable interlaboratory resnlts. An alternative, more practical approach is the nse of analytical equivalence, in which a range of acceptable resnlts is chosen prior to the study and used to judge the acceptability of the results obtained from the various laboratories. 

The wide methodological repertoire available provides great latitude for laboratories using HIER, but also underscores the need for standardization for better intra- and interlaboratory reproducibility. 

A common rationale for the acquisition of an automated immunostainer is that there will be a significant reduction in skilled technicians time. The true value of automation, however, is improved reproducibility and reliability of staining. Intra- and interlaboratory standardization of reagents and automated... 

The acute Daphnia bioassay is recognized to be one of the most standardized aquatic toxicity tests presently available and several intercalibration exercises report a reasonable degree of intra- and interlaboratory reproducibility [84-87]. 

We have seen two different approaches to estimate the measurement uncertainty. One was using data from control charts, CRM analysis, PT results and/or recoveiy tests and sometimes maybe also experience of the analyst, the other was just using the reproducibility standard deviations from interlaboratory tests. In most cases the second method delivers higher estimates. 

There are different standard deviations depending on the measurements conditions repeatability conditions, between-batch and interlaboratory reproducibility conditions. 

Flow cytometric assessment generates results that reproduce microscopic methods for the in vivo assessment of micronucleus frequency in peripheral blood micronucleated reticulocytes [30] and the method is increasingly applied in in vitro assessments. Litron Laboratories (Rochester, N.Y., USA) recently launched their MicroFlow In Vitro kit for the in vitro micronucleus test, following a six-compound interlaboratory evaluation [31]. This method permits 50 samples to be analyzed over... 

The accuracy of results (the interlaboratory reproducibility of measurements) was poor or even dramatically low. The scatter in M and values obtained for identical samples in the different participating laboratories reached several hundred of percent (2000% for poly(acrylic acid). 

The popular weak solvents, which promote polymer adsorption are cyclohexane, dichloro methane, dichloroethane and toluene. Unfortunately, any standardization of the eluent purity/content of admixtures does not exist. This might be one of reasons why it is sometimes difficult to maintain long-term repeatability and interlaboratory reproducibility of measurements in polymer HPLC. 

Table 4 Dioxin responsive-chemically activated lucerferase (DR CALUX ) repeatability and reproducibility for the bioanalysis of the defined standard solutions and sediment samples of the various phases of the present interlaboratory validation study. 

The interlaboratory results obtained from the analysis of defined standard solutions, but also from the analysis of sediment extracts prepared either by the coordinator of the study or by the participants themselves, also provide a measure of the variation between laboratories. The results show that the interlaboratory reproducibility ranges from 6.5% for the defined dioxin sample to 27.9% for the sediment sample extracted by the participants themselves. As was mentioned before, the reproducibility for this last sample is relatively high and most presumably due to the introduction of extra handlings (extraction and cleanup) to the total procedure. In addition, the fact that not all the participants had prior experience with the extraction protocol to be used could have added to the increase in variability of the process. Furthermore, the dilution factor was not dictated. This also introduces a certain degree of variation. For the reproducibility of the DR CALUX bioassay itself and not caused by differences in operating extraction conditions, the maximum variation between laboratories was observed to be 18.0%. The results for the sediment extract samples can also be used to estimate the method variability for extracts, that is, based on samples of unknown composition. Again, given the intra-as well as the interlaboratory variations observed in this study, it appears justified to conclude that the standard deviation of the means provides a reasonable estimate of the method variability, based on the overall aver-... 

Two types of precision are usually distinguished, namely the repeatability and the reproducibility. Repeatability is the precision obtained in the best possible circumstances (same analyst, one instrument, within one day when possible) and reproducibility under the most adverse possible circumstances (different laboratories, different analysts, different instruments, longer periods of time, etc.). Reproducibility can be determined only with interlaboratory experiments. Intermediate situations may and do occur. They are for instance defined in terms of M-factor-different intermediate precision measures, where M is one, two, three or even higher [8,9]. In this definition M refers to the number of factors that are varied to make the estimation. The most likely factors to be varied are time, analyst and instrument. According to this terminology, one estimates e.g. the time-and-analyst-different intermediate precision measure (M=2), when the precision is determined by measuring a sample over a longer period of time in one laboratory by two analysts with one instrument. 

In a protocol about collaborative studies [10] it is also considered what is called preliminary estimates of precision. Among these the protocol defines the total within-laboratory standard deviation . This includes both the within-run or intra-assay variation (= repeatability) and the between-run or inter-assay variation. The latter means that one has measured on different days and preferably has used different calibration curves. It can be considered as a within-laboratory reproducibility. These estimates can be determined prior to an interlaboratory method performance study. The total within-laboratory standard deviation may be estimated fi-om ruggedness trials [10]. 

Reproducibility uncertainty is derived from reproducibility precision (interlaboratory tests) and counts for the repeatability error, run, and laboratory... 

Precision measures are divided into (1) repeatability precision measures s or SD (sT or SDr) and RSD (RSDr), (2) intralaboratory reproducibility precision or intermediate-precision measures SD and RSD, and (3) interlaboratory reproducibility precision s or SD (sR or SDR) and RSD (RSDr) [66]. 

Ruggedness measure for variability (reproducibility of results obtained under variety of conditions) expressed as %RSD (interlaboratory)... 

Any or all of these conditions can be varied. To provide some guidance, intralaboratory reproducibility is used to express changes only within a laboratory, and interlaboratory reproducibility" is used to refer to the changes that occur between laboratories, for example in proficiency testing, interlaboratory method validation studies, and the like. Interlaboratory reproducibility is usually two to three times the repeatability. 

Standard methods of analysis published by bodies such as the American Society for Testing and Materials (ASTM), de Normalisation (CEN), or ISO are rigorously tested and validated in method performance, or validation, studies. These interlaboratory trials can establish reproducibility and method bias and also give some confidence that the method can be used in different environments. Laboratories are chosen with an expectation that they can competently follow the proposed method, which will have already been extensively validated before an interlaboratory trial is contemplated. To this end, a pilot trial is sometimes undertaken to ensure the method description can be followed and to give an initial estimate of the precision of the method. 

One of the aims of a validation study is to establish the repeatability and reproducibility precision. Experiments performed under repeatability conditions (see chapter 2) are those repeated over a short period of time, with the same analyst and same equipment. For an analysis, the repeats must be independent, involving the entire method, and not simply the operation of the instrument several times. Reproducibility conditions are the sine qua non of the interlaboratory study. Different laboratories, with different analysts, instruments, reagents, and so on, analyze the sample at different times, but all with the same method. The relationship among these parameters is shown in figure 5.5. 

Figure 5.5. Diagram showing the relationship between repeatability and reproducibility and laboratory bias and method bias in the context of an interlaboratory study.

---