Variability, interlaboratory

Wang et al. also addressed the mass spectral reproducibility. They conducted a carefully controlled interlaboratory experiment where the effects of a number of parameters were systematically investigated.22 They demonstrated that nearly identical spectra could be obtained in carefully controlled experiments. Minor variations in the sample/matrix preparation procedures for MALDI and in the experimental conditions used for bacterial protein extraction or analysis were shown to result in changes in the resulting spectra. They also noted that a subset of peaks was less sensitive to experimental variables. These ions appeared to be conserved in spectra obtained even under different experimental conditions so long as they were obtained using genetically identical bacteria. The existence of these conserved peaks helped explain... 

On the other hand, reproducibility is the closeness of the agreement between the results of measurements of the same measurand carried out under changed conditions of measurement . The changed conditions include principle of measurement, method of measurement, observer, measuring instrument, reference standards, location, conditions of use, and time. Such variable conditions are typical for interlaboratory studies (laboratory intercomparisons). 

Horwitz points out the universal recognition of irreproducible differences in supposedly identical method results between laboratories. It has even been determined that when the same analyst is moved between laboratories that the variability of results obtained by that analyst increases. One government laboratory study concluded that variability in results could be minimized only if one was to conduct all analyses in a single laboratory. .. by the same analyst . So if we must always have interlaboratory variability how much allowance in results should be regarded as valid - or legally permissible as indicating identical results. What are the practical limits of acceptable variability between methods of analysis - especially for regulatory purposes. 

Horwitz claims that irrespective of the complexity found within various analytical methods the limits of analytical variability can be expressed or summarized by plotting the calculated mean coefficient of variation (CV), expressed as powers of two [ordinate], against the analyte level measured, expressed as powers of 10 [abscissa]. In an analysis of 150 independent Association of Official Analytical Chemists (AOAC) interlaboratory collaborative studies covering numerous methods, such as chromatography, atomic absorption, molecular absorption spectroscopy, spectrophotometry, and bioassay, it appears that the relationship describing the CV of an analytical method and the absolute analyte concentration is independent of the analyte type or the method used for detection. 

Weil, C.S. and Scala, R.A. (1971). Study of intra- and interlaboratory variability in the results of rabbit eye and skin irritation tests. Toxicol. Appl. Pharmacol. 19 276-360. 

The results of interlaboratory study II are presented in Fig. 4.5.1. Five sets of results were obtained for the LAS exercise, and four sets for the NPEO exercise. For LAS, the within-laboratory variability ranged between 2 and 8% (RSD) for sample III (distilled water spiked with lmgL-1 LAS), 1 and 13% for sample 112 (wastewater influent), and 3 and 8% for sample 113 (sample 112 spiked with lmgL-1 LAS). Between-laboratory variations (calculated from the mean of laboratory means, MOLM) amounted to RSDs of 15, 30 and 30% for samples III, 112 and 113, respectively. The LAS values reported were in the range of 700—1100 p,g L-1 in sample III, 1100-1800 p,g L-1 in sample 112 and 1900-3000 p,g L-1 in sample 113, indicating that even in the matrix wastewater influent, the spiked concentration of lmgL-1 LAS could be almost quantitatively determined by all laboratories. 

Although many laboratories have shown a correlation between their Caco-2 Papp data and Fabs in humans,35 39 the size of most data sets is too small to be able to derive useful permeability models. Thus, the possibility of pooling data from different sources to increase the size of the database used for modeling seems reasonable however, the success of this approach is highly unlikely considering the magnitude of the interlaboratory variability in Fapp values.40 The possibility of developing useful in silico models to predict absorption and permeability will remain limited unless databases of appropriate quality are developed. 

Interlaboratory or between-laboratory precision is defined in terms of the variability between test results obtained on the aliquots of the same homogeneous material in different laboratories using the same test method. 

The interlaboratory results obtained from the analysis of defined standard solutions, but also from the analysis of sediment extracts prepared either by the coordinator of the study or by the participants themselves, also provide a measure of the variation between laboratories. The results show that the interlaboratory reproducibility ranges from 6.5% for the defined dioxin sample to 27.9% for the sediment sample extracted by the participants themselves. As was mentioned before, the reproducibility for this last sample is relatively high and most presumably due to the introduction of extra handlings (extraction and cleanup) to the total procedure. In addition, the fact that not all the participants had prior experience with the extraction protocol to be used could have added to the increase in variability of the process. Furthermore, the dilution factor was not dictated. This also introduces a certain degree of variation. For the reproducibility of the DR CALUX bioassay itself and not caused by differences in operating extraction conditions, the maximum variation between laboratories was observed to be 18.0%. The results for the sediment extract samples can also be used to estimate the method variability for extracts, that is, based on samples of unknown composition. Again, given the intra-as well as the interlaboratory variations observed in this study, it appears justified to conclude that the standard deviation of the means provides a reasonable estimate of the method variability, based on the overall aver-... 

Several overall conclusions can be drawn based on the statistical evaluation of the data submitted by the participants of the DR CALUX intra-and interlaboratory validation study. First, differences in expertise between the laboratories are apparent based on the results for the calibration curves (both for the curves as provided by the coordinator and for the curves that were prepared by the participants) and on the differences in individual measurement variability. Second, the average results, over all participants, are very close to the true concentration, expressed in DR CALUX 2,3,7,8-TCDD TEQs for the analytical samples. Furthermore, the interlaboratory variation for the different sample types can be regarded as estimates for the method variability. The analytical method variability is estimated to be 10.5% for analytical samples and 22.0% for sediment extracts. Finally, responses appear dependent on the dilution of the final solution to be measured. This is hypothesized to be due to differences in dose-effect curves for different dioxin responsive element-active substances. For 2,3,7,8-TCDD, this effect is not observed. Overall, based on bioassay characteristics presented here and harmonized quality criteria published elsewhere (Behnisch et al., 2001a), the DR CALUX bioassay is regarded as an accurate and reliable tool for intensive monitoring of coastal sediments. 

Ruggedness measure for variability (reproducibility of results obtained under variety of conditions) expressed as %RSD (interlaboratory)... 

The top-down approach is often used when there are method validation data from properly conducted interlaboratory studies, and when the laboratory using reproducibility as the measurement uncertainty can demonstrate that such data are applicable to its operations. Chapter 5 describes these types of studies in greater detail. In assigning the reproducibility standard deviation, sR, to the measurement uncertainty from method validation of a standard method, it is assumed that usual laboratory variables (mass, volume, temperature, times, pH) are within normal limits (e.g., 2°C for temperature, 5% for timing of steps, 0.05 for pH). Clause 5.4.6.2 in ISO/ 17025 (ISO/IEC 2005) reads, In those cases where a well-recognized test method specifies limits to the values of the major sources of uncertainty of measurement and specifies the form of presentation of the calculated results, the laboratory is considered to have satisfied this clause by following the test method and reporting instructions. ... 

Accurate measurement of pH is critically dependent on good analytical procedures, a fact that may not be appreciated by laboratory personnel [1,2]. The assumption is often made that if the electrode has been calibrated, there will be no variability in pH between laboratories. The pH measurement can erroneously be seen as merely dipping the electrode into the analyte and recording the value. In 1985, Davidson and Gardner [3] drew the following conclusion from their study Interlaboratory Comparisons of the Determination of pH in Poorly Buffered Fresh Waters ... 

When ASTM, followed by ISO and others, started conducting systematic interlaboratory trials to obtain precision data for test methods, the true state of affairs became apparent15. For many standards the variability was worse than realised and in some cases was so bad as to question whether the tests were worth doing at all or whether specifications based on them could be considered valid. The general advance of the quality movement prompted these investigations and have ensured that reproducibility has continued to occupy one of the top spots for attention in recent years. 

Interlaboratory trials with the organiser making detailed assessments of the laboratories is clearly particularly suited to helping individual laboratories and will at least qualitatively indicate the parameters requiring attention. This approach is, however, very expensive in total effort. The Intercal approach does not identify the causes of variability immediately but certainly alleviates the effect and, because trials are on-going, allows improvement to be monitored. Systematic quantification of the effect of individual parameters is probably the most cost effective approach and is the most useful for aiding standards committees to improve the specification of methods, but is of less direct help to individual laboratories. 

Any shortcoming in a standard can only be put right after analysis has pinpointed the problems. Hence, standards committees cannot act quickly if an interlaboratory trial reveals excessive variability. It is highly unlikely that faults in standards account for the majority of variance, although clearly it is important that any that do exist are identified and action taken. 

Based on an interlaboratory study, the AOCS estimates that the repeatability of measurements should be 1.3% and the between-laboratory variability 3.3% (Firestone, 1998). 

A recent interlaboratory comparison of HPLC and microbiological methods for total riboflavin revealed significant variability between the 13 participating laboratories (42). The extraction and hydrolysis of the riboflavin coenzymes were cited as the most likely sources of this variability. A later intercomparison (70) of riboflavin methods showed lower variability between laboratories, although coefficients of variability (CV) of 12-40% were still reported. 

---