Statistical significance tests, limitations

Fig. 9. The two materials, A and B, have overlapping 95% confidence limits at the LD q level. Because the slopes of the dose—mortahty regression lines for both materials are similar, there is no statistically significant difference in mortahty at the LD q and LD q levels. Both materials may be assumed to be lethahy equitoxic over a wide range of doses, under the specific conditions of the test.

Immunological abnormalities were reported in 23 adults in Woburn, Massachusetts, who were exposed to contaminated well water and who were family members of children with leukemia (Byers et al. 1988). These immunological abnormalities, tested for 5 years after well closure, included persistent lymphocytosis, increased numbers of T-lymphocytes, and depressed helper suppressor T-cell ratio. Auto-antibodies, particularly anti-nuclear antibodies, were detected in 11 of 23 adults tested. This study is limited by the possible bias in identifying risk factors for immunological abnormalities in a small, nonpopulation-based group identified by leukemia types. Other limitations of this study are described in Section 2.2.2.8. A study of 356 residents of Tucson, Arizona, who were exposed to trichloroethylene (6-500 ppb) and other chemicals in well water drawn from the Santa Cmz aquifer found increased frequencies of 10 systemic lupus erythematosus symptoms, 5 (arthritis, Raynaud s phenomenon, malar rash, skin lesions related to sun exposure, seizure or convulsions) of which were statistically significant (Kilbum and Warshaw 1992). 

Statistical interval, e.g., of a mean, y, cnfCy ) = y ycnfwhich express the uncertainty of measured values. CIs are applied for significance tests and to establish quantities for limit values (CV). 

The statistical evaluation leads to a limit value, i.e., a critical effect, and all effects. Ex, that are in absolute value larger than or equal to the limit value are considered significant. The limit value is usually based on the t-test statistic given in the following equation " " ... 

A statistically significant increase in the number of deaths was noted only in female miee following chronie exposure to marine diesel fuel and JP-5 at doses of 250 and 500 mg/kg/day for both fuel oils (NTP/NIH 1986). Although the number of deaths in males under these eonditions was inereased over that of the eontrols, the effect was not significant. Deaths were observed as early as week 1 of exposure to marine diesel fuel and week 2 of exposure to JP-5. The data are limited for eaeh of these experiments because it was not specified whether the animals were protected against oral exposure and/or removal of the test material. 

As there are 18 degrees of freedom, the critical value for t at the 95% confidence limit for a two-tailed test is 2.10. Given that the calculated value for t is greater than the critical value, the null hypothesis is rejected, and we conclude there is a statistical significant difference between the new method and the reference method. We conclude that the two methods have similar precision but significantly different accuracy. 

Figure 6.9 showed the interplay between various factors in the outcome of a two-sample f-test. Two of these factors were the extent of the difference observed and the sample sizes. It is perfectly possible for a study to produce a statistically significant outcome even where the difference is very small, so long as the sample size is correspondingly large. In principal, there is no lower limit to the size of experimental effect that could be detected, if we were prepared to perform a big enough experiment. However, this can cause problems, as the next example demonstrates. 

If the confidence limit excludes the possibilities proposed by the null hypothesis, the outcome is statistically significant. With a properly conducted one-sided test, the risk of an accidental false positive when investigating a treatment that has no real effect is held at the usual 5 per cent. 

The experiments described here are principally diagnostic in nature where cellular biomass was significantly enhanced in bottles after resource (iron or light) amendment, relative to control (or other) treatments, we infer that algal growth rates in the control (or other) treatments were limited by a deficiency in that resource. The statistical significance of differences between mean values of parameters measured in different treatments were assessed using a two-tailed r-test for comparisons between two treatments, or a one-way analysis of variance (ANOVA) for comparisons between three or more treatments, at a confidence level of 95% (P = 0.05). 

This value can then be compared with a limiting F vi - V2,v ) = F, 3) value at the 95 percent confidence level from Table XXI-1 or, more conveniently, from the Lotus 1-2-3 function FDIST(1-0.95,1,3,1) = 10.128 [this function is FINV(l-0.95,1,3) in Quat-tro Pro]. Since F is less than the limiting value, the better fit of Model 2 is not statistically significant and the simpler linear model should be used. Put another way, if this test is done on an infinite number of new sets of similar absorption coefficient measurements, the decision to leave out the quadratic term will be correct 95 percent of the time. 

---