Hypothesis tests multiple comparisons

Hence, hypothesis testing (ANOVA analysis followed by multiple comparison analysis) was used to determine NOEC and LOEC values expressed as % v/v of effluent. In order to satisfy statistical analysis requirements enabling NOEC and LOEC determinations, some bioassay protocols were adjusted to make sure that there were at least three replicates per effluent concentration and at least five effluent concentrations tested. TC % effluent values were then determined as follows ... 

Note that performing multiple hypothesis tests (e.g., Student s t-test) may inflate the false positive rate. That is, the number of genes detected as active by chance alone will increase with the number of genes tested. For example, a microarray with 7000 features would require at least 7000 hypothesis tests per treatment comparison. Several methods have been developed to control the false positive rate, such as the conservative Bonferroni correction and the FDR control method (49). 

For the post-ANOVA, pair-wise evaluations, there are procedures to deal with the multiple comparison problem. One such procedure is based on the F-distribution with one and N — k degrees of freedom. This test also relies on the value of sj from ANOVA. The test statistic is F = /sj) [(xi — n + I/M2)], where x.i, x.2 are the means of the n and 2 values for the two lots in the pair-wise comparison. Comparing lot A and lot C F = (1/6.575) [(99.5 - 90.5)2/(1/20 + 1/20)] = 123.2. This far exceeds the critical Fi 75 value at even a 1% level, which is <7.08, based on Fi go, and we therefore reject the hypothesis that lot A and lot C means are equal. Because, the means for lot B and lot D differ from that of lot C by an even greater amoimt, they also are foimd to be statistically different from the lot C mean. By contrast, the comparison of lots A and D, with means of 99.5%i and 100.3%), respectively, have an F-test value of 0.97, far less than the critical 5%o value, which is <4.00. 

Bonferroni s test is the most straightforward of several statistical methodologies that can appropriately be used in the context of multiple comparisons. That is, Bonferroni s test can appropriately be used to compare pairs of means after rejection of the null hypothesis following a significant omnibus F test. Imagine that we have c groups in total. Bonferroni s method makes use of the following inequality ... 

Multiple hypothesis testing. One statistical comparison in 20 is likely to be... 

The familywise Type I error rate increases with repeated hypothesis testing, a phenomenon referred to as multiplicity, leading the analyst to falsely choose a model with more parameters. For example, the Type I error rate increases to 0.0975 with two model comparisons, and there is a 1 in 4 chance of choosing an overparameterized model with six model comparisons. 

We touched on this problem in Chapter 9, where we drew attention to Cournot s criticism of multiple comparisons. To use the language of hypothesis testing, the problem is that as we carry out more and more tests, the probability of making at least one type I error increases. This probability of at least one type I error is sometimes referred to as the family-wise error rate (FWER) (Benjamini and Hochberg, 1995). Thus, controlling the type I error rates of individual tests does not guarantee control of the FWER. To put... 

The results of such multiple paired comparison tests are usually analyzed with Friedman s rank sum test [4] or with more sophisticated methods, e.g. the one using the Bradley-Terry model [5]. A good introduction to the theory and applications of paired comparison tests is David [6]. Since Friedman s rank sum test is based on less restrictive, ordering assumptions it is a robust alternative to two-way analysis of variance which rests upon the normality assumption. For each panellist (and presentation) the three products are scored, i.e. a product gets a score 1,2 or 3, when it is preferred twice, once or not at all, respectively. The rank scores are summed for each product i. One then tests the hypothesis that this result could be obtained under the null hypothesis that there is no difference between the three products and that the ranks were assigned randomly. Friedman s test statistic for this reads... 

To test this hypothesis beyond CO adsorption on Pt(l 11), Weaver et al. compared CO and NO stretching frequencies on multiple crystal facets of Pt, Rh, Pd, and Ir in UHV and electrochemical environments.58 With the exception of NO and CO on Pt(l 11), in which both unsolvated and D20 solvated environments were examined, only unsolvated UHV environments were considered. In this comparison, the same... 

Table 2. Comparison of clustering methods and distance functions. The agreement between the sets of clusters resulting from the four clustering methods was measured using the k test. The standard deviations of the statistic under the null hypothesis were estimated to range between 0.014 and 0.023 from multiple simulations. From Chen and Murphy (2005).

---