Multiple Hypothesis Testing

Shaffer, J.P. (1995) Multiple hypothesis testing. Annu. Rev. Psychol. 46, 561-584. 

Note that performing multiple hypothesis tests (e.g., Student s t-test) may inflate the false positive rate. That is, the number of genes detected as active by chance alone will increase with the number of genes tested. For example, a microarray with 7000 features would require at least 7000 hypothesis tests per treatment comparison. Several methods have been developed to control the false positive rate, such as the conservative Bonferroni correction and the FDR control method (49). 

Multiple hypothesis testing. One statistical comparison in 20 is likely to be... 

Dudoit, S., Popper Shaffer J., Boldrick J.C. 2003. Multiple Hypothesis Testing in Microarray Experiments. Statistical Science 18 71-103. 

As an aside, the error rate in Eq. (1.32) assumes that the hypothesis tests are independent. Although this assumption has never been proven in the regression literature, model development using stepwise methods undoubtedly produces multiple hypothesis tests which are correlated, probably in relation to the degree of correlation between predictor variables used in a model. In the case where two statistical tests are correl-... 

Hence, hypothesis testing (ANOVA analysis followed by multiple comparison analysis) was used to determine NOEC and LOEC values expressed as % v/v of effluent. In order to satisfy statistical analysis requirements enabling NOEC and LOEC determinations, some bioassay protocols were adjusted to make sure that there were at least three replicates per effluent concentration and at least five effluent concentrations tested. TC % effluent values were then determined as follows ... 

Fishing trips We could push this idea one stage further and publish indiscriminately (all the data and all the analyses) without any correction for multiple testing, but accept that all conclusions are tentative and should only be used as the basis for hypothesis generation not hypothesis testing. 

Ge, Y., Dudoit, S., and Speed, T. R 2003. Resampling-based multiple testing for microarray data hypothesis. Test 12(1) 1—44. 

Multiplicative multilinear models can be used for modeling ANOVA data. Such multilinear models can be interesting, for example, in situations where traditional ANOVA interactions are not possible to estimate. In these situations GEMANOVA can be a feasible alternative especially if a comparably simple model can be developed. As opposed to traditional ANOVA models, GEMANOVA models suffer from less developed hypothesis testing and often the modeling is based on a more exploratory approach than in ANOVA. 

The familywise Type I error rate increases with repeated hypothesis testing, a phenomenon referred to as multiplicity, leading the analyst to falsely choose a model with more parameters. For example, the Type I error rate increases to 0.0975 with two model comparisons, and there is a 1 in 4 chance of choosing an overparameterized model with six model comparisons. 

There is also the issue of Type I error rate, which is the rate at which a covariate is deemed statistically important when in fact it is not. Hypothesis testing a large number of models is referred to as multiplicity and results in an inflated Type I error rate. Because a large number of models are tested as some level of significance, usually p < 0.05, then 5% of those models will be selected as being an improvement over the comparator model based on chance alone. 

We touched on this problem in Chapter 9, where we drew attention to Cournot s criticism of multiple comparisons. To use the language of hypothesis testing, the problem is that as we carry out more and more tests, the probability of making at least one type I error increases. This probability of at least one type I error is sometimes referred to as the family-wise error rate (FWER) (Benjamini and Hochberg, 1995). Thus, controlling the type I error rates of individual tests does not guarantee control of the FWER. To put... 

---