Familywise Error Rate

Suppose that we wish to make inferences on the parameters 0i,i = 1,g, where 9i represents the logarithm of the ratio of the expression levels of gene i under normal and disease conditions. If the ith gene has no differential expression, then the ratio is 1 and hence 0 = 0. In testing the g hypotheses Ho, 0 = 0, / = 1,..., g, suppose we set R, = 1 if H0, is rejected and Ri = 0 otherwise. Then, for any multiple testing procedure, one could in theory provide a complete description of the joint distribution of the indicator variables R, ..., Rg as a function of 0i,..., 0g in the entire parameter space. This is impractical if g > 2. Different controls of the error rate control different aspects of this joint distribution, with the most popular being weak control of the familywise error rate (FWER), strong control of the familywise error rate, and control of the false discovery rate (FDR). 

One should verify either conditions SI to S3 are satisfied, or subset pivotality is satisfied, before implementing a stepdown test for, otherwise, the stepdown test may not strongly control the familywise error rate. Such conditions are easier to check with a model that connects the observations with the parameters, but harder to check with a model (such as the randomization model) that only describes the distribution of the observations under the null hypotheses. Indeed, Westfall and Young (1993, page 91) cautioned that the randomization model does not guarantee that the subset pivotality condition holds. Outside the context of bioinformatics, there are in fact examples of methods that were in use at one time that violate... 

For this data set, at a familywise error rate of 5%, one-step testing leads to the conclusion that 174 genes are differentially expressed. Closed and partition stepdown testing are guaranteed to infer at least those same 174 genes to be differentially expressed. In fact, these last two methods both infer three additional genes to be differentially expressed that is, a total of 177 genes. 

Familywise error rate The error rate for one or more false positives among all the tests conducted. Controlling a familywise error rate leads to more stringent control than the false discovery rate control at the same nominal level. 

The familywise Type I error rate increases with repeated hypothesis testing, a phenomenon referred to as multiplicity, leading the analyst to falsely choose a model with more parameters. For example, the Type I error rate increases to 0.0975 with two model comparisons, and there is a 1 in 4 chance of choosing an overparameterized model with six model comparisons. 

---