Multiple comparisons

Pocock (1977) developed a procedure which divides the type I error rate of 5 per cent equally across the various analyses. In the example above with two interim looks and a final analysis, Bonferroni would suggest using an adjusted significance level of 0.017 (= 0.05 4- 3). The Pocock method however gives us the correct adjusted significance level as 0.022 and this exactly preserves the overall 5 per cent type I error rate. 

The methods as presented here assume that the analyses are equally spaced in terms of the numbers of patients involved at each stage. It is possible to deviate from this in a planned way using so-called alpha-spending functions. 

It is also possible to stop trials for reasons other than overwhelming efficacy, for example for futility, where at an interim stage it is clear that if the trial were to continue it would have little chance of giving a positive result. We will say more about interim analysis in a later chapter and in particular consider the practical application of these methods. 

In the case of multiple treatment groups it is important to recognise the objectives of the trial. For example, in a three-arm trial with test treatment, active comparator and placebo, the primary objective may well be to demonstrate the effectiveness 

---

METH-induced changes in neuropeptide levels, selective Dj (SCH 23390) and D2 (sulpiride) dopaminergic receptor antagonists were coadministered. The results are expressed as percent of control to facilitate comparisons each value represents the mean SEM of five to seven animals. Data were subjeeted to either a Student s r-test (figures 4 and 5) or ANOVA analysis followed by a multiple comparisons test (figures 1, 2, and 3). Signifieanee was set at the. 05 level. 

Multiple Comparisons and Multiple Tests Using SAS Text and Workbook Set (books in this set also sold separately) by Peter H. Westfall, Randall D. Tobias,... 

F exceeds the corresponding quantile of the F-distribution F a>V >V2 if at least one of the means differs significantly from the others. This global statement of variance analysis may be specified in the way to detect which of the mean(s) differ(s) from the others. This can be done by pairwise multiple comparisons (Tukey [1949] Games and Howell [1976] see Sachs [1992]). 

Games PA, Howell JF (1976) Pairwise multiple comparison procedures with unequal N s and/or variances a Monte Carlo study. J Educat Statist 1 113... 

Witte JS, Elston RC, Cardon LR. On the relative sample size required for multiple comparisons. Stat Med 2000 19 369-372. 

One-way-ANOVA tests were made to control the quality of the data. To achieve the proposed objectives, multiple comparison of means of the four groups was made by Tukey (p<0,05) test. The Dunnett (p<0,05) test was used to compare the means of three groups of mining sites with the control group (Seaman et al. 1991). Pearson correlation coefficients were also obtained to confirm the tests results. 

Seaman, M.A., Levin, J.R., Serlin, R.C. 1991. New developments in pairwise multiple comparisons Some powerful and... 

Three methods of analysis—linear regression (Gad, 1999 Steel and Torrie, 1960) a multiple comparison analysis, Dunnett s method (Dunnett, 1955) and a nonparametric analysis, such as Kruskal-Wallis (Gad, 1999)—can all be recommended. Each has its strengths and weaknesses, and other methods are not excluded. 

Dunnett, C.W. (1955). A multiple comparison procedure for comparing several treatments with a control. J. Am. Stat. Assoc. 50 1096-1121. 

Biologically meaningful and easier to resolve contrasts and multiple comparison tests. 

The analysis of rank data, what is generally called nonparametric statistical analysis, is an exact parallel of the more traditional (and familiar) parametric methods. There are methods for the single comparison case (just as Student s t-test is used) and for the multiple comparison case (just as analysis of variance is used) with appropriate post hoc tests for exact identification of the significance with a set of groups. Four tests are presented for evaluating statistical significance in rank data the Wilcoxon Rank Sum Test, distribution-free multiple comparisons, Mann-Whitney U Test, and the Kruskall-Wallis nonparametric analysis of variance. For each of these tests, tables of distribution values for the evaluations of results can be found in any of a number of reference volumes (Gad, 1998). 

The distribution-free multiple comparison test should be used to compare three or more groups of nonparametric data. These groups are then analyzed two at a time for any significant differences (Hollander and Wolfe, 1973, pp. 124-129). The test can be used for data similar to those compared by the rank-sum test. We often employ this test for reproduction and mutagenicity studies (such as comparing survival rates of offspring of rats fed various amounts of test materials in the diet). 

When we find a significant difference, we do not know which groups are different. It is not correct to then perform a Mann-Whitney U Test on all possible combinations rather, a multiple comparison method must be used, such as the distribution-free multiple comparisons. 

This is the most commonly misused test method, except in those few cases where one is truly only comparing two groups of data and the group sizes are roughly equivalent. Not valid for multiple comparisons (because of resulting additive errors) or where group sizes are very unequal. 

It is not appropriate to use a /-test (or a two groups at a time version of ANOVA) to identify where significant differences are within the design group. A multiple-comparison post hoc method must be used. 

If ANOVA reveals no significance it is not appropriate to proceed to perform a post hoc test in hope of finding differences. To do so would only be another form of multiple comparisons, increasing the type I error rate beyond the desired level. 

Dunnett, C.W. (1964). New tables for multiple comparison with a control. Biometrics. 16 671— 685. 

Shapiro Wilks W-test for normal data Shapiro Wilks W-test for exponential data Maximum studentlzed residual Median of deviations from sample median Andrew s rho for robust regression Classical methods of multiple comparisons Multivariate methods... 

Statistical analysis should be appropriate to the types of outcome data collected and the number of genotypes nsed in the analysis. The handling of missing data should be clearly stated. Corrections for multiple comparisons (e.g., controlling for false discovery rates 36) should be performed if multiple statistical tests are carried out. 

Multiple comparisons - in which comparisons are made amongst more than two treatments. 

One uses ANOVA when comparing differences between three or more means. For two samples, the one-way ANOVA is the equivalent of the two-sample (unpaired) t test. The basic assumptions are (a) within each sample, the values are independent and identically normally distributed (i. e., they have the same mean and variance) (b) samples are independent of each other (c) the different samples are all assumed to come from populations having the same variance, thereby allowing for a pooled estimate of the variance and (d) for a multiple comparisons test of the sample means to be meaningful, the populations are viewed as fixed, meaning that the populations in the experiment include all those of interest. 

---