Significance level adjusted

Pocock (1977) developed a procedure which divides the type I error rate of 5 per cent equally across the various analyses. In the example above with two interim looks and a final analysis, Bonferroni would suggest using an adjusted significance level of 0.017 (= 0.05 4- 3). The Pocock method however gives us the correct adjusted significance level as 0.022 and this exactly preserves the overall 5 per cent type I error rate. 

One area that we briefly mentioned was interim analysis, where we are looking at the data in the trial as it accumulates. The method due to Pocock (1977) was discussed to control the type I error rate across the series of interim looks. The Pocock methodology divided up the 5 per cent type I error rate equally across the analyses. So, for example, for two interim looks and a final analysis, the significance level at each analysis is 0.022. For the O Brien and Fleming (1979) method most of the 5 per cent is left over for the final analysis, while the first analysis is at a very stringent level and the adjusted significance levels are 0.00052, 0.014 and 0.045. 

These two methods are the most common approaches seen in pharmaceutical applications. A third method, which we see used from time to time, is due to Haybittle (1971) and Peto et al. (1976). Here a significance level of 0.001 is used for each of the interims, again leaving most of the 5 per cent left over for the final analysis. For two interims and a final, the adjusted significance level for the final analysis is in fact 0.05 to two decimal places, for three interims we have 0.049 left over . Clearly these methods have little effect on the final evaluation, but associated with that there is also little chance of stopping at an interim stage. 

In line with this, the interim analysis plan was revised and only one interim was to be conducted for both efficacy and futility after 40 per cent of the patients had completed 3 months of follow-up. Since the two proposed analyses were not equally spaced, a spending functions were needed to revise the adjusted significance levels and these turned out to be 0.0007 and 0.0497. 

As mentioned in the previous section, multiplicity can lead to adjustment of the significance level. There are, however, some situations when adjustment is not needed although these situations tend to have restrictions in other ways. We will focus this discussion in relation to multiple primary endpoints and in subsequent sections use similar arguments to deal with other aspects of multiple testing. 

A significant benefit for both primary endpoints, lung function and the symptom based clinical endpoint, should be demonstrated so that no multiplicity adjustment to significance levels is indicated. ... 

Under such circumstances no adjustment to the significance level is needed we have to show significance for both endpoints. 

Another way of avoiding adjustment is to combine the multiple measurements into a single composite variable. Examples would be disease-lfee survival in oncology, where the variable is the time to either disease recurrence or death, whichever occurs first, or a composite of death, non-fatal stroke, MI and heart failure, a binary outcome in a cardiovascular setting. This approach does not require adjustment of the significance level we are back to having a single primary endpoint. 

The proposed sample size was 600 patients and two interims were planned after 200 and 400 patients (completing 3 months follow-up) using the O Brien and Fleming scheme with adjusted two-sided significance levels of 0.00052, 0.014 and 0.045. A futility rule was also introduced, based on conditional power (under the current trend) being below 30 per cent for the trial to be stopped. 

CPMP (2002) Points to Consider on Multiplicity Issues in Clinical Trials General aspects of multiple testing were considered in this guideline together with discussion on adjustment of significance levels or specific circumstances where adjustment is not needed (see Chapter 10). 

It is easy to compensate for this increased risk of type I errors by dividing the probability level (usually 5 %) by the number of tests (Bonferoni method [8]). For instance, if ten variables have been recorded and the maximum acceptable risk for type I errors is 5 %, the significance level for each individual /-test should be adjusted to5%/10 = 0.5%. The corrected significance level will... 

Keeps the overall risk of any false positives down to 5 per cent by adjusting the level at which P is considered significant. 

Procedure Notes Some sample types may naturally contain significant levels of yttrium. In these cases, choose a suitable alternative internal standard, or mn the test without an internal standard. Use of the internal standard is not required, but it is helpful when there are variations in the viscosity among sample types. Samples may be prepared in higher or lower concentrations as needed. Standard concentrations may be adjusted as needed. Alternative procedures should be validated before use. 

For a given analysis, it is usual to distinguish between the error of precision of the method as a whole (including the preparation of the sample) and the error due to the atomic absorption measurement. For flame analysis, the precision level is approximately 0.5 to 3% of the concentration for the majority of elements and will obviously be poorer near the detection limit the presence of significant levels of salts leads to a reduction in the level of precision. The highest precision is obtained for absorption values of between 10 and 50% consequently, it is advisable to adjust the concentration of the solution to be analysed by using an appropriate dilution. 

There is no reason why, from the point of view of controlling the type one error rate, one cannot test for noninferiority and superiority in the same trial without having to adjust the individual significance levels. This is because the null hypotheses in question form a nested set. For example, the null hypothesis that the new drug is inferior in terms of effect on mean diastolic blood pressure by at least 2 mmHg logically implies... 

In practical terms, this means that if we perform multiple tests and make multiple inferences, each one at a reasonably low error probability, the likelihood that some of these inferences will be erroneous could be appreciable. To correct for this, one must conduct each individual test at a decreased significance level, with the result that either the power of the tests will be reduced as well, or the sample size must be increased to accommodate the desired power. This could make the trial prohibitively expensive. Statisticians sometimes refer to the need to adjust the significance level so that the experimentwise error rate is controlled, as the statistical penalty for multiplicity. 

For example, the gene expression values are 12.79,12.53, and 12.46 for the naive condition and 11.12, 10.77, and 11.38 for the 48-h activated condition from the T-cell immune response data. The sample sizes are nj = 2 = 3. The sample means are 12.60 and 11.09 and the sample variances are 0.0299 and 0.0937, resulting in a pooled variance of (0.2029). The i-statistic is (12.60 - 11.09)/0.2029 = 7.44 and the degree of freedom is ni -I- 2 2 = 3 -I- 3 — 2 = 4. Then we ean find a p-value of 0.003. If using Welch s t-test, the t-statistic is still 7.42 sinee i = n, but we find the p-value of 0.0055 since the degree of freedom is 3.364 rather than 4. We claim that the probe set is differentially expressed under the two eonditions because its p-value is less than a predetermined significance level (e.g., 0.05). In this manner, p-values for the other probe sets ean be calculated and interpreted. In Section 4.4, the overall interpretation for p-values of all of the probe sets is described with adjustments for multiple testing. The Student s t-test and Weleh s t-test are used for samples drawn independently from two eonditions. When samples from the two conditions are paired, a different version ealled the paired t-test is more appropriate than independent t-tests ... 

---