Error, types

Since significance tests are based on probabilities, their interpretation is naturally subject to error. As we have already seen, significance tests are carried out at a significance level, a, that defines the probability of rejecting a null hypothesis that is true. For example, when a significance test is conducted at a = 0.05, there is a 5% probability that the null hypothesis will be incorrectly rejected. This is known as a type 1 error, and its risk is always equivalent to a. Type 1 errors in two-tailed and one-tailed significance tests are represented by the shaded areas under the probability distribution curves in Figure 4.10. 

The probability of a type 1 error is inversely related to the probability of a type 2 error. Minimizing a type 1 error by decreasing a, for example, increases the likelihood of a type 2 error. The value of a chosen for a particular significance test, therefore, represents a compromise between these two types of error. Most of the examples in this text use a 95% confidence level, or a = 0.05, since this is the most frequently used confidence level for the majority of analytical work. It is not unusual, however, for more stringent (e.g. a = 0.01) or for more lenient (e.g. a = 0.10) confidence levels to be used. 

Normal distribution curves showing the definition of detection limit and limit of identification (LOI). The probability of a type 1 error is indicated by the dark shading, and the probability of a type 2 error is indicated by light shading. 

Establishment of areas where the signal is never detected, always detected, and where results are ambiguous. The upper and lower confidence limits are defined by the probability of a type 1 error (dark shading), and the probability of a type 2 error (light shading). 

The issue covered in the second and third examples relate to elevating the type-1 error as we carry out more and more individual tests. There are three circumstances in which this may occur ... 

From which we can determine the probability of at least one type-1 error as... 

There are two potential solutions to this problem. First, we can pre-specify a single test (group) on the primary endpoint at a single point in time, much in line with Hill s views. Or, we can attempt a statistical solution based on adjusting the type-1 error for the individual tests. 

We see that in contrast to the type-1 error, the type-11 error is defined as occurring when accepting the null hypothesis if it is false. The power of a test is defined to be the probability of detecting a true difference and is equal to 1 — probability (type-11 error). The type-11 error and power depend upon the type-1 error, the sample size, the clinically relevant difference (CRD) that we are interested in detecting and the expected variability. Where do these values come from ... 

Type-1 error - this is preset and usually takes the value of either 0.05 or 0.01 ... 

Statistical Analysis. Analysis of variance (ANOVA) of toxicity data was conducted using SAS/STAT software (version 8.2 SAS Institute, Cary, NC). All toxicity data were transformed (square root, log, or rank) before ANOVA. Comparisons among multiple treatment means were made by Fisher s LSD procedure, and differences between individual treatments and controls were determined by one-tailed Dunnett s or Wilcoxon tests. Statements of statistical significance refer to a probability of type 1 error of 5% or less (p s 0.05). Median lethal concentrations (LCjq) were determined by the Trimmed Spearman-Karber method using TOXSTAT software (version 3.5 Lincoln Software Associates, Bisbee, AZ). 

Selected characteristics were compared between cases and controls by using test. The analyses of data were performed using the computer software SPSS for Windows version 11.5. Max type 1 error was accept as 0.05. Binary logistic regression was performed to calculate the odds ratios (ORs), and 95% confidence intervals (Cls) to assess the risk of breast cancer. 

The smaller the P-value the less likely it is that, under the null hypothesis, i.e. HO is true, we would have obtained the observed value of the test statistic. So, a small value of P is likely to mean that HO is untrue, in which case we should prefer HI. However, it is possible that HO is true but a rather unlikely event has taken place. Thus accepting the 5% level of significance (P<0.05) in rejecting HO means that 95 times out of 100 we are probably correct in our decision, but 5 times out of 100 we run the risk of rejecting HO when in fact it is true. Rejecting the null hypothesis when it is actually true is referred to as a Type 1 error. Accepting HO when it is not true is a Type 2 error. The probability of a Type 1 error is given the symbol a the probability of a Type 2 error the symbol 3 (Table 21.3). 

As would be expected, in order to be able to have at least 95% confidence that the true CV p does not exceed its target level, we must suffer the penalty of sometimes falsely accepting a "bad" method (i.e. one whose true CV p is unsatisfactory). Such decision errors, referred to as "type-1 errors", occur randomly but have a controlled long-term frequency of less than 5% of the cases. (The 5% probability of type-1 error is by definition the complement of the confidence level.) The upper confidence limit on CV p is below the target level when the method is judged acceptable... 

One technique employed to arrive, at an appropriate value has been postulated by L. Torbeck [16], who has taken a statistical and practical approach that in the absence of any other retest rationale can be judged as a technically sound plan. According to Torbeck, the question to be answered— how big should the sample be —is not easily resolved. One answer is that we first need a prior estimate of the inherent variability, the variance, under the same conditions to be used in the investigation. What is needed is an estimate of a risk level (defined as the percentage of probability that a significant difference exists between samples when there is none what statisticians call a type 1 error), the (i risk level ([1 is the probability of concluding that there is no difference between two means when there is one also known as a type 2 error) and the size of the difference (between the OOS result and the limit value) to be detected. The formula for the sample size for a difference from the mean is expressed as ... 

Translated into statistics, this implies that for safety pharmacology the risk of Type 2 errors (false negatives) should be decreased as much as possible, even if there is an increase in the risk of Type 1 errors (false positives). In other words, the statistical tests employed in safety pharmacology should err in the direction of oversensitivity rather than the reverse. A test substance found not to have significant safety risks based on preclinical studies, even after the use of oversensitive statistics, is more likely to be truly devoid of risk. As a consequence, the statistical analyses proposed for the CNS safety procedures described below (mainly two-by-two comparisons with control using Student s t tests) have been selected for maximal sensitivity to possible effects per dose at the acknowledged risk of making more Type 1 errors. 

Robust statistical method Significance tests Standard uncertainty True value Type 1 error Type II error Uncertainty j -Residuals... 

As suggested above it is eustomary to work at the 95 percent or sometimes at the 99 percent probability level. The 95 percent probability level, which gives a 5 percent chance of a Type I error, represents the usual optimum for minimizing the two types of statistical error. A Type 1 error is a false rejection by a statistical test of the null hypothesis when it is true. Conversely, a Type 11 error is a false acceptance of the null hypothesis by a statistical test. The probability level at which statistical decisions are made will obviously depend on which type of error is more important. 

The Type 1 error finds a difference between treatments when in reality, none exists. 

One shortcoming of the hypothesis testing approach is the arbitrary choice of a value for a. Depending upon our risk tolerance for committing a type 1 error, the conventional value of 0.05 may not be acceptable. Another way to convey the "extremeness" of the resulting test statistic is to report a p value. 

Regulatory authorities have many reasons to be concerned about type 1 errors. As a review at the end of this chapter, the reader is encouraged to think about the implications for a pharmaceutical company of committing a type 1 ot 11 error at the conclusion of a confirmatory efficacy study. 

C No. of hypotheses tested at a = 0.05 Maximum probability of type 1 error... 

Method Minimally significant Overall type 1 error rate P(rejecting... 

As the difference in mean ranks exceeds the MSD for the comparison of 10 vs 20 mg and 10 vs 30 mg, we can conclude that these distributions differ in location. This testing procedure ensured that the overall type 1 error did not exceed 0.05. To interpret the clinical relevance of the differences detected by the test requires some additional point estimates. As the initial procedure was a nonparametric one, the differences in sample means are not appropriate. A more reasonable choice would be to compare the medians as an estimate of the treatment effect. 

In Chapter 11 we discussed the issue of multiple comparisons and multiplicity in the context of pairwise treatment comparisons following a significant omnibus F test. When we adopt the 5% significance level (a = 0.05), by definition it is likely that a type 1 error will occur when 20 separate comparisons are made. That is, a statistically significant result will be "found" by chance alone. The greater the number of objectives presented in a study protocol, the greater the number of comparisons that will be... 

---