Error in hypothesis testing

Table 2.1 summarizes possible conclusions of the decision-making process and common statistical terms used for describing decision errors in hypothesis testing. 

It is important when thinking about errors in hypothesis testing to determine the consequences of making a type I or a type II error. If a type I error is much more likely to have serious consequences than a type II error, it is reasonable to choose a small value of a. On the other hand, in some situations a type II error would be quite serious, and so a larger value of a is employed to keep the type II eiTor rate under control. As a general rule of thumb, the largest a that is tolerable for the situation should be used. This ensures the smallest type II error while keeping the type I error within acceptable limits. 

The uncertainty in detection is calculated as the probability that the damage is true after having detected a fault. This is established using the choice of the significance level a in the hypothesis test. The probabilities of different types of errors in hypothesis testing are well established in literature. The error of rejecting a correct null hypothesis is known as Type-I error, while the error of not rejecting a false null hypothesis is known as Type-II error. The probability of Type-I error is equal to a and the probability of Type-II error is denoted by p. This information can be written as ... 

Similarly, in statistical tests to determine whether two quantifies are the same, two types of errors can be made. A type I error occurs when wereject the hypothesis that two quantities are the same when they are statistically identical. A,type li error occurs when we accept that they are the same when they are not stattsticMy identical. The characteristics of these errors in statistical testing and the ways we can minimize them are among the subjects of this chapter. 

The standard error of is more problematic, since it depends on the e.s.d. s of the Xj, which, as already noted, tend to be of dubious reliability. In consequence, standard errors of weighted means can be grossly over-optimistic [27] and it is undesirable to use weighted means in hypothesis testing. 

Power n Also known as the power of the test, it is the probability of rejecting the null hypothesis when it is false. In hypothesis testing this is equal to one minus the probability of a type II error, which is often represented by jl. The power is, therefore, equal to 1 — jS. See also Type I Error. 

Since significance tests are based on probabilities, their interpretation is naturally subject to error. As we have already seen, significance tests are carried out at a significance level, a, that defines the probability of rejecting a null hypothesis that is true. For example, when a significance test is conducted at a = 0.05, there is a 5% probability that the null hypothesis will be incorrectly rejected. This is known as a type 1 error, and its risk is always equivalent to a. Type 1 errors in two-tailed and one-tailed significance tests are represented by the shaded areas under the probability distribution curves in Figure 4.10. 

Significance tests, however, also are subject to type 2 errors in which the null hypothesis is falsely retained. Consider, for example, the situation shown in Figure 4.12b, where S is exactly equal to (Sa)dl. In this case the probability of a type 2 error is 50% since half of the signals arising from the sample s population fall below the detection limit. Thus, there is only a 50 50 probability that an analyte at the lUPAC detection limit will be detected. As defined, the lUPAC definition for the detection limit only indicates the smallest signal for which we can say, at a significance level of a, that an analyte is present in the sample. Failing to detect the analyte, however, does not imply that it is not present. 

It can be seen that k in Eq. (10) replaces the system-describing parameters L and Ah in Eq. (1). A direct test of the hypothesis is therefore to plot (j> against k for fixed values of P, G, and d, with L and Ah varying. For the hypothesis to be correct, the data points must all lie on a smooth curve. Experience shows, however, that plotting (f> against k often produces an undue amount of scatter which may obscure and distort any true relationship existing. This enhanced scatter is caused by the cumulative effect of experimental errors in the various terms in the heat-balance equation from which the quality k is derived. 

In this case we assume that we know precisely the value of the standard experimental error in the measurements (of). Using Equation 11.2 we obtain an estimate of the experimental error variance under the assumption that the model is adequate. Therefore, to test whether the model is adequate we simply need to test the hypothesis... 

This value characterizes the upper level of relative scattering of estimated elements concentration in all the considered snow samples, which is associated both with estimation errors in groups 1-3 and natural variation of the elements abundance in samples. Thus, the results of successive testing of ypotheses Ht -H3 allow us to conclude that the basic hypothesis Hb is true and only global source of chemical contamination exists on the territory of Karabash. 

Space remains for only a brief glance at detection in higher dimensions. The basic concept of hypothesis testing and the central significance of measurement errors and certain model assumptions, however, can be carried over directly from the lower dimensional discussions. In the following text we first examine the nature of dimensionality (and its reduction to a scalar for detection decisions), and then address the critical issue of detection limit validation in complex measurement situations. 

Computer packages such as SAS can fit these models, provide estimates of the values of the b coefficients together with standard errors, and give p-values associated with the hypothesis tests of interest. These hypotheses will be exactly as Hqj, Hq2 and Hq3 in Section 6.3. Methods of stepwise regression are also available for the identification of a subset of the baseline variables/factors that are predictive of outcome. 

We will focus our attention to the situation of non-inferiority. Within the testing framework the type I error in this case is as before, the false positive (rejecting the null hypothesis when it is true), which now translates into concluding noninferiority when the new treatment is in fact inferior. The type II error is the false negative (failing to reject the null hypothesis when it is false) and this translates into failing to conclude non-inferiority when the new treatment truly is non-inferior. The sample size calculations below relate to the evaluation of noninferiority when using either the confidence interval method or the alternative p-value approach recall these are mathematically the same. 

In chemistry, as in many other sciences, statistical methods are unavoidable. Whether it is a calibration curve or the result of a single analysis, interpretation can only be ascertained if the margin of error is known. This section deals with fundamental principles of statistics and describes the treatment of errors involved in commonly used tests in chemistry. When a measurement is repeated, a statistical analysis is compulsory. However, sampling laws and hypothesis tests must be mastered to avoid meaningless conclusions and to ensure the design of meaningful quality assurance tests. Systematic errors (instrumental, user-based, etc.) and gross errors that lead to out-of-limit results will not be considered here. 

We note from Table 1.19 that the sums of squares between rows and between columns do not add up to the defined total sum of squares. The difference is called the sum of squares for error, since it arises from the experimental error present in each observation. Statistical theory shows that this error term is an unbiased estimate of the population variance, regardless of whether the hypotheses are true or not. Therefore, we construct an F-ratio using the between-rows mean square divided by the mean square for error. Similarly, to test the column effects, the F-ratio is the be-tween-columns mean square divided by the mean square for error. We will reject the hypothesis of no difference in means when these F-ratios become too much greater than 1. The ratios would be 1 if all the means were identical and the assumptions of normality and random sampling hold. Now let us try the following example that illustrates two-way analysis of variance. 

First, the hypothesis that the residuals Tmj represent a sample of the universe of the experimental errors can be tested via different methods in order to single out the presence of systematic errors deriving from the inadequacy of the mathematical model. In particular, it is possible to test whether, for any measured component m, the mean of the residuals Tmj (for j = 1,..., IVd) is significantly different from zero (which is the expected value) and whether, for any measured component, the corrected residual variance s m is significantly different from the universe variance er2, which can be computed by resorting to repeated measurements, as shown by (3.8). 

---