Significance testing

Analytical chemists make a distinction between error and uncertainty Error is the difference between a single measurement or result and its true value. In other words, error is a measure of bias. As discussed earlier, error can be divided into determinate and indeterminate sources. Although we can correct for determinate error, the indeterminate portion of the error remains. Statistical significance testing, which is discussed later in this chapter, provides a way to determine whether a bias resulting from determinate error might be present. 

Next, an equation for a test statistic is written, and the test statistic s critical value is found from an appropriate table. This critical value defines the breakpoint between values of the test statistic for which the null hypothesis will be retained or rejected. The test statistic is calculated from the data, compared with the critical value, and the null hypothesis is either rejected or retained. Finally, the result of the significance test is used to answer the original question. 

A statement that the difference between two values can be explained by indeterminate error retained if the significance test does not fail Ho). 

Examples of (a) two-tailed, (b) and (c) one-tailed, significance tests. The shaded areas in each curve represent the values for which the null hypothesis is rejected. 

Significance test in which the null hypothesis is rejected for values at either end of the normal distribution. 

If the significance test is conducted at the 95% confidence level (a = 0.05), then the null hypothesis will be retained if a 95% confidence interval around X contains p,. If the alternative hypothesis is... 

Since significance tests are based on probabilities, their interpretation is naturally subject to error. As we have already seen, significance tests are carried out at a significance level, a, that defines the probability of rejecting a null hypothesis that is true. For example, when a significance test is conducted at a = 0.05, there is a 5% probability that the null hypothesis will be incorrectly rejected. This is known as a type 1 error, and its risk is always equivalent to a. Type 1 errors in two-tailed and one-tailed significance tests are represented by the shaded areas under the probability distribution curves in Figure 4.10. 

The probability of a type 1 error is inversely related to the probability of a type 2 error. Minimizing a type 1 error by decreasing a, for example, increases the likelihood of a type 2 error. The value of a chosen for a particular significance test, therefore, represents a compromise between these two types of error. Most of the examples in this text use a 95% confidence level, or a = 0.05, since this is the most frequently used confidence level for the majority of analytical work. It is not unusual, however, for more stringent (e.g. a = 0.01) or for more lenient (e.g. a = 0.10) confidence levels to be used. 

The most commonly encountered probability distribution is the normal, or Gaussian, distribution. A normal distribution is characterized by a true mean, p, and variance, O, which are estimated using X and s. Since the area between any two limits of a normal distribution is well defined, the construction and evaluation of significance tests are straightforward. 

A typical application of this significance test, which is known as a f-test of A to p, is outlined in the following example. 

Relationship between confidence intervals and results of a significance test, (a) The shaded area under the normal distribution curves shows the apparent confidence intervals for the sample based on fexp. The solid bars in (b) and (c) show the actual confidence intervals that can be explained by indeterminate error using the critical value of (a,v). In part (b) the null hypothesis is rejected and the alternative hypothesis is accepted. In part (c) the null hypothesis is retained. 

Since there is no reason to believe that X must be either larger or smaller than p, the use of a two-tailed significance test is appropriate. The null and alternative hypotheses are... 

The variance for the sample of ten tablets is 4.3. A two-tailed significance test is used since the measurement process is considered out of statistical control if the sample s variance is either too good or too poor. The null hypothesis and alternative hypotheses are... 

Significance testing for comparing two mean values is divided into two categories depending on the source of the data. Data are said to be unpaired when each mean is derived from the analysis of several samples drawn from the same source. Paired data are encountered when analyzing a series of samples drawn from different sources. 

The value of fexp is compared with a critical value, f(a, v), as determined by the chosen significance level, a, the degrees of freedom for the sample, V, and whether the significance test is one-tailed or two-tailed. 

The value of fexp is then compared with a critical value, f(a, v), which is determined by the chosen significance level, a, the degrees of freedom for the sample, V, and whether the significance test is one-tailed or two-tailed. For paired data, the degrees of freedom is - 1. If fexp is greater than f(a, v), then the null hypothesis is rejected and the alternative hypothesis is accepted. If fexp is less than or equal to f(a, v), then the null hypothesis is retained, and a significant difference has not been demonstrated at the stated significance level. This is known as the paired f-test. 

On occasion, a data set appears to be skewed by the presence of one or more data points that are not consistent with the remaining data points. Such values are called outliers. The most commonly used significance test for identifying outliers is Dixon s Q-test. The null hypothesis is that the apparent outlier is taken from the same population as the remaining data. The alternative hypothesis is that the outlier comes from a different population, and, therefore, should be excluded from consideration. 

Significance tests, however, also are subject to type 2 errors in which the null hypothesis is falsely retained. Consider, for example, the situation shown in Figure 4.12b, where S is exactly equal to (Sa)dl. In this case the probability of a type 2 error is 50% since half of the signals arising from the sample s population fall below the detection limit. Thus, there is only a 50 50 probability that an analyte at the lUPAC detection limit will be detected. As defined, the lUPAC definition for the detection limit only indicates the smallest signal for which we can say, at a significance level of a, that an analyte is present in the sample. Failing to detect the analyte, however, does not imply that it is not present. 

This value of fexp is compared with the critical value for f(a, v), where the significance level is the same as that used in the ANOVA calculation, and the degrees of freedom is the same as that for the within-sample variance. Because we are interested in whether the larger of the two means is significantly greater than the other mean, the value of f(a, v) is that for a one-tail significance test. 

Ohm s law the statement that the current moving through a circuit is proportional to the applied potential and inversely proportional to the circuit s resistance (E = iR). (p. 463) on-column injection the direct injection of thermally unstable samples onto a capillary column, (p. 568) one-taUed significance test significance test in which the null hypothesis is rejected for values at only one end of the normal distribution, (p. 84)... 

---