Error in significance testing

Cox DE (1992) High resolution powder diffraction and stmcture determination. In Coppens P (ed) Synchrotron Radiation Crystallography. Academic Press, London, p 186-254 Cox DE, Papoular RJ (1996) Stmcture refinement with synchrotron data R-factors, errors and significance tests. Mater Sci Foram 228 233-238... 

Reflux pumparound rates can usually be calculated from a heat balance around individual towers. This is the best way to account for unrecorded flows. It is not uncommon for a reflux or pumparound duty, calculated from the tower heat balance, to be 10% to 20% higher than the measured flow would indicate. This is probably due to ambient heat losses. For the sake of consistency of the total test report, it is best to stick with the duties from the heat balance calculations. If the difference between a duty calculated in the two ways described above is much more than 20%, there is a significant error in the test data. 

It must be noted that water and many reagents often contain traces of copper which introduce significant errors in comparative tests. Such traces can be removed from water by shaking it with calcium fluoride or talc and then separating the adsorbent by centrifuging. ... 

The uncertainty in detection is calculated as the probability that the damage is true after having detected a fault. This is established using the choice of the significance level a in the hypothesis test. The probabilities of different types of errors in hypothesis testing are well established in literature. The error of rejecting a correct null hypothesis is known as Type-I error, while the error of not rejecting a false null hypothesis is known as Type-II error. The probability of Type-I error is equal to a and the probability of Type-II error is denoted by p. This information can be written as ... 

An analytical procedure is often tested on materials of known composition. These materials may be pure substances, standard samples, or materials analyzed by some other more accurate method. Repeated determinations on a known material furnish data for both an estimate of the precision and a test for the presence of a constant error in the results. The standard deviation is found from Equation 12 (with the known composition replacing /x). A calculated value for t (Eq. 14) in excess of the appropriate value in Table 2.27 is interpreted as evidence of the presence of a constant error at the indicated level of significance. 

Analytical chemists make a distinction between error and uncertainty Error is the difference between a single measurement or result and its true value. In other words, error is a measure of bias. As discussed earlier, error can be divided into determinate and indeterminate sources. Although we can correct for determinate error, the indeterminate portion of the error remains. Statistical significance testing, which is discussed later in this chapter, provides a way to determine whether a bias resulting from determinate error might be present. 

Since significance tests are based on probabilities, their interpretation is naturally subject to error. As we have already seen, significance tests are carried out at a significance level, a, that defines the probability of rejecting a null hypothesis that is true. For example, when a significance test is conducted at a = 0.05, there is a 5% probability that the null hypothesis will be incorrectly rejected. This is known as a type 1 error, and its risk is always equivalent to a. Type 1 errors in two-tailed and one-tailed significance tests are represented by the shaded areas under the probability distribution curves in Figure 4.10. 

The probability of a type 1 error is inversely related to the probability of a type 2 error. Minimizing a type 1 error by decreasing a, for example, increases the likelihood of a type 2 error. The value of a chosen for a particular significance test, therefore, represents a compromise between these two types of error. Most of the examples in this text use a 95% confidence level, or a = 0.05, since this is the most frequently used confidence level for the majority of analytical work. It is not unusual, however, for more stringent (e.g. a = 0.01) or for more lenient (e.g. a = 0.10) confidence levels to be used. 

Relationship between confidence intervals and results of a significance test, (a) The shaded area under the normal distribution curves shows the apparent confidence intervals for the sample based on fexp. The solid bars in (b) and (c) show the actual confidence intervals that can be explained by indeterminate error using the critical value of (a,v). In part (b) the null hypothesis is rejected and the alternative hypothesis is accepted. In part (c) the null hypothesis is retained. 

Significance tests, however, also are subject to type 2 errors in which the null hypothesis is falsely retained. Consider, for example, the situation shown in Figure 4.12b, where S is exactly equal to (Sa)dl. In this case the probability of a type 2 error is 50% since half of the signals arising from the sample s population fall below the detection limit. Thus, there is only a 50 50 probability that an analyte at the lUPAC detection limit will be detected. As defined, the lUPAC definition for the detection limit only indicates the smallest signal for which we can say, at a significance level of a, that an analyte is present in the sample. Failing to detect the analyte, however, does not imply that it is not present. 

Alternatively, the experimental error can be given a particular value for each reaction of the series, or for each temperature, based on statistical evaluation of the respective kinetic experiment. The rate constants are then taken with different weights in further calculations (205,206). Although this procedure seems to be more exact and more profoundly based, it cannot be quite generally recommended. It should first be statistically proven by the F test (204) that the standard errors in fact differ because of the small number of measurements, it can seldom be done on a significant level. In addition, all reactions of the series are a priori of the same importance, and it is always a... 

By calibration on pHb at Tc the pH meter scale expresses the voltage in pH units at that temperature, so difficulties may arise when the test solution is measured at a deviating temperature T. If we assume for the present that the true pH value is not significantly altered by temperature variation (see later) and that the error in the pH read from the scale bears a linear relationship to the relative temperature difference we can correct pH, by... 

Because both quantities, xtest(i) and xmt(i ) are subject to error in this processing, EBV fitting according to Eqs. (6.41)-(6.43) or principal component analysis (Eq. 6.46) must be applied. The test on significant deviations from a = 0 and p = 1 are carried out as above. 

---