Statistical significance test

Analytical chemists make a distinction between error and uncertainty Error is the difference between a single measurement or result and its true value. In other words, error is a measure of bias. As discussed earlier, error can be divided into determinate and indeterminate sources. Although we can correct for determinate error, the indeterminate portion of the error remains. Statistical significance testing, which is discussed later in this chapter, provides a way to determine whether a bias resulting from determinate error might be present. 

Table V. Results from blend food questionnaire for corn-cottonseed (CC) and modified corn-soy-milk (MCSM). Statistical significance tested by blend type (BT), by clinic (CN), and by interaction of blends on clinic...

When using such a statistical significance test, it is important to recognise that this generally has low power in a trial designed to detect the main effect of treatment/... 

When an unreplicated experiment is run, the error or residual sum of squares is composed of both experimental error and lack-of-fit of the model. Thus, formal statistical significance testing of the factor effects can lead to erroneous conclusions if there is lack-of-fit of the model. Therefore, it is recommended that the experiment be replicated so that an independent estimate of the experimental error can be calculated and both lack-of-fit and the statistical significance of the factor effects can be formally tested. 

Figure 1 provides adjustments to critical values for CV p when a method is biased. The dotted curve gives critical values of CV-p as a function of bias for a statistical significance test performed at the 5% probability level. Because uniform replicate determinations of the bias were not made in the validation tests, the bias is treated as a known constant rather than an estimated value. The experimental design could be modified to permit determination of the imprecision in the bias by providing for uniform replication of the independent method as well as the method under evaluation. Then the decision chart could be modified to include allowance for variability of replicate bias determinations. 

Statistical Significance Testing and Cumulative Knowledge in Psychology Implications for Training of Researchers (p. 126) Volume 1, Number 2,1996... 

A statistical significance test (e.g. the Student s t test, the Chi-Square test) will tell how often an observed difference would occur due to chance (random influences) if there is, in reality, no difference between the treatments. Where the statistical significance test shows that an observed difference would only occur five times if the experiment were repeated 100 times, this is often taken as sufficient evidence that the null hypothesis is unlikely to be true. Therefore the conclusion is that there is (probably) a real difference between the treatments. This level of probability is generally expressed in therapeutic trials as the difference was statistically significant, or significant at the 5% level or, P = 0.05 (P = probability based on chance alone). Statistical significance simply means that the result is unlikely to have occurred if there is no genuine treatment difference, i.e. there probably is a difference. 

Interim analyses can reduce the power of statistical significance tests to a serious degree if they are scheduled to occur more than, say, about four times in a trial. Such sequential designs recognise the reality of medical practice and provide a reasonable balance between statistical, medical and ethical needs. It is a necessity to have expert statistical advice when undertaking such trials poorly designed and executed studies cannot be salvaged after the event. 

As a general rule the NOEL/NOAEL is a dose from a controlled animal experiment where no adverse effect (i.e., an effect not considered harmful) is noted. The experiment does not establish that no effect can possibly occur at that dose under any conditions - it only denotes that none of the effects looked for in the experiment was observed. Since a statistical significance test is typically used to establish whether or not an effect occurred, the NOEL/ NOAEL will tend to diminish as the sensitivity of the measurement or the number of observations increases. Since the burden of proof is on science to show that an effect has occurred, greater uncertainty tends to raise the level of exposure that is deemed accept-able/tolerable. The selection of the effect that is considered adverse is a matter of societal values (i.e., localized, reversible, mild discomfort versus frank, irreversible systemic toxicity). That is, establishing that an effect has occurred is a separate consideration from how much one cares if it will occur or not. 

One outcome of the dependence of the NOEL/ NOAEL on the statistical significance test is that it tends to penalize chemicals for which there is more or better data. To remedy this problem, the benchmark dose (BMD) concept was introduced as an alternative approach. The BMD depends on the specification of a low level effect that would typically be unobservable. The endpoint may be the specified percentage (5 or 10%) above background of a population for an endpoint deemed to be adverse. Since the endpoint is defined, determinations for different chemicals and different data sets tend to be more comparable. 

The confidence intervals provide information on the likelihood of falling into one of these errors. However, the person interpreting the efficacy results must decide, as a guide for action, what target difference and what probability level (for either type of error) he or she will accept when using the results. The statistical significance test alone will not provide this... 

Barroso, J.M. and Besalu, E. (2005) Design of experiments applied to QSAR ranking a set of compounds and establishing a statistical significance test. /. Mol Struct. (Theochem), 717, 89-96. 

Fisher E-test A statistical significance test which decides whether there is a significant difference between two variances (and therefore two sample standard deviations). This test is used in ANOVA. For two standard deviations X and s2, F=s /s where, V >,v2. (Sections 3.7, 4.4)... 

Null hypothesis (H0) The hypothesis that the population parameters being compared (e.g., mean or variance) on the basis of the data are the same, and the observed differences arise from random variation only. This is the hypothesis used in many statistical significance tests that there is no difference between the factors that are being compared. (The null hypothesis is first introduced in section 3.2 but is used throughout chapters 3 and 4). (Section 3.2)... 

Paired r-tcst A statistical significance test for comparing two sets of data where there are no repeat measurements of a single test material but there are single measurements of a number of different test samples. To perform this test you use t = ( v(i, /h/yi) where x,, , v( are the mean and standard deviation of n differences. (Section 3.9)... 

Based on the above two statistical significance test methods, it was found that there is no significant difference between ICP XRF methods for the analysis of Al, Ni, V, Ti Fe. However, it is also important in such cases to compare the standard deviations, i.e., the random errors of two sets of data. This comparison can take two forms, viz., whether ICP method is more precise than XRF method (one-tailed F-test) or ICP XRF differ in their preeision (two-tailed F-test). 

The most powerful use of the STEM imaging capabilities is to combine information from many sources into a useful model of a multiunit complex. This is done by comparing many possible models with actual STEM images. Models with too many or too few subunits or units in incorrect places can be ruled out by statistically significant tests. 

The most typical analytical approach to interpreting the data from scientific studies, including clinical trials, is the statistical significance test, also known as the null hypothesis significance test (NHST). This approach, which has recently come under much scrutiny and debate, formally... 

In the realm of statistical significance testing, there are typically several tests for each type of hypothesis. The Grubbs test, recommended by the International Standards Organization (ISO) and the American Society for Testing and Materials (ASTM), is another approach to the identification of outliers ... 

All connected subnetworks containing three nodes in the interaction network are collated into isomorphic patterns, and the number of times each pattern occurs is counted. If the number of occurrences is at least five and significantly higher than in randomized networks, the pattern is considered as a NM. The statistical significance test is performed by... 

---