Interpreting statistical significance

Just how strong is the evidence when we obtain a statistically significant result It has already been emphasized on a number of occasions that it is not proof absolute. Even 

Essential Statistics for the Pharmaceutical Sciences Philip Rowe 

CH11 WHAT DOES A STATISTICALLY SIGNIFICANT RESULT REALLY TELL US  

1 You cannot simply interpret a confidence interval as providing any set level of proof 

---

Because the entire subject of LFER is empirical in nature, these attempted extensions of the field may be justified, provided they meet reasonable criteria of chemical and statistical significance. Whether or not they will successfully extend our ability to correlate data or will lead to improved physical insight must be established by further effort. For the present, however, the experimentalist probably should base interpretations on the quantities and concepts outlined earlier in this section, for the essential worth of these simpler ideas has been established by example and practice. 

Notice that in comparisons such as these sometimes slight inconsistencies in the results can be obtained. In two cases A was considered better than B, and B better than C, yet C was judged superior to A This inconsistency or non-transitivity is known as Simpson s or de Condorcet s paradox. In this particular case it can perfectly well be attributed to random variation. Assessors who are not sure about their conclusion are forced to make a choice, which then can only be a random guess. It is possible, however, to obtain results which are conflicting and statistically significant at the same time A < B and B < C, but C < A. This situation may occur when the attribute to be assessed in the comparisons is open to different interpretations. Actually, this is a case of multicriteria decision making (see Chapter 26) and it may be impossible to rank the three products unambiguously... 

If the probability distribution of the data is or assumed Gaussian, several statistical measures are available for interpreting the data. These measures can be used to interpret the latent variables determined by a selected data analysis method. Those described here are a combination of statistical measures and graphical analysis. Taken together they provide an assessment of the statistical significance of the analysis. 

In this example, data interpretations are based on g-statistic limits. These are computed by assuming the data are normally distributed in the multivariate sense. The diagnostic limits are used to establish when a statistically significant shift has occurred. Charts based on these statistics and used in this manner are analogous to conventional SPC charts. 

Johnson, Blair T. and Irving Kirsch, Interpreting the Efficacy of Antidepressants Statistical Significance Versus Clinical Benefits , Significance 5 (2008) 54-58... 

The Chi-square test statistic is 8,01, giving a significance level of 0.09, implying that there is no statistically significant relation between the results of the different companies, as shown in figure 43. However, from a practical perspective the differences, if they are real, might be of importance, therefore the results and possible interpretations will be discussed in more detail in sub-Sections 7.2.7 and 7.3. 

As a point of departure toward this objective, it is essential that any analysis of study results be interpreted by a professional who firmly understands three concepts the difference between biological significance and statistical significance, the nature and value of different types of data, and causality. 

It is important to appreciate that the statistical significance of the results is wholly dependent on the quality of the data obtained from the trial. Data that contain obvious gross errors should be removed prior to statistical analysis. It is essential that participants inform the trial co-ordinator of any gross error that they know has occurred during the analysis and also if any deviation from the method as written has taken place. The statistical parameters calculated and the outlier tests performed are those used in the internationally agreed Protocol for the Design, Conduct and Interpretation of Collaborative Studies.14... 

Before this topic is left behind, it should be noted that statistical significance is by no means the only consideration in interpretation of toxicity test results. If, in our particular case, the pathologist were to inform us that the brain lesion observed was extremely unusual or rare, we should certainly hesitate to dismiss our concerns because of lack of statistical significance. The toxicologist needs equally to understand biological significance, and, in this case, would almost certainly pursue other lines of investigation (perhaps an ADME study to determine if the pesticide reaches the brain, or a toxicity test in other species) to determine whether the effect was truly caused by the chemical. 

Statistically significant changes in electroencephalogram (EEG) patterns were reported in female adult Wistar rats administered heptachlor in the diet at levels of 1 and 5 mg/kg/day for three generations (Formanek et al. 1976). Interpretation of these findings is difficult because details of the dosing, the procedures used, and conditions of the rats were not described. 

Trends displayed by measurements should be interpreted statistically and numerical targets should be established. The measurements are reported and reviewed and trends are evaluated. The cost of measurement is significant and should always be considered. 

---