Error/statistical significance confidence

The distribution of the /-statistic (x — /ji)s is symmetrical about zero and is a function of the degrees of freedom. Limits assigned to the distance on either side of /x are called confidence limits. The percentage probability that /x lies within this interval is called the confidence level. The level of significance or error probability (100 — confidence level or 100 — a) is the percent probability that /X will lie outside the confidence interval, and represents the chances of being incorrect in stating that /X lies within the confidence interval. Values of t are in Table 2.27 for any desired degrees of freedom and various confidence levels. 

The split-sample method is often used with so few samples in the test set, however, that the validation is almost meaningless. One can evaluate the adequacy of the size of the test set by computing the statistical significance of the classification error rate on the test set or by computing a confidence interval for the test set error rate. Because the test set is separate from the training set, the number of errors on the test set has a binomial distribution. 

ANOVA of the data confirms that there is a statistically significant relationship between the variables at the 99% confidence level. The i -squared statistic indicates that the model as fitted explains 96.2% of the variability. The adjusted P-squared statistic, which is more suitable for comparing models with different numbers of independent variables, is 94.2%. The prediction error of the model is less than 10%. Results of this protocol are displayed in Table 18.1. Validation results of the regression model are displayed in Table 18.2. 

By comparing absolute values of regression coefficients with errors in their estimates, it becomes evident that all regression coefficients are statistically significant with 0.95% confidence, except for bn and b22. A check of lack of fit of the obtained regression model proved that it is adequate with 95% confidence (FR[Pg.332]

All regression coefficients except bn and b22 are with 95 % confidence statistically significant since they are above the errors in their estimations (Ab). 

Results of these tests are shown in Figure 2. We found no statistical justification in any case for slopes different from 1.0. Using the Student-t test, there is no statistically significant difference, at the 90% confidence level, between the observed intercept and the atomic weight in Equation 4 for the Hg/Pb or Hg/Zn data sets. The standard error of an estimated entropy, using Pb(II) compounds as references, is about 21 JK mol" (at 95% confidence level). 

The statistical testing setup, as we have already seen, is geared toward the declaration of statistical significance. When a test is significant, we draw a conclusion about the cause of the effect of interest. If we decide to reject the null hypothesis, the p-value is the type I error associated with this decision. Therefore, the level of confidence in the correctness of the decision depends on the p-value the smaller the p-value, the more confident one is that the decision is correct. 

The choice of a rejection region for the null hypothesis is made so that we can readily understand the errors involved. At the 95% confidence level, for example, there is a 5% chance that we will reject the null hypothesis even though it is true. This could happen if an unusual result occurred that put our test statistic z or t into the rejection region. The error that results from rejecting when it is true is called a type I error. The significance level a gives the frequency of rejecting Hq when it is true. 

The confidence intervals provide information on the likelihood of falling into one of these errors. However, the person interpreting the efficacy results must decide, as a guide for action, what target difference and what probability level (for either type of error) he or she will accept when using the results. The statistical significance test alone will not provide this... 

The pooled variance calculated from the 16 duplicate runs is 0.9584. The variance of any effect will be one-eighth of this value, which is 0.1198. The square root of this last value is the standard error of an effect. Multiplying it by tie, we arrive at the limiting value for the statistical significance of the absolute value of an effect, 0.734 (95% confidence level). 

From the three experiments at the center point, we obtain an estimate of 0.40 for the standard error of the response, which in this case is the same as the standard error of an effect. Therefore, the limiting value for statistical significance of the absolute value of an effect (at the 95% confidence level) will be given by... 

Exercise 5.11. The result of Exercise 5.10 should show that there is no evidence of lack of fit in the model of Exercise 5.5. Use the residual mean square as an estimate of the variance of the observations and determine the standard errors of the coefficient estimates. Are they statistically significant at the 95% confidence level ... 

Exercise 6.10. Knowing that the standard error estimate was obtained from the value of MSpe in Table 6.10, find, at the 95% confidence level, the statistically significant coefficients in Eq. (6.10). 

---