Test statistic choices

The choice among the three sets depends upon what we wish to test. If we wish to show that p is higher than p0, we use the first test. If we wish to test whether p is less than p0, we use the second set. To show that p is simply unequal to p0, we use the third set with a two-sided critical region. All of these sets use the test statistic ... 

The choice of a rejection region for the null hypothesis is made so that we can readily understand the errors involved. At the 95% confidence level, for example, there is a 5% chance that we will reject the null hypothesis even though it is true. This could happen if an unusual result occurred that put our test statistic z or t into the rejection region. The error that results from rejecting when it is true is called a type I error. The significance level a gives the frequency of rejecting Hq when it is true. 

One shortcoming of the hypothesis testing approach is the arbitrary choice of a value for a. Depending upon our risk tolerance for committing a type 1 error, the conventional value of 0.05 may not be acceptable. Another way to convey the "extremeness" of the resulting test statistic is to report a p value. 

In contrast to the hypothesis testing style of model selection/discrimination, the posterior predictive check (PPC) assesses the predictive performance of the model. This approach allows the user to reformulate the model selection decision to be based on how well the model performs. This approach has been described in detail by Gelman et al. (27) and is only briefly discussed here. PPC has been assessed for PK analysis in a non-Bayesian framework by Yano et al. (40). Yano and colleagues also provide a detailed assessment of the choice of test statistics. The more commonly used test statistic is a local feature of the data that has some importance for model predictions for example, the maximum or minimum concentration might be important for side effects or therapeutic success (see Duffull et al. (6)) and hence constitutes a feature of the data that the model would do well to describe accurately. The PPC can be defined along the fines that posterior refers to conditioning of the distribution of the parameters on the observed values of the data, predictive refers to the distribution of future unobserved quantities, and check refers to how well the predictions reflect the observations (41). This method is used to answer the question Does the observed data look plausible under the posterior distribution This method is therefore solely a check of internal consistency of the model in question. 

The point about this example is that all analytical chemistry should be fit for purpose. When you make a decision based on a statistical test, the choice of the probability level at which the null hypothesis is rejected is made by the user, not by a book or software package. Do not adopt probability levels blindly, but consider the risk of making the different types of error. 

Test Statistic A test statistic is a quantity calculated from the experimental data set, that is used to decide whether or not the nidi hypothesis should be rejected in a hypothesis test. The choice of a test statistic depends on the assumed probability model and the hypotheses under question common choices are the Student s-t and Fisher s F parameters. 

Revealed Preference Research, whereby people s actual behaviour is observed, and a model to summarise these choices is inferred and tested statistically. This has the advantage that it is based on real life data, but the disadvantage that it may be difficult to get hold of these data with reasonable accuracy. 

The information obtained during the background search and from the source inspection will enable selection of the test procedure to be used. The choice will be based on the answers to several questions (1) What are the legal requirements For specific sources there may be only one acceptable method. (2) What range of accuracy is desirable Should the sample be collected by a procedure that is 5% accurate, or should a statistical technique be used on data from eight tests at 10% accuracy Costs of different test methods will certainly be a consideration here. (3) Which sampling and analytical methods are available that will give the required accuracy for the estimated concentration An Orsat gas analyzer with a sensitivity limit of 0.02% would not be chosen to sample carbon monoxide... 

Statistical and algebraic methods, too, can be classed as either rugged or not they are rugged when algorithms are chosen that on repetition of the experiment do not get derailed by the random analytical error inherent in every measurement,i° 433 is, when similar coefficients are found for the mathematical model, and equivalent conclusions are drawn. Obviously, the choice of the fitted model plays a pivotal role. If a model is to be fitted by means of an iterative algorithm, the initial guess for the coefficients should not be too critical. In a simple calculation a combination of numbers and truncation errors might lead to a division by zero and crash the computer. If the data evaluation scheme is such that errors of this type could occur, the validation plan must make provisions to test this aspect. 

The limit of detection (LoD) has already been mentioned in Section 4.3.1. This is the minimum concentration of analyte that can be detected with statistical confidence, based on the concept of an adequately low risk of failure to detect a determinand. Only one value is indicated in Figure 4.9 but there are many ways of estimating the value of the LoD and the choice depends on how well the level needs to be defined. It is determined by repeat analysis of a blank test portion or a test portion containing a very small amount of analyte. A measured signal of three times the standard deviation of the blank signal (3sbi) is unlikely to happen by chance and is commonly taken as an approximate estimation of the LoD. This approach is usually adequate if all of the analytical results are well above this value. The value of Sbi used should be the standard deviation of the results obtained from a large number of batches of blank or low-level spike solutions. In addition, the approximation only applies to results that are normally distributed and are quoted with a level of confidence of 95%. 

Contrasted with these continuous data, however, we have discontinuous (or discrete) data, which can only assume certain fixed numerical values. In these cases our choice of statistical tools or tests is, as we will find later, more limited. 

A comparison of the imprecision of two methods may assist in the choice of one for routine use. Statistical comparison of values for the standard deviation using the F test (Procedure 1.2) may be used to compare not only different methods but also the results from different analysts or laboratories. Some caution has to be exercised in the interpretation of statistical data and particularly in such tests of significance. Although some statistical tests are outlined in this book, anyone intending to use them is strongly recommended to read an appropriate text on the subject. 

Clearly, any measurement that differentiates between the properties of high and low temperature forms of H20(as), and/or delineates the relationship between H20(as) and liquid H20, can be used to test the hypotheses advanced vis a vis their structures. These and the experimental tests suggested, together with the construction of continuous random network models more sophisticated than that for Ge(as), the increased use of computer simulation, and exploitation of the available experimental information to guide the choice of appproximations in a statistical mechanical theory should increase our understanding of H20(as) and, uitimately, liquid H20. 

Section 1.6.2 discussed some theoretical distributions which are defined by more or less complicated mathematical formulae they aim at modeling real empirical data distributions or are used in statistical tests. There are some reasons to believe that phenomena observed in nature indeed follow such distributions. The normal distribution is the most widely used distribution in statistics, and it is fully determined by the mean value p. and the standard deviation a. For practical data these two parameters have to be estimated using the data at hand. This section discusses some possibilities to estimate the mean or central value, and the next section mentions different estimators for the standard deviation or spread the described criteria are fisted in Table 1.2. The choice of the estimator depends mainly on the data quality. Do the data really follow the underlying hypothetical distribution Or are there outliers or extreme values that could influence classical estimators and call for robust counterparts ... 

---