Frequentist inference

Frequentist methods are fundamentally predicated upon statistical inference based on the Central Limit Theorem. For example, suppose that one wishes to estimate the mean emission factor for a specific pollutant emitted from a specific source category under specific conditions. Because of the cost of collecting measurements, it is not practical to measure each and every such emission source, which would result in a census of the actual population distribution of emissions. With limited resources, one instead would prefer to randomly select a representative sample of such sources. Suppose 10 sources were selected. The mean emission rate is calculated based upon these 10 sources, and a probability distribution model could be fit to the random sample of data. If this process is repeated many times, with a different set of 10 random samples each time, the results will vary. The variation in results for estimates of a given statistic, such as the mean, based upon random sampling is quantified using a sampling distribution. From sampling distributions, confidence intervals are obtained. Thus, the commonly used 95% confidence interval for the mean is a frequentist inference... 

Most of the methods for analyzing data from supersaturated designs have been adapted from methods tailored for saturated or unsaturated designs. This might be a mistake as supersaturated designs are fundamentally different. Consider from first principles how we should carry out frequentist inference and Bayesian analysis. 

Inference based on the likelihood function using Fisher s ideas is essentially constructive. That means algorithms can be found to construct the solutions. Efron (1986) refers to the MLE as the "original jackknife" because it is a tool that can easily be adapted to many situations. The maximum likelihood estimator is invariant under a one-to-one reparameterization. Maximum likelihood estimators are compatible with the likelihood principle. Frequentist inference based on the likelihood function has some similarities with Bayesian inference as well as some differences. These similarities and differences will be explored in Section 3.3. 

Another aspect in which Bayesian methods perform better than frequentist methods is in the treatment of nuisance parameters. Quite often there will be more than one parameter in the model but only one of the parameters is of interest. The other parameter is a nuisance parameter. If the parameter of interest is 6 and the nuisance parameter is ( ), then Bayesian inference on 6 alone can be achieved by integrating the posterior distribution over ( ). The marginal probability of 6 is therefore... 

In frequentist statistics, by contrast, nuisance parameters are usually treated with point estimates, and inference on the parameter of interest is based on calculations with the nuisance parameter as a constant. This can result in large errors, because there may be considerable uncertainty in the value of the nuisance parameter. 

The standard tools of statistical inference, including the concept and approaches of constructing a null hypotheses and associated p values, are based on the frequentist view of probability. From a frequentist perspective, the probability of an event is defined as the fraction of times that the event occurs in a very large number of trials (known as a probability limit). Given a hypothesis and data addressing it, the classical procedure is to calculate from the data an appropriate statistic, which is typically... 

Throughout this book, the approach taken to hypothesis testing and statistical analysis has been a frequentist approach. The name frequentist reflects its derivation from the definition of probability in terms of frequencies of outcomes. While this approach is likely the majority approach at this time, it should be noted here that it is not the only approach. One alternative method of statistical inference is the Bayesian approach, named for Thomas Bayes work in the area of probability. 

Statistical methods that are based upon analysis of empirical data without prior assumptions about the type and parameter of distributions are typically termed frequentist methods, although sometimes the term classical is used (e.g. Morgan Henrion, 1990 Warren-Hicks Butcher, 1996 Cullen Frey, 1999). However, the term classical is sometimes connotated with thought experiments (e.g. what happens with a roll of a die) as opposed to inference from empirical data (DeGroot, 1986). Therefore, we use the term frequentist . 

Data are not random but are representative in other ways. This may mean, for example, that the data are a stratified sample applicable to the real-world situation for the assessment scenario of interest. In this case, frequentist methods can be used to make inferences for the strata that are represented by the data (e.g. particular exposed subpopulations), but not necessarily for all aspects of the scenario. However, for the components of the scenario for which the data cannot be applied, there is a lack of representative data. For example, if the available data represent one subpopulation, but not another, frequentist methods can be applied to make inferences about the former, but could lead to biased estimation of the latter. Bias correction methods, such as comparison with benchmarks, use of surrogate (analogous) data or more formal application of expert judgement, may be required for the latter. 

As Morgan Henrion (1990) point out, for many quantities of interest in models used for decision-making, there may not be a relevant population of trials of similar events upon which to perform frequentist statistical inference. For example, some events may be unique or in the future, for which it is not possible to obtain empirical sample data. Thus, frequentist statistics are powerful with regard to their domain of applicability, but the domain of applicability is limited compared with the needs of analysts attempting to perform studies relevant to the needs of decision-makers. 

The cornerstone of Bayesian methods is Bayes Theorem, which was first published in 1763 (Box Tiao, 1973). Bayes Theorem provides a method for statistical inference in which a prior distribution, based upon subjective judgement, can be updated with empirical data, to create a posterior distribution that combines both judgement and data. As the sample size of the data becomes large, the posterior distribution will tend to converge to the same result that would be obtained with frequentist methods. In situations in which there are no relevant sample data, the analysis can be conducted based upon the prior distribution, without any updating. 

So, from a frequentist viewpoint, supersaturated designs do not allow us to carry out any useful estimation or inference. However, estimation and inference are not the objectives of running supersaturated designs. Identifying the dominant factors is the objective. The estimation- and inference-based methods that have been recommended for analysis are used indirectly for this objective, but there is no reason to assume that they should be good for this. It is important to recognize that data analysis from supersaturated designs should be exploratory and not inferential. 

This conflict will appear in various places, in particular in the chapter on sequential analysis, and the differences between the approaches are discussed in Chapter 4. However, it is not specifically covered here in the sense that this is not a book which tries to evaluate the general claims of either Bayesian or frequentist (classical) statistics to be the theory of statistical inference. 

R.A. Fisher s own views on inference are outlined in Fisher (1956) For a modern text covering frequentist approaches see Cox (2006) and for a mathematically elementary but otherwise profound outline of the Bayesian approach see Bindley (2006). Likelihood approaches are covered in Lindsey (1996), Royall (1999), Pawitan (2001) and the classic by Edwards (1992). Classic expositions of the fully subjective Bayesian view of probability are give by Savage (1954) and de Finetti (1974, 1975) and the case for its use in science is made by the philosophers Howson and Urbach (1993). A comparative overview of the various statistical systems is given by Barnett (1999). 

Both major schools of statistical inference, the Bayesian and the frequentist, require prior consideration of possible analyses at the time trials are run. The Bayesian school requires specification of prior probabilities. To make these genuine priors they have to be truly prior and hence determined before the data are obtained. The frequentist school requires specification of the tests that will be performed and the decisions that will be made depending on various outcomes. For both schools, in order to design salient experiments, it is necessary to pay attention to very many aspects of design. 

It is often argued that formal analyses of safety data are inappropriate, usually because pre-specified hypotheses are not available. This raises the issues of multiplicity and intentions (in designing the experiment) for a frequentist analysis. For a Bayesian analysis the equivalent problem is raised of having to establish prior distributions after the data are in. Neither mode of inference deals well with carrying out post-hoc analyses. 

In the area of statistical analysis, there are two main parties, mra y frequency probability and Bayesian inference. Frequentists define probability (of an event) as the limit of its relative... 

A classical statistical perspective is supported by BES when (i) one can motivate the existence of a statistical model of the processes captured by the simulator and (ii) there is a lot of data to support all parameters in the Monte Carlo simulation. The first condition happens when observations have been, or can (within reasonable time) be made, on similar events that are to be assessed. This would mean that the right hand side in Figure 3 is included in the left hand side. If the second condition is met, classical statistical frequentistic and Bayesian principles of inference result in precise and similar parameters values. Therefore the difference in quantified uncertainty in output, under these two principles, can be small and the BES could be seen as supporting a classical statistical approach. 

Nevertheless, what currently passes for frequentist parametric statistics includes a collection of techniques, concepts, and methods from each of these two schools, despite the disagreements between the founders. Perhaps this is because, for the very important cases of the normal distribution and the binomial distribution, the MLE and the UMVUE coincided. Efron (1986) suggested that the emotionally loaded terms (unbiased, most powerful, admissible, etc.) contributed by Neyman, Pearson, and Wald reinforced the view that inference should be based on likelihood and this reinforced the frequentist dominance. Frequentist methods work well in the situations for which they were developed, namely for exponential families where there are minimal sufficient statistics. Nevertheless, they have fundamental drawbacks including ... 

Bayes theorem is the only consistent way to modify our belief about the parameters given the data that actually occurred. A Bayesian inference depends only on the data that occurred, not on the data that could have occurred but did not. Thus, Bayesian inference is consistent with the likelihood principle, which states that if two outcomes have proportional likelihoods, then the inferences based on the two outcomes should be identical. For a discussion of the likelihood principle see Bernardo and Smith (1994) or Pawitan (2001). In the next section we compare Bayesian inference with likelihood inference, a frequentist method of inference that is based solely on the likelihood function. As its name implies, it also satisfies the likelihood principle. 

Statistical inferences such as point estimation, confidence intervals, and hypothesis testing developed under the frequentist framework use the sampling distribution of the statistic given the unknown parameter. They answer questions about where we are in the parameter dimension using a probability distribution in the observation dimension. 

It is important to note that in Bayesian statistics, the Bayesian interpretation of probability is pertained as opposed to the classical/rcfyueufisf interpretation. In the frequentist interpretation, the probability attributed to a random variable is seen as a measure for the long-term fi equency of occurrence of that variable. The Bayesian interpretation is a subjective interpretation where probability reflects a measure of plausibility or degree of belief attributed to a variable, given the current state of informatiOTi. It is clear that only the Bayesian interpretation is meaningful in the context of forming inferences on model parameters using observed data. 

---