Nuisance parameters

Another aspect in which Bayesian methods perform better than frequentist methods is in the treatment of nuisance parameters. Quite often there will be more than one parameter in the model but only one of the parameters is of interest. The other parameter is a nuisance parameter. If the parameter of interest is 6 and the nuisance parameter is ( ), then Bayesian inference on 6 alone can be achieved by integrating the posterior distribution over ( ). The marginal probability of 6 is therefore... 

In frequentist statistics, by contrast, nuisance parameters are usually treated with point estimates, and inference on the parameter of interest is based on calculations with the nuisance parameter as a constant. This can result in large errors, because there may be considerable uncertainty in the value of the nuisance parameter. 

Consider now robustness. If the estimators A are computed from independent response variables then, as noted in Section 1, the estimators have equal variances and are usually at least approximately normal. Thus the usual assumptions, that estimators are normally distributed with equal variances, are approximately valid and we say that there is inherent robustness to these assumptions. However, the notion of robust methods of analysis for orthogonal saturated designs refers to something more. When making inferences about any effect A, all of the other effects At (k i) are regarded as nuisance parameters and robust means that the inference procedures work well, even when several of the effects ft are large in absolute value. Lenth s method is robust because the pseudo standard error is based on the median absolute estimate and hence is not affected by a few large absolute effect estimates. The method would still be robust even if one used the initial estimate 6 of op, rather than the adaptive estimator 6L, for the same reason. 

Most real world models often require multiparameter solutions. So, to characterize a normal distribution an estimate of the mean p and variance a2 is needed. In some instances, only a subset of parameters in the model may be of interest, e.g., only the mean of the distribution may be of interest, in which case the other parameters are considered nuisance parameters. In fitting a likelihood function, however, maximization is done relative to all model parameters. But, if only a subset of model parameters are of interest, a method needs to be available to concentrate the likelihood on the model parameters of interest and eliminate the nuisance parameters. 

Nuisance parameters are generally eliminated by computing the marginal likelihood. In the two-dimensional case, with random variables X and Y, the marginal likelihood can be obtained by integrating out the nuisance parameter from the joint likelihood. For example,... 

In practice, however, the nuisance parameters Cx, Uy and p are not known and have to be estimated if analysis of covariance is to be applied. In that case, the variance formula given by (7.6) is not quite right and must be multiplied by a further factor... 

An increasingly employed approach to conducting meta-analyses is to perform a Bayesian analysis. In fact, the most commonly employed analysis is not fully Bayesian but a hybrid. That is because, although prior distributions are employed for the treatment parameters, the nuisance parameters themselves are treated as fixed (Senn, 2007). That is to say that the variances of the treatment contrasts from various trials are treated as if they were known. This means that such analyses do not avoid many of the problems associated with frequentist approaches outlined in section 16.2.14. 

Fouladirad, M. and M. Nikiforov (2006). On line change detection with nuisance parameters. In Safeprocess, Bei-jin, China, CD ROM. 

Frequentist statistics have problems dealing with nuisance parameters, unless an ancillary statistic exists. 

Sometimes, only one of the parameters is of interest to us. We don t want to estimate the other parameters and call them "nuisance" parameters. All we want to do is make sure the nuisance parameters don t interfere with our inference on the parameter of interest. Because using the Baye.sian approach the joint posterior density is a probability density, and using the likelihood approach the joint likelihood function is not a probability density, the two approaches have different ways of dealing with the nuisance parameters. This is true even if we use independent flat priors. so that the posterior density and likelihood function have the same shape. 

Suppose that 0i is the parameter of interest, and 02 is a nuisance parameter. If there is an ancillary sufficient statistic, conditioning on it will give a likelihood that only depends on 0, the parameter of interest, and inference can be based on that conditional likelihood. This can only be true in certain exponential families, so is of limited general use when nuisance parameters are present. Instead, likelihood... 

Function of the data that is independent of the parameter of interest. Fisher developed ancillary statistics as a way to make inferences when nuisance parameters are present. However, it only works in the exponential family of densities so it cannot be used in the genend case. See Cox and Hinkley (1974). 

Bayesian statistics has a single way of dealing with nuisance parameters. Because the joint posterior is a probability density in all dimensions, we can find the marginal densities by integration. Inference about the parameter of interest 0i is based on the marginal posterior g 0i data), which is found by integrating the nuisance parameter 2 out of the joint posterior, a process referred to as marginalization ... 

The Bayesian posterior estimator for 0i found from the marginal posterior will be the same as that found from the joint posterior when we arc using the posterior mean as our estimator. For this example, the Bayesian posterior density of 0i found by marginalizing 02 out of the joint posterior density, and the profile likelihood function of 01 turn out to have the same shape. This will not always be the case. For instance, suppose we wanted to do inference on 02, and regarded 0i as the nuisance parameter. 

The two approaches have different ways of dealing with nuisance parameters. The likelihood approach often uses the profile likelihood where the maximum conditional likelihood value of the nuisance parameter is plugged into the joint likelihood. The Bayesian approach is to integrate the nuisance parameter out of the joint posterior. 

We are treating the parameter as a nuisance parameter, and marginalizing it out of the joint posterior. One of the advantages of the Bayesian approach is that we have a clear-cut way to find the predictive distribution that works in all circumstances. 

More realistically, both parameters are unknown, and then the observations would be from a two-dimensional exponential family. Usually we want to do our inference on the mean y and regard the variance as a nuisance parameter. Using a joint prior,... 

A more realistic model when we have a random sample of observations from a normalifi, cr ) distribution is that both parameters are unknown. The parameter fi is usually the only parameter of interest, and is a nuisance parameter. We want to do inference on /x while taking into account the additional uncertainty caused by the unknown value of tr. The Bayesian approach allows us to do this by marginalizing out the nuisance parameter from the joint posterior of the parameters given the data. The joint posterior will be proportional to the joint prior times the joint likelihood. 

We find the marginal posterior density for fi by marginalizing the nuisance parameter out of the joint posterior. 

When we treat the variance as a nuisance parameter, we find the marginal posterior density of /x is m plus time a Student s t with degrees of freedom. 

We observe a random sample of size 10 from a normal ii,a ) distribution where both the mean p and the variance are unknown. We con-sider the variance to be a nuisance parameter. The random sample is... 

Suppose we want to find the distribution for observing a new individual given some particular values of the predictor variables. The new observation given the parameters and the previous observations will not depend on the previous data since it is just another random observation from the data. Using the Bayesian approach we treat the parameters as nuisance parameters. If we knew the exact posterior distribution, we would find the joint posterior of the new observation and the parameters given the previous observations. Then we would marginalize out the parameters to find the predictive distribution of the new observation. Let j/ +i be the new observation. The predictive density of Vn+i given the observed j/i,..., j/n would be... 

In this section we look at using the Gibbs sampler for a simple situation where we have a random sample of size n from a normal fj., a ) distribution with both parameters unknown. To do Bayesian inference we will need to find a joint prior density for the two parameters. Usually, we want to make inferences about the mean /i, and regard cr as a nuisance parameter. There are two approaches we can take to choosing the joint prior distributions for this problem. 

In the previous example, we found the exact marginal posterior distribution for each of the individual means analytically. They were the parameters of interest and the hyperparameter was considered a nuisance parameter. In many cases we cannot find the posterior analytically. Instead we use the Gibbs sampler to draw a sample from the joint posterior of all the parameters for the hierarchical model. The Gibbs sample for a particular parameter is a sample from its marginal posterior. We will base our inference about that parameter on the thinned sample from its marginal posterior. We don t even have to marginalize out the other parameters as looking at the sample for that particular parameter does it automatically. 

The approach to the nuisance parameter are considered above was based on the use of an ordered parametrization whose first and second components were ((p,0)), respectively, referred as the parameter of interest and the nuisance parameter. The reference prior for the ordered... 

From Corollary of Proposition 3 where the nuisance parameter space k P) = a is independent of v, it is easy to... 

---