Normal distribution likelihood

If this criterion is based on the maximum-likelihood principle, it leads to those parameter values that make the experimental observations appear most likely when taken as a whole. The likelihood function is defined as the joint probability of the observed values of the variables for any set of true values of the variables, model parameters, and error variances. The best estimates of the model parameters and of the true values of the measured variables are those which maximize this likelihood function with a normal distribution assumed for the experimental errors. 

Step 1. From a histogram of the data, partition the data into N components, each roughly corresponding to a mode of the data distribution. This defines the Cj. Set the parameters for prior distributions on the 6 parameters that are conjugate to the likelihoods. For the normal distribution the priors are defined in Eq. (15), so the full prior for the n components is... 

The principle of Maximum Likelihood is that the spectrum, y(jc), is calculated with the highest probability to yield the observed spectrum g(x) after convolution with h x). Therefore, assumptions about the noise n x) are made. For instance, the noise in each data point i is random and additive with a normal or any other distribution (e.g. Poisson, skewed, exponential,...) and a standard deviation s,. In case of a normal distribution the residual e, = g, - g, = g, - (/ /i), in each data point should be normally distributed with a standard deviation j,. The probability that (J h)i represents the measurement g- is then given by the conditional probability density function Pig, f) ... 

The above implicit formulation of maximum likelihood estimation is valid only under the assumption that the residuals are normally distributed and the model is adequate. From our own experience we have found that implicit estimation provides the easiest and computationally the most efficient solution to many parameter estimation problems. 

The values of the elements of the weighting matrices R, depend on the type of estimation method being used. When the residuals in the above equations can be assumed to be independent, normally distributed with zero mean and the same constant variance, Least Squares (LS) estimation should be performed. In this case, the weighting matrices in Equation 14.35 are replaced by the identity matrix I. Maximum likelihood (ML) estimation should be applied when the EoS is capable of calculating the correct phase behavior of the system within the experimental error. Its application requires the knowledge of the measurement... 

It is reasonable to assume that the most probable values of the parameters have normal distributions with means equal to the values that were obtained from well test and core data analyses. These are the prior estimates. Each one of these most probable parameter values (kBj, j=l,...,p) also has a corresponding standard deviation parameter estimate. As already discussed in Chapter 8 (Section 8.5) using maximum likelihood arguments the prior information is introduced by augmenting the LS objective function to include... 

In a well-behaved calibration model, residuals will have a Normal (i.e., Gaussian) distribution. In fact, as we have previously discussed, least-squares regression analysis is also a Maximum Likelihood method, but only when the errors are Normally distributed. If the data does not follow the straight line model, then there will be an excessive number of residuals with too-large values, and the residuals will then not follow the Normal distribution. It follows, then, that a test for Normality of residuals will also detect nonlinearity. 

If it is assumed that the measurement errors are normally distributed, the resolution of problem (5.3) gives maximum likelihood estimates of process variables, so they are minimum variance and unbiased estimators. 

Following Tjoa and Biegler (1991) we have modeled P <5) as a bivariate likelihood distribution, a contaminated normal distribution as shown in Eq. (11.10) with... 

If the errors are normally distributed, the OLS estimates are the maximum likelihood estimates of 9 and the estimates are unbiased and efficient (minimum variance estimates) in the statistical sense. However, if there are outliers in the data, the underlying distribution is not normal and the OLS will be biased. To solve this problem, a more robust estimation methods is needed. 

Mendal et al. (1993) compared eight tests of normality to detect a mixture consisting of two normally distributed components with different means but equal variances. Fisher s skewness statistic was preferable when one component comprised less than 15% of the total distribution. When the two components comprised more nearly equal proportions (35-65%) of the total distribution, the Engelman and Hartigan test (1969) was preferable. For other mixing proportions, the maximum likelihood ratio test was best. Thus, the maximum likelihood ratio test appears to perform very well, with only small loss from optimality, even when it is not the best procedure. 

This criterion, and others, can be derived using maximum likelihood arguments (H8). It has been shown that Eq. (64) is applicable (a) when each of the responses has normally distributed error (b) when the data on each response are equally precise, and (c) when there is no correlation between the measurements of the three responses. These assumptions are rather restrictive. 

Confidence intervals nsing freqnentist and Bayesian approaches have been compared for the normal distribntion with mean p and standard deviation o (Aldenberg and Jaworska 2000). In particnlar, data on species sensitivity to a toxicant was fitted to a normal distribntion to form the species sensitivity distribution (SSD). Fraction affected (FA) and the hazardons concentration (HC), i.e., percentiles and their confidence intervals, were analyzed. Lower and npper confidence limits were developed from t statistics to form 90% 2-sided classical confidence intervals. Bayesian treatment of the uncertainty of p and a of a presupposed normal distribution followed the approach of Box and Tiao (1973, chapter 2, section 2.4). Noninformative prior distributions for the parameters p and o specify the initial state of knowledge. These were constant c and l/o, respectively. Bayes theorem transforms the prior into the posterior distribution by the multiplication of the classic likelihood fnnction of the data and the joint prior distribution of the parameters, in this case p and o (Fignre 5.4). 

Since the final form of a maximum likelihood estimator depends on the assumed error distribution, we partially answered the question why there are different criteria in use, but we have to go further. Maximum likelihood estimates are only guaranteed to have their expected properties if the error distribution behind the sample is the one assumed in the derivation of the method, but in many cases are relatively insensitive to deviations. Since the error distribution is known only in rare circumstances, this property of robustness is very desirable. The least squares method is relatively robust, and hence its use is not restricted to normally distributed errors. Thus, we can drop condition (vi) when talking about the least squares method, though then it is no more associated with the maximum likelihood principle. There exist, however, more robust criteria that are superior for errors with distributions significantly deviating from the normal one, as we will discuss... 

Derive the log-likelihood function for the model in (13-18) assuming that, it and u, are normally distributed. [Hints Write the log-likelihood function as InL = Z "=1 InLj where InT, is the log-likelihood function for the T observations in group i. These T observations are joint nonnally distributed with covariance matrix given in (14-20).] The log-likelihood is the sum of the logs of the joint normal densities of the n sets of T observations,... 

Show that the likelihood inequality in Theorem 17.3 holds for the normal distribution. 

For random sampling from a normal distribution with nonzero mean u and standard deviation ct, find the asymptotic joint distribution of the maximum likelihood estimators of ct/ ii and u2/ct2. 

With the assumption that y is normally distributed, the likelihood (L) that Hx is... 

A comparison of the various fitting techniques is given in Table 5. Most of these techniques depend either explicitly or implicitly on a least-squares minimization. This is appropriate, provided the noise present is normally distributed. In this case, least-squares estimation is equivalent to maximum-likelihood estimation.147 If the noise is not normally distributed, a least-squares estimation is inappropriate. Table 5 includes an indication of how each technique scales with N, the number of data points, for the case in which N is large. A detailed discussion on how different techniques scale with N and also with the number of parameters, is given in the PhD thesis of Vanhamme.148... 

Since this monograph is devoted only to the conception of mathematical models, the inverse problem of estimation is not fully detailed. Nevertheless, estimating parameters of the models is crucial for verification and applications. Any parameter in a deterministic model can be sensibly estimated from time-series data only by embedding the model in a statistical framework. It is usually performed by assuming that instead of exact measurements on concentration, we have these values blurred by observation errors that are independent and normally distributed. The parameters in the deterministic formulation are estimated by nonlinear least-squares or maximum likelihood methods. 

Actually, least squares is often applied in cases where it is not known with any certainty that measurements of jy conform to a normal distribution or even when it is in fact known that they do not conform to a normal distribution. Does this destroy the applicability of the maximum-likelihood criterion The answer is, not necessarily. The central-limit theorem is discussed briefly in Chapter 11. Simply stated, it says that the sum (or average) of a large number of measurements conforms very nearly to a normal distribution, regardless of the distributions of the individual measurements, provided that no one measurement contributes more than a small fraction to the sum (or average) and that the variations in the widths of the individual distributions are within reasonable bounds. (As we shall see, the average of a group of numbers is a special case of a least-squares determination.)... 

When experimental data is to be fit with a mathematical model, it is necessary to allow for the fact that the data has errors. The engineer is interested in finding the parameters in the model as well as the uncertainty in their determination. In the simplest case, the model is a linear equation with only two parameters, and they are found by a least-squares minimization of the errors in fitting the data. Multiple regression is just linear least squares applied with more terms. Nonlinear regression allows the parameters of the model to enter in a nonlinear fashion. The following description of maximum likelihood applies to both linear and nonlinear least squares (Ref. 231). If each measurement point has a measurement error Ayi that is independently random and distributed with a normal distribution about the true model y x) with standard deviation then the probability of a data set is... 

The maximum likelihood method (MLM) is used effectively to identify the unknown parameters of mathematical models when the parameters are distributed. If we consider Fig. 3.1, the actions of the normal distributed perturbations on the process carmot be neglected. Indeed, all process exits will be distributed with individual parameters that depend on the distribution functions associated to the perturbations. 

The observations are to be modeled by Eq. (5.1-8), with normally distributed random errors having expectation zero and variance = 40. Hence, the likelihood function after n observations will be given by Eq. (5.1-15). Normalizing that equation to unit total area, as in Problem 5.B. we obtain... 

The standard deviation cr of a normally distributed variable y with known mean y has the likelihood function (see Eq. (5.1-11))... 

---