Parameter estimation weighted least-squares method

Carroll and Ruppert (1988) and Davidian and Gil-tinan (1995) present comprehensive overviews of parameter estimation in the face of heteroscedasticity. In general, three methods are used to provide precise, unbiased parameter estimates weighted least-squares (WLS), maximum likelihood, and data and/or model transformations. Johnston (1972) has shown that as the departure from constant variance increases, the benefit from using methods that deal with heteroscedasticity increases. The difficulty in using WLS or variations of WLS is that additional burdens on the model are made in that the method makes the additional assumption that the variance of the observations is either known or can be estimated. In WLS, the goal is not to minimize the OLS objective function, i.e., the residual sum of squares,... 

The rate expressions Rj — Rj(T,ck,6m x) typically contain functional dependencies on reaction conditions (temperature, gas-phase and surface concentrations of reactants and products) as well as on adaptive parameters x (i.e., selected pre-exponential factors k0j, activation energies Ej, inhibition constants K, effective storage capacities i//ec and adsorption capacities T03 1 and Q). Such rate parameters are estimated by multiresponse non-linear regression according to the integral method of kinetic analysis based on classical least-squares principles (Froment and Bischoff, 1979). The objective function to be minimized in the weighted least squares method is... 

WEIGHTED LEAST SQUARES PARAMETER ESTIMATION IN MULTIVARIABLE NONLINEAR MODELS 6AUSS - NEWTON - MARQUARDT METHOD... 

The methods of Chapter 6 are not appropriate for multiresponse investigations unless the responses have known relative precisions and independent, unbiased normal distributions of error. These restrictions come from the error model in Eq. (6.1-2). Single-response models were treated under these assumptions by Gauss (1809, 1823) and less completely by Legendre (1805), co-discoverer of the method of least squares. Aitken (1935) generalized weighted least squares to multiple responses with a specified error covariance matrix his method was extended to nonlinear parameter estimation by Bard and Lapidus (1968) and Bard (1974). However, least squares is not suitable for multiresponse problems unless information is given about the error covariance matrix we may consider such applications at another time. 

Unlike linear models where normal equations can be solved explicitly in terms of the model parameters, Eqs. (3.14) and (3.15) are nonlinear in the parameter estimates and must be solved iteratively, usually using the method of nonlinear least squares or some modification thereof. The focus of this chapter will be on nonlinear least squares while the problem of weighted least squares, data transformations, and variance models will be dealt with in another chapter. 

Copolymer-reactivity ratios obtained from the feed and copolymer composition data with linearized equations, as in the Fineman-Ross procedure, do not allow proper weighting of the experimental data, and cannot provide a proper estimate of the precision of the parameters, which, being interdependent, have joint confidence limits. Computer-based methods for determining reactivity ratios have been summarized and non-linear least squares methods described.. Errors in the dependent variables were included by Yamada... 

At the high recycling ratios the loop reactor operates as an ideal stirred-tank reactor. Therefore, the reaction rate can immediately be determined from the difference in concentration between the feed and the outlet, the throughput and the quantity of catalyst.The rate equation, describing the consumption of xylene and the formation of the reaction products, are considered to be pseudo first order. The parameter of the rate equations, which are the frequency factors and the activation energies, are determined by least square methods. In the above function (Fig. 6b) r is the measured rate, r is calculated with estimated parameters, w represent appropriate weight factors and N is the number of measured values. Because the rate equations could be differentiated v/ith respect to the unknown kinetic parameters, the objective function was minimized by a step-wise regression. 

On the other hand, false coverage probabilities do not allow any practical interpretation. In particular, an experimenter would like to know, in practical terms, how well a method performs with respect to others and how its performance is affected by the sample size. Let s take a closer look at the behavior oi Ip. K simulation of Ip values, which will be explained later, yielded the probability density functions for the two methods illustrated in Fig. 1. For this simulation the parameters are set as m = 5, CTo = 100, p = Q.Ql, and the sample size is chosen as = 20. The density function of Ip estimated by the ML method is slightly to the right of the function estimated by the GLR method 1. Therefore, for any a p significant difference from the viewpoint of an experimenter. Further, Method 1 is one of the worst methods in Table 1, so the difference between the ML method and the GLR methods with weighted least squares is expected to be smaller. One practical way for quantifying the difference is to compute the sample size necessary for the ML method to have approximately the same density function as the GLR method 1. 

Due to the nrmlinear dependence of the a priori, unknown residual sequence E on the parameter vector 0, the last equation leads to a nonlinear-weighted least squares problem, which has to be tackled by nrmlinear optimization methods. However, nonUnear least squares techniques are sensitive to the initial parameter values and if no acciuate estimates are available, the nuniniization procedure is very likely to converge to a local minimum. In order to avoid potential inaccurate convergence problems associated with arbitrary initial estimates, initial values for the coefficients of projection may be obtained by identifying conventional ARMA models for each of the K data... 

The fitting procedure In this block the differences between experimental and theoretical data are minimized. A weighted least squares cost function is formulated. The Gauss-Newton and Levenberg-Marquardt method are implemented to minimize this cost fxmction and eventually provide the parameter values which best describe the data. Moreover, the standard deviations of the estimated parameters are also calculated. 

The least squares method was originally proposed by Gauss in 1795 in the context of parameter estimation. The weighting function in the ieast squares method is dR x a) jda which results in ... 

Although satisfactory criteria for deciding whether data are better analyzed by distributions or multiexponential sums have yet to established, several methods for determining distributions have been developed. For pulse fluorometry, James and Ware(n) have introduced an exponential series method. Here, data are first analyzed as a sum of up to four exponential terms with variable lifetimes and preexponential weights. This analysis serves to establish estimates for the range of the preexponential and lifetime parameters used in the next step. Next, a probe function is developed with fixed lifetime values and equal preexponential factors. An iterative Marquardt(18) least-squares analysis is undertaken with the lifetimes remaining fixed and the preexponential constrained to remain positive. When the preexponential... 

Selected entries from Methods in Enzymology [vol, page(s)] Association constant determination, 259, 444-445 buoyant mass determination, 259, 432-433, 438, 441, 443, 444 cell handling, 259, 436-437 centerpiece selection, 259, 433-434, 436 centrifuge operation, 259, 437-438 concentration distribution, 259, 431 equilibration time, estimation, 259, 438-439 molecular weight calculation, 259, 431-432, 444 nonlinear least-squares analysis of primary data, 259, 449-451 oligomerization state of proteins [determination, 259, 439-441, 443 heterogeneous association, 259, 447-448 reversibility of association, 259, 445-447] optical systems, 259, 434-435 protein denaturants, 259, 439-440 retroviral protease, analysis, 241, 123-124 sample preparation, 259, 435-436 second virial coefficient [determination, 259, 443, 448-449 nonideality contribution, 259, 448-449] sensitivity, 259, 427 stoichiometry of reaction, determination, 259, 444-445 terms and symbols, 259, 429-431 thermodynamic parameter determination, 259, 427, 443-444, 449-451. 

It should be stressed that, in any experiment for which there is to be a least-squares refinement of parameters, intended to yield not only the best possible parameter values but also respectable estimates of their uncertainties and a test of the validity of the model, it is vital to take the trouble to analyze the methods and the circumstances of the experiment carefully in order to get the best possible values of the a priori weights. 

Several of the procedures for deriving structural parameters from moments of inertia make use of the method of least squares. Since the relation between moments of inertia and Cartesian coordinates or internal coordinates is nonlinear, an iterative least squares procedure must be used.18 In this procedure an initial estimate of the structural parameters is made and derivatives of the n moments of inertia with respect to each of the k coordinates are calculated based on this estimate. These derivatives make up a matrix D with n rows and k columns. We then define a vector X to be the changes in the k coordinates and a vector B to be the differences between the experimental moments and the calculated moments. We also define a weight matrix W to be the inverse of the ma-... 

A second method would be to fit the force field by least squares to the infrared data and to the r s. This technique has the advantage that the force field will be independent of the microwave data base. Again, it is not obvious that the values obtained for the force constants will be close to the true values, but the force field obtained will come closest to reproducing the observed data. The method would be enhanced if each parameter (centrifugal distortion constant or vibrational fundamental) were to be weighted by the inverse square of its estimated model uncertainty (which will usually be considerably larger than its measurement uncertainty). The force field thus obtained would probably provide a close approximation to the true force field and could be used for intermolecular comparisons, but it would have the drawback that model error information would be lost. 

The particular choice of a residual variance model should be based on the nature of the response function. Sometimes 4> is unknown and must be estimated from the data. Once a structural model and residual variance model is chosen, the choice then becomes how to estimate 0, the structural model parameters, and <, the residual variance model parameters. One commonly advocated method is the method of generalized least-squares (GLS). First it will be assumed that < is known and then that assumption will be relaxed. In the simplest case, assume that 0 is known, in which case the weights are given by... 

---