Gauss-Newton approximation

The terms in the second line of Eq. (7.2-14) vanish if the model is linear in 6 and are unimportant if the data are well fitted. They also are computationally expensive. GREGPLUS omits these terms this parallels the Gauss-Newton approximation in the normal equations of Chapter 6. 

If we can assume that the WSS surface between the initial estimate and the global minimum is convex, a Taylor series expansion leads to the Gauss-Newton approximation for a step closer to the minimum. Thus, the next point on the surface can be calculated as in Eq. (20) ... 

From the sensitivities and model solution, we can then calculate the gradient of the objective function and the Gauss-Newton approximation of the Hessian matrix. Reliable and robust numerical optimization programs are available to find the optimal values of the parameters. These programs are generally more efficient if we provide the gradient in addition to the objective function. The Hessian is normally needed only to calculate the confidence intervals after the optimal parameters are determined. If we define e to be the residual vector... 

Gauss-Newton approximation and sensitivities. In the Gauss-Newton approximation of the Hessian, we neglect the second term in Equation 9.35 to yield,... 

The use of the fiiU Hessian in a Newton iteration requires the computation of (n X m) second order derivatives. Instead, for the Gauss-Newton approximation of the Hessian only n yc m first order derivatives are to be computed in each iteration... 

Foresee, ED. and Hagan, M. (1997) Gauss-Newton approximation to Bayesian learning, Proceedings of International Joint Conference on Neural Networks, 3 1930-5. 

As shown in many comparative studies (see, e.g., refs. 10-12), apart from some special cases (ref. 13) the most efficient algorithms to minimize sum-of-squares objective functions are the various versions of the Gauss-Newton method. The method is based on the local linear approximation... 

The derivatives F r are called the first-order parametric sensitivities of the model. Their direct computation via Newton s method is implemented in Subroutines DDAPLUS (Appendix B) and PDAPLUS. Finite-difference approximations are also provided as options in GREGPLUS to produce the matrix A in either the Gauss-Newton or the full Newton form these approximations are treated in Problems 6.B and 6.C. 

Interval estimates for nonlinear models are usually approximate, since exact calculations are very difficult for more than a few parameters. But, as our colleague George Box once said, One needn t be excessively precise about uncertainty. In this connection. Donaldson and Schnabel (1987) found the Gauss-Newton normal equations to be more reliable than the full Newton equations for computations of confidence regions and intervals. 

A common procedure for solving this overdetermined system is the method of variation of parameters (also referred to in the mathematical literature as Gauss-Newton non-linear least squares algorithm) (Vanicek and Krakiwsky 1982), and this procedure is described in the following. As approximate values of coordinates x° are known a priori, by Taylor s series expansion of the function / about point x°. 

Remark 6.1 IftheJacobian Je(0) has full rank, then the approximation (0)Je(0) of the Hessian H is positive definite and the Gauss-Newton search direction A0 is a downhill direction. 

The term with second order derivatives is ignored by the Gauss-Newton method. Let 0 denote the set of parameters that makes the value of the objective function a minimum. If any rv(0 ) (1 < v < n) is not small then the approximation of the Hessian matrix H (cf. (6.10)) is poor and a line search may be needed for the method to be convergent. 

To find a minimum of the functional Q 9), the Gauss-Newton iterative method or its modifications based on linear approximation of the regression function in the neighborhood of a point 0 are used ... 

Figure 7.1 Four steps of the Gauss-Newton algorithm showing convergence to matched-curvature normal distribution. The left-hand panels are the logarithm of the target and the quadratic that matches the first two derivatives at the value 0 i, and the right-hand panels are the corresponding target and normal approximation.

An early method used for solving these equations is the Gauss-Newton method, for which the Taylor series is used. Other methods include steepest descent and Marquardt s compromise. It is not the scope of this book to go into the details of solving non-linear normal equations because they are often solved computationally by using various software. However, this information is readily available in various textbooks. In this book, we will use linearization to determine approximate confidence levels and correlation matrices. For many applications, linearization is a very important tool that is used as a standard method in research. 

This method, known as the Gauss-Newton method, converts the nonlinear problem into a linear one by approximating the function F by a Taylor series expansion around an estimated value of the parameter vector b ... 

---