The Levenberg - Marquardt method

The Levenberg-Marquardt method is used to model data with non-linear dependencies. Here, we may have m functions /i, /2, fz-.-fm that depend on n parameters pi,P2---pn written in vector form as 

Since we are dealing with non-linear equations, this is an iterative procedure that begins by making an initial guess for p, such as for example (lll.l)T. In each iteration step, the values of p are replaced by a new value of p + q. To determine, p we approximate the vector f(p + q) by 

Here too, we will minimize X2 to find the best fit parameters. We repeat the above procedure until X2 stops decreasing. 

Details of this methodology can be found in the literature (see for instance [19]). The Levenberg-Marquardt technique, as many other numerical schemes, can be found as freeware on the world wide web [31], 

Fitting a Bird-Carreau model to viscosity data. Table 7.5 shows the measurements of Ballenger et. al [5] of the viscosity as a function of shear rate for polystyrene at 453 K. 

---

The Levenberg-Marquardt method is used when the parameters of the model appear nonlinearly (Ref. 231). We stiU define... 

The second term in the second derivative is dropped because it is usually small [remember that will be close to y xi, a)]. The Levenberg-Marquardt method then iterates as follows... 

A number of modifications to eliminate some less favorable aspects of the Levenberg-Marquardt method were considered by Fletcher. For instance, the arbitrary initial choice of the adjustable parameter A, if poor, can cause an excessive number of evaluations of squared error, before a realistic value is obtained. This is especially noticeable if v, i.e., J R x), is chosen to be small, i.e., v = 2. Another disadvantage of the method is that the reduction of A to v at the start of each iteration may also cause excessive evaluations, especially when V is chosen to be large, i.e., = 10. The... 

We perform nonlinear fitting using the Levenberg-Marquardt method implemented in the MRQMIN routine [75], From the experimental end, eight families of data are involved, namely, x (T) and x jT) at four frequencies, taken from Ref. 64. From the theory end, we employ formulas (4.121)-(4.124) with the numerical dispersion factors. The results of fitting are presented in Figures 4.6 and 4.7 and Table I. 

The procedure was tested on simulated time domain MRS data where the model data consisted of metabolite peaks at 3.2, 3.0 and 2.0 ppm representing choline, creatine and IV-acetylaspartate (NAA) respectively, with corresponding values of Ak of 1.0, 1.0 and 3.0 units.89 White noise of specified standard deviation, crt, was then added. The Levenberg-Marquardt method requires suitable initial values for each of the nine parameters being fitted. The initial values of the three frequencies were taken as their known values. An exponentially decaying curve with a constant offset parameter was fitted, using a nonlinear least-squares fit, to the envelope of the free induction decay, Mv(t), in order to obtain an initial value for T and for the amplitudes, each of which was taken to be one-third of the amplitude of the envelope. The constant offset was added to account for the presence of the noise. 

The Levenberg-Marquardt Method described in this section represents a compromise between the Gauss-Newton Method described in Section 19.4.1 and the Method of Steepest Descent described in Section 19.4.2. The Method of Steepest Descent is used far from the converged value, moving smoothly to the Gauss-Newton Method as the solution is approached. 

The critical concepts encompassed by the Levenberg-Marquardt Method are the selection of the scaling factor for the Method of Steepest Descent and an approach for making a smooth transition from one method to the other. The curvature matrix a is replaced by cc such that... 

In the case of complex stoichiometries, and when several complexes can coexist in solution, data must be processed using several wavelengths simultaneously. This requires specific software. For instance, the commercially available SPEC FIT Global Analysis System (V3.0 for 32-bit Window Systems) deserves attention. This software uses singular value decomposition and nonlinear regression modeling by the Levenberg-Marquardt method [8]. 

Figure 3.8 Example of parameter redundancy in nonlinear models. Symbols were generated using the model given by Eq. (3.97). Solid line is the predicted fit using Eq. (3.98). The biexponential model predicted values cannot be distinguished from data generated using a triexponential equation. Starting values were 10,1,10, and 0.25. Model was fit using the Levenberg— Marquardt method within the NLIN procedure in SAS.

The Levenberg-Marquardt method is often recommended for least-squares problems since the special structure of these problems allows for a simple approximation to the Hessian matrix using only first derivatives (Fletcher (1987), pp 110-112). The method is available as one of the minimization options in MathCad and often found in other packages. 

The Local Least Squares algorithm performs an independent fit for each individual peak. The calculation is thereby restricted to the range around the band maximum. This drastically reduces the amount of data required for the calculation, enhancing the speed compared to the Levenberg-Marquardt method. Some loss of precision versus the Levenberg-Marquardt method occurs. The Local Least Squares algorithm has some conditions ... 

The Levenberg-Marquardt method is able to move between Newton s method and the gradient method. This feature will be discussed later conversely, it is now important to consider this method for removing the problem of a nonpositive definite Hessian matrix. 

The Levenberg-Marquardt method, already analyzed for functions with nonpositive definite Hessian, use the system... 

Therefore, the Levenberg-Marquardt method solves the problem of the constrained minimum for the specific value of d, for which the relation (3.112), in which the solution d of the system (3.122) is placed, is verified. This means that d is function of d obtained for a specific y. By varying y, a new constrained minimization problem is solved with a different d. 

The Levenberg-Marquardt method may be considered obsolete for several different reasons, the most significant being that it requires the iterative solution of the linear system (3.122) to find a satisfactory set of values of d . 

The continuous curve obtained using the Levenberg-Marquardt method by varying y is approximated by a piecewise joining the Newton prediction, XNavtm, and another point, Xg, placed on the gradient direction. 

It is also possible to look at the Levenberg-Marquardt method in a completely different light. 

As per the Levenberg-Marquardt method, even the dogleg method can be considered from other points of view ... 

Besides being an artifice to reduce Jacobian illLevenberg-Marquardt method is an algorithm that couples Newton s method with the gradient one. 

The Levenberg-Marquardt method may be considered from three distinct perspectives ... 

One valid alternative to the Levenberg-Marquardt method is the dogleg method, also known as Powell s hybrid method (Rabinowitz, 1970). Once again, this couples the Newton and gradient methods. The original version of Powell s method was close to the tmst region concept. Powell proposed a strategy for the modification of parameter d subject to both the successes and failures of the procedure. 

The data from spin-lattiee relaxation time measurements were fitted using the Levenberg-Marquardt method according to the monoexponential function presented in Equation (1). 

---