Marquardts Modification

In order to improve the convergence characteristics and robustness of the Gauss-Newton method, Levenberg in 1944 and later Marquardt (1963) proposed to modify the normal equations by adding a small positive number, y2, to the diagonal elements of A. Namely, at each iteration the increment in the parameter vector is obtained by solving the following equation 

A more interesting interpretation of Levenberg-Marquardt s modification can be obtained by examining the eigenvalues of the modified matrix (A+y2 ). If we consider the eigenvalue decomposition of A, V1 AV we have, 

---

A second method of improving the Gauss-Newton method is the Marquardt modification. In this case, the equation for P ew is modified by the addition of another term, pi, as in Eq. (21) ... 

The learning rate tj is used to adjust the stabihty and convergence rate, especially when the optimum is very flat. The Levenberg-Marquardt modification of the Gauss-Newton method introduces an additional factor /u >=0, such that ... 

Modifications of Levenberg-Marquardt Method Fletcher s Modification... 

A number of modifications to eliminate some less favorable aspects of the Levenberg-Marquardt method were considered by Fletcher. For instance, the arbitrary initial choice of the adjustable parameter A, if poor, can cause an excessive number of evaluations of squared error, before a realistic value is obtained. This is especially noticeable if v, i.e., J R x), is chosen to be small, i.e., v = 2. Another disadvantage of the method is that the reduction of A to v at the start of each iteration may also cause excessive evaluations, especially when V is chosen to be large, i.e., = 10. The... 

In general there are two remedies to this problem (i) use a pseudoinverse and/or (ii) use Levenberg-Marquardt s modification. 

The remedies to increase the region of convergence include the use of a pseudoinverse or Marquardt s modification that overcome the problem of ill-conditioning of matrix A. However, if the basic sensitivity information is not there, the estimated direction Ak +I) cannot be obtained reliably. 

The minimization of SLG(k,co) can now be accomplished by applying the Gauss-Newton method with Marquardt s modification and a step-size policy as described in earlier chapters. 

As mentioned in Chapter 4, although this is a dynamic experiment where data are collected over time, we consider it as a simple algebraic equation model with two unknown parameters. The data were given for two different conditions (i) with 0.75 g and (ii) with 1.30 g of methanol as solvent. An initial guess of k =1.0 and k2=0.01 was used. The method converged in six and seven iterations respectively without the need for Marquardt s modification. Actually, if Mar-quardt s modification is used, the algorithm slows down somewhat. The estimated parameters are given in Table 16.1 In addition, the model-calculated values are... 

In this work, we first regressed the isothermal data. The estimated parameters from the treatment of the isothermal data are given in Table 16.6. An initial guess of (ki=l.O, k2=1.0, k3=1.0) was used for all isotherms and convergence of the Gauss-Newton method without the need for Marquardt s modification was achieved in 13, 16 and 15 iterations for the data at 375, 400, and 425°C respectively. 

In this example the number of measured variables is less than the number of state variables. Zhu et al. (1997) minimized an unweighted sum of squares of deviations of calculated and experimental concentrations of HPA and PD. They used Marquardt s modification of the Gauss-Newton method and reported the parameter estimates shown in Table 16.24. 

Using an initial guess of kj=350 and k2=l the Gauss-Newton method converged in five iterations without the need for Marquardt s modification. The estimated parameters are k,= 334.27 2.10% and k2=0.38075 5.78%. The model-calculated values are compared with the experimental data in Table 17.1. As seen the agreement is very good in this case. 

Indeed, using the Gauss-Newton method with an initial estimate of k(0)=(450, 7) convergence to the optimum was achieved in three iterations with no need to employ Marquardt s modification. The optimal parameter estimates are k = 420.2 8.68% and k2= 5.705 24.58%. It should be noted however that this type of a model can often lead to ill-conditioned estimation problems if the data have not been collected both at low and high values of the independent variable. The convergence to the optimum is shown in Table 17.5 starting with the initial guess k(0)=(l, 1). 

Equation 17.10 can now be used to obtain the two unknown parameters (kLa and Cq2 ) by fitting the data from the gassing-in period of the experiment. Indeed, using the Gauss-Newton method with an initial guess of (10, 10) convergence is achieved in 7 iterations as shown in Table 17.6. There was no need to employ Marquardt s modification. The FORTRAN program used for the above calculations is also provided in Appendix 2. 

J. Garcia-Pena, S.P. Azen and R.N. Bergman, On a modification of Marquardt s compromise Rationale and applications. Appl. Math. Computing,... 

Operator independence. The GEX method is virtually operator independent. The only inputs required before fitting are parameters concerning the precision of the numerical integration and exit criteria for the Marquardt algorithm. The same set of Inputs was used for all five MWDs. Currently the method also requires upper and lower limits on the GEX parameters but a simple modification of the code can eliminate this need. The CONTIN algorithm has several operator and case dependent parameters that have to be chosen before analysis. However, it is fairly stable with respect to bad choices for some of these inputs. The GEX fit method cannot fit multimodal MWDs without prior knowledge of the number of peaks in the distribution. While CONTIN does not impose this limitation it 1s recommended that the number of peaks be specified before analysis. In our experience with CONTIN if this condition is not met the algorithm tends to compute unimodal solutions for multimodal MWDs and sometimes visa versa. 

The binary interaction parameter, k j, is initially assumed to be zero, and a modification of the Levenberg-Marquardt algorithm (MINPACK) is applied to minimize the sum of the squares given by Equation (1). This calculation was applied to the following systems at the indicated temperatures ... 

---