Nonlinear quasi-Newton methods

Gabay, D. Reduced quasi-Newton methods with feasibility improvement for nonlinearly constrained optimization, Math. Prog. Study 16 18 (1982). 

The quasi-Newton methods estimate the matrix = H-1 by updating a previous guess of C in each iteration using only the gradient vector. These methods are very close to the quasi-Newton methods of solving a system of nonlinear equations. The order of convergence is between 1 and 2, and the minimum of a positive definite quadratic function is found in a finite number of steps. 

Schubert, L. K., "Modification of a Quasi-Newton Method for Nonlinear Equations with Sparse Jacobian", Math. Comp. (1970) 2 27-30. 

Quasi-Newton methods form an interesting class of algorithms that are theoretically closely related to nonlinear CG methods.6 95 96 They are found to perform very well in practice.6 100-102 109 110 QN research has been developing... 

Quasi-Newton methods can be viewed as extensions of nonlinear CG methods, in which additional curvature information is used to accelerate convergence. Thus, the required analytic Hessian information, memory, and computational requirements are kept as low as possible, and the main strength of Newton methods—employing curvature information to detect and move away from saddle points efficiently—is retained. 

A method for solving individual models such as Newton or quasi-Newton methods combined with sparse matrix methods to convert the nonlinear alge-... 

In the previous subsection, the successive substitution and Wegstein methods were introduced as the two methods most commonly implemented in recycle convergence units. Other methods, such as the Newton-Raphson method, Broyden s quasi-Newton method, and the dominant-eigenvalue method, are candidates as well, especially when the equations being solved are highly nonlinear and interdependent. In this subsection, the principal features of all five methods are compared. 

The BzzNonLinearSystem class is designed to solve nonlinear systems using a quasi-Newton method as the main algorithm and by availing of all the devices... 

Schubert, L.K. (1979) Modification of a quasi-Newton method for nonlinear equations with a sparse Jacobian. Mathematical and Computer, 24, 27. 

In general, the error e tic-q-i+j, 0) is a non-linear function of the parameter vector 0. Therefore, the above problem is a well-known nonlinear least squares problem (NLSP) that may be solved by various optimisation algorithms such as the Levenberg-Marquardt algorithm [2], the quasi-Newton method or the Gauss-Newton (GN) algorithm [3]. 

As previously commented, the standard method for solving equations is Newton s method. But this requires the calculation of a Jacobian matrix at each iteration. Even assuming that accurate derivatives can be calculated, this is frequently the most time-consuming activity for some problems, especially if nested nonlinear procedures are used. On the other hand, we can also consider the class of quasi-Newton methods where the Jacobian is approximated based on differences in x and/(x), obtained from previous iterations. Here, the motivation is to avoid evaluation of the Jacobian matrix. 

---