Gradient algorithm

Nonlinear least squares requires an iterative solution starting from set of initial value progressing to a different set of parameters which (hopefully) minimizes some objective function. As the name implies, gradient algorithms use the information contained in the gradient 

All the algorithms that will be presented assume that the objective function, its gradient, and Hessian exist and are continuous functions. They also all use a similar iteration function, namely 

---

McIntosh, A. Fitting Linear Models An Application of Conjugate Gradient Algorithms, Springer-Verlag, New York (1982). 

Although the overall cost of the conjugate gradient algorithm may be higher than that of some of the iterative algorithms described in Section III.B, the algorithm allows us easily to restrict the spectral and temporal stmcture of optimal pulses and enables us to incorporate the exact form of the laser-molecule interactions. 

EXAMPLE 6.2 APPLICATION OF THE FLETCHER-REEVES CONJUGATE GRADIENT ALGORITHM... 

X. Gonze, "First-principle responses of solids to atomic displacements and homogeneous electric fields a conjugate-gradient algorithm, used with plane waves and pseudopotentials," Phys. Rev. B 55 (1997), 10337-10354. 

A. Cohen, SIAM J. Numer. Anal., 9, 248 (1972). Rate of Convergence of Several Conjugate Gradient Algorithms. 

Mailer, M. (1993). A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks 6,525-33. 

M. E. Davis and J. A. McCammon, J. Comput. Chem., 10, 386 (1989). Solving the Finite Difference Linearized Poisson-Boltzmann Equation A Comparison of Relaxation and Conjugate Gradient Algorithms. 

---