Limited-memory quasi-Newton

Two specific classes are emerging as the most powerful techniques for large-scale applications limited-memory quasi-Newton (LMQN) and truncated Newton methods. LMQN methods attempt to combine the modest storage and computational requirements of CG methods with the superlinear convergence properties of standard (i.e., full memory) QN methods. Similarly, TN... 

Thus, TN methods require more care in implementation details and user interface, but their performance is typically at least as good overall as limited-memory quasi-Newton methods. If simplicity is important, the latter is a better choice. If partial second-derivative information is available, the objective function has many quadratic-like regions, and the user is interested in repeated minimization applications, TN algorithms may be worth the effort (see Table 1). In... 

Figure 14 Minimization paths by different methods for the two-dimensional Rosen-brock function f(xx,x2) = (1 — x,)2 + 100(x2 - x,2)2 (a) truncated Newton, (b) conjugate gradient, (c) BFGS quasi-Newton, (d) limited-memory BFGS, (e) steepest descent, first 150 iterations. See program output in Figure 15.

---