Optimization techniques steepest descent method

The steepest descent method is quite old and utilizes the intuitive concept of moving in the direction where the objective function changes the most. However, it is clearly not as efficient as the other three. Conjugate gradient utilizes only first-derivative information, as does steepest descent, but generates improved search directions. Newton s method requires second derivative information but is veiy efficient, while quasi-Newton retains most of the benefits of Newton s method but utilizes only first derivative information. All of these techniques are also used with constrained optimization. 

The basic difficulty with the steepest descent method is that it is too sensitive to the scaling of/(x), so that convergence is very slow and what amounts to oscillation in the x space can easily occur. For these reasons steepest descent or ascent is not a very effective optimization technique. Fortunately, conjugate gradient methods are much faster and more accurate. 

All the techniques are iterative and, except for the simplest chemical systems, require a computer. The methods include optimization by steepest descent (White et al., 1958 Boynton, 1960) and gradient,descent (White, 1967), tback substitution (Kharaka and Barnes, 1973 Truesdell and Jones, 1974), and progressive narrowing of the range of the values allowed for each variable. (.the monotone sequence method Wolery and Walters, 1975). 

An extension of the linearization technique discussed above may be used as a basis for design optimization. Such an application to natural gas pipeline systems was reported by Flanigan (F4) using the so-called constrained derivatives (W4) and the method of steepest descent. We offer a more concise derivation of this method following a development by Bryson and Ho (B14). 

Direct minimization techniques. The variational principle indicates that we want to minimize the energy as a function of the MO coefficients or the corresponding density matrix elements, as given by eq. (3.54). In this formulation, the problem is no different from other types of non-linear optimizations, and the same types of technique, such as steepest descent, conjugated gradient or Newton-Raphson methods can be used (see Chapter 12 for details). 

---