Direct methods conjugate search directions

There are two basic types of unconstrained optimization algorithms (I) those reqmring function derivatives and (2) those that do not. The nonderivative methods are of interest in optimization applications because these methods can be readily adapted to the case in which experiments are carried out directly on the process. In such cases, an ac tual process measurement (such as yield) can be the objec tive function, and no mathematical model for the process is required. Methods that do not reqmre derivatives are called direc t methods and include sequential simplex (Nelder-Meade) and Powell s method. The sequential simplex method is quite satisfac tory for optimization with two or three independent variables, is simple to understand, and is fairly easy to execute. Powell s method is more efficient than the simplex method and is based on the concept of conjugate search directions. 

Random Search / 6.1.2 Grid Search / 6.1.3 Univariate Search / 6.1.4 Simplex Search Method / 6.1.5 Conjugate Search Directions / 6.1.6 Summary... 

The basis used in this method is conjugate search directions and orthogonal residuals which is equivalent to finding a minimum point along the search directions. 

A con jugate gradicri I method differs from the steepest descent technique by using both the current gradient and the previous search direction to drive the rn in im i/ation. , A conjugate gradient method is a first order in in im i/er. 

The steepest descent method is quite old and utilizes the intuitive concept of moving in the direction where the objective function changes the most. However, it is clearly not as efficient as the other three. Conjugate gradient utilizes only first-derivative information, as does steepest descent, but generates improved search directions. Newton s method requires second derivative information but is veiy efficient, while quasi-Newton retains most of the benefits of Newton s method but utilizes only first derivative information. All of these techniques are also used with constrained optimization. 

The conjugate gradient method is one of the oldest in the Retcher-Reeves approach, the search direction is given by... 

In generating a third iterate for the conjugate-gradient method, we now estimate the search direction by VEfe) but insist that the search direction is orthogonal to both do and di. This idea is then repeated for subsequent iterations. 

---