Nongradient methods of optimization

In practice, we often meet situations of search for optimum when there is no mathematical model of the research subject, when calculation of response or optimization criterion is complicated, and when big errors are present in response measure- 

The method of simplex design of experiments in mathematical theory of experiments belongs to a group of nongradient optimization techniques in multidimensional factor space. As a difference to gradient methods, this method does not require a mathematical model of researched phenomenon or does not require derivative of a response. 

Movement to optimum by a simplex method is done step by step and by comparing obtained response values, whereby a single response value is determined in each step. These properties of simplex design are considered its important advantage  

Simplex designs of experiments were first published in 1962 [53], and since then application of this methodology has been constantly growing [31, 54, 10]. Under simplex designing, we understand finding of the optimum of a response function by moving simplex figure on the response surface. Simplex movement is done step by step, whereby in each new step-trial the simplex vertex with the most inconvenient response value is rejected. 

Movement to optimum is realized step by step in such a way that the vertex with the most inconvenient response value is successively rejected and a new point or vertex, which is the physical mirror image of the rejected vertex, is constructed. In the next step, the experiment is done in the new vertex, and then again the vertex with the most inconvenient response value is rejected and the procedure repeated. Movement to optimum is here realized after each step and not after a series of trials as in the method of steepest ascent. Simplex movement to optimum is geometrically in a zig-zag line, while the center of those simplexes moves along a line close to the gradient. The geometrical interpretation is given in Fig. 2.49. 

---

Steepest descent can terminate at any type of stationary point, that is, at any point where the elements of the gradient of /(x) are zero. Thus you must ascertain if the presumed minimum is indeed a local minimum (i.e., a solution) or a saddle point. If it is a saddle point, it is necessary to employ a nongradient method to move away from the point, after which the minimization may continue as before. The stationary point may be tested by examining the Hessian matrix of the objective function as described in Chapter 4. If the Hessian matrix is not positive-definite, the stationary point is a saddle point. Perturbation from the stationary point followed by optimization should lead to a local minimum x. ... 

---