Gradient descent technique

We mentioned above that a typical problem for a Boltzman Machine is to obtain a set of weights such that the states of the visible neurons take on some desired probability distribution. For example, the task may he to teach the net to learn that the first component of an Ai-component input vector has value +1 40% of the time. To accompli.sh this, a Boltzman Machine uses the familiar gradient-descent technique, but not on the energy of the net instead, it maximizes the relative entropy of the system. 

A con jugate gradicri I method differs from the steepest descent technique by using both the current gradient and the previous search direction to drive the rn in im i/ation. , A conjugate gradient method is a first order in in im i/er. 

The advantage of a conjugate gradient minimizer is that it uses the minimization history to calculate the search direction, and converges faster than the steepest descent technique. It also contains a scaling factor, b, for determining step size. This makes the step sizes optimal when compared to the steepest descent technique. 

The Back-Propagation Algorithm (BPA) is a supervised learning method for training ANNs, and is one of the most common forms of training techniques. It uses a gradient-descent optimization method, also referred to as the delta rule when applied to feedforward networks. A feedforward network that has employed the delta rule for training, is called a Multi-Layer Perceptron (MLP). 

Bayesian regularization (BR) This technique searches for the simplest network which adjusts itself to the function to be approximated, but which also is able to predict most efficiently the points that did not participate in the training [63], In contrast to gradient-descent, in this case not only the global error of the ANN is taken... 

To minimize the total energy, successive adjustements 5 C of C are made using either the steepest-descent technique or the conjugate gradient technique 5C is written in the form... 

All the techniques are iterative and, except for the simplest chemical systems, require a computer. The methods include optimization by steepest descent (White et al., 1958 Boynton, 1960) and gradient,descent (White, 1967), tback substitution (Kharaka and Barnes, 1973 Truesdell and Jones, 1974), and progressive narrowing of the range of the values allowed for each variable. (.the monotone sequence method Wolery and Walters, 1975). 

Simulated annealing " is an optimization technique particularly well suited to the multiple-minima characteristic of macromolecular structure refinement. Unlike gradient-descent methods, simulated annealing can overcome barriers between... 

Vector quantities, such as a magnetic field or the gradient of electron density, can be plotted as a series of arrows. Another technique is to create an animation showing how the path is followed by a hypothetical test particle. A third technique is to show flow lines, which are the path of steepest descent starting from one point. The flow lines from the bond critical points are used to partition regions of the molecule in the AIM population analysis scheme. 

Techniques used to find global and local energy minima include sequential simplex, steepest descents, conjugate gradient and variants (BFGS), and the Newton and modified Newton methods (Newton-Raphson). 

The steepest descent method is quite old and utilizes the intuitive concept of moving in the direction where the objective function changes the most. However, it is clearly not as efficient as the other three. Conjugate gradient utilizes only first-derivative information, as does steepest descent, but generates improved search directions. Newton s method requires second derivative information but is veiy efficient, while quasi-Newton retains most of the benefits of Newton s method but utilizes only first derivative information. All of these techniques are also used with constrained optimization. 

---