Generalized delta rule

The general principle behind most commonly used back-propagation learning methods is the delta rule, by which an objective function involving squares of the output errors from the network is minimized. The delta rule requires that the sigmoidal function used at each neuron be continuously differentiable. This methods identifies an error associated with each neuron for each iteration involving a cause-effect pattern. Therefore, the error for each neuron in the output layer can be represented as ... 

We are now ready to introduce the backpropagation learning rule (also called the generalized delta rule) for multidayercd perceptrons, credited to Rumelhart and McClelland [rumel86a]. Figure 10.12 shows a schematic of the multi-layered per-ceptron s structure. Notice that the design shown, and the only kind we will consider in this chapter, is strictly feed-forward. That is to say, information always flows from the input layer to each hidden layer, in turn, and out into the output layer. There are no feedback loops anywhere in the system. 

Backpropagation is a generalized version of the delta rule, extended to multiple layers. The central assumption of BP is that when the target output and actual output at a node differ, the responsibility for the error can be divided between ... 

The basic learning mechanism for networks of multilayer neurons is the generalized delta rule, commonly referred to as back propagation. This learning rule is more complex than that employed with the simple perceptron unit because of the greater information content associated with the continuous output variable compared with the binary output of the perceptron. 

The final output from the network for our input pattern is compared with the known, correct result and a measure of the error is computed. In order to reduce this error, the weight vectors between neurons are adjusted by using the generalized delta rule and back-propagating the error from one layer to the previous layer. 

Gaussian distribution, 2 Generalized delta rule, 150 Goodness of fit, 159 Group average clustering, 106... 

I will discuss the generalized delta rule in chapter 14 in the presentation of the learning model. 

The delta learning rule can be generalized for multilayer networks. Using an approach simihar to the delta rule, the gradient of the global error can be computed with respect to each weight in the network. Interestingly,... 

The generalized delta rule for the feth output unit is given by... 

Train each output node using the generalized delta rule and the values from Step 4. 

Wang (1966) has considered the sum kernel (a(x, y) = x y) and the product kernel (a(x, y) = xy) for their self-similar forms and found them to be generalized functions, viz., Dirac delta functions thus ruling out the possibility of observable self-similar behavior. However, this conclusion was clearly in error, as it is now known that both the sum and product kernels have the respective self-similar solutions... 

---