Rule learning

Other rules are of course possible. One popular choice called the Delta Rule was introduced in 1960 by Widrow and Hoff ([widrowGO], [widrow62]). Their idea was to adjust the weights in proportion to the error (= A) between the desired output (= D(t)) and the actual output (= y(t)) of the perceptron  

To summarize, the simple perceptron learning algorithm consists of the following four steps  

Pseudo-Code Implementation of Perceptron Learning Algorithm  

Step 1 Initialize weights Wi(t = 0) to be small random values and choose the threshold r. 

Step 4 Adjust the synaptic weights according to either of the two learning rules given in equations 10.3 and 10.4. 

Initialize the weights with small random values in a range around 0 (e.g. -0.3 to +0.3). 

For each output unit, j, calculate the output value with the current weight setting and the error, E, based on the difference between this value and the target or desired output value. 

Ej is determined by the weights (through Oj which is a function of NET, see eq. (44.8)). Note that this error is in fact the same as the error term used in a usual least squares procedure. 

Repeat this process for all input patterns. One iteration or epoch is defined as one weight correction for all examples of the training set. 

---

Let us start with a classic example. We had a dataset of 31 steroids. The spatial autocorrelation vector (more about autocorrelation vectors can be found in Chapter 8) stood as the set of molecular descriptors. The task was to model the Corticosteroid Ringing Globulin (CBG) affinity of the steroids. A feed-forward multilayer neural network trained with the back-propagation learning rule was employed as the learning method. The dataset itself was available in electronic form. More details can be found in Ref. [2]. 

Now, one may ask, what if we are going to use Feed-Forward Neural Networks with the Back-Propagation learning rule Then, obviously, SVD can be used as a data transformation technique. PCA and SVD are often used as synonyms. Below we shall use PCA in the classical context and SVD in the case when it is applied to the data matrix before training any neural network, i.e., Kohonen s Self-Organizing Maps, or Counter-Propagation Neural Networks. 

Square nodes in the ANFIS structure denote parameter sets of the membership functions of the TSK fuzzy system. Circular nodes are static/non-modifiable and perform operations such as product or max/min calculations. A hybrid learning rule is used to accelerate parameter adaption. This uses sequential least squares in the forward pass to identify consequent parameters, and back-propagation in the backward pass to establish the premise parameters. 

Several different learning rules have been proposed by various researchers [15,19,201, but the aim of every... 

A natural question to ask is whether the basic model can be modified in some way that would enable it to correctly learn the XOR function or, more generally, any other non-linearly-separable problem. The answer is a qualified yes in principle, all that needs to be done is to add more layers between what we have called the A-units and R-units. Doing so effectively generates more separation lines, which when combined can successfully separate out the desired regions of the plane. However, while Rosenblatt himself considered such variants, at the time of his original analysis (and for quite a few years after that see below) no appropriate learning rule was known. 

To say that Minsky and Papert s stinging, but not wholly undeserved, criticism of the capabilities of simple perceptrons was taken hard by perceptron researchers, would be an understatement. They were completely correct in their assessment of the limited abilities of simple perceptrons and they were correct in pointing out that XOR-like problems require perceptrons with more than one decision layer. Where Minsky and Papert erred - and erred strongly - was in their conclusion that since no learning rule for multi-layered nets was then known and will never be found, perceptrons represent a dead end field of research. ... 

Per.sonnaz, et.ai. [pers86] modified the Hebbiau learning rule so that even non-orthogonal states remain stable, but the modification effectively makes the learning rule a non-local one. 

Being able to construct an e xplicit solution to a nonlinearly separable problem such as the XOR-problem by using a multi-layer variant of the simple perceptron does not, of course, guarantee that a multi-layer perceptron can by itself learn the XOR function. We need to find a learning rule that works not just for information that only propagates from an input layer to an output layer, but one that works for information that propagates through an arbitrary number of hidden layers as well. 

We are now ready to introduce the backpropagation learning rule (also called the generalized delta rule) for multidayercd perceptrons, credited to Rumelhart and McClelland [rumel86a]. Figure 10.12 shows a schematic of the multi-layered per-ceptron s structure. Notice that the design shown, and the only kind we will consider in this chapter, is strictly feed-forward. That is to say, information always flows from the input layer to each hidden layer, in turn, and out into the output layer. There are no feedback loops anywhere in the system. 

Richards, et. al. comment that while the exact relationship between the rule found by their genetic algorithm and the fundamental equations of motion for the solidification remains unknown, it may still be possible to connect certain features of the learned rule to phenomenological models. 

---