Perceptron

The input device of the machine consists of a series of receptors. Each receptor corresponds to one component x/ of a binary encoded patterh x (x has the value 0 or 1). The second layer of the machine consists of a series of association units. Each association unit has several inputs but only one output. The inputs of the association units 

FIGURE 35. Perceptron. R receptors (with binary encoded outputs 

equivalent to a binary encoded pattern vector). A associ-1 d 

A response unit compares the result (scalar product) with a threshold and produces the final classification result, e.g. a 1 for positive scalar products (class 1) and a 0 for negative scalar products (class 2). 

A geometrica I interpretation of the perceptron shows that the pattern space is divided by the association units into several polyhedrons. The partitioning is random because the receptors are randomly connected to the association units. The perceptron coordinates each polyhedron either to class 1 or to class 2. 

The learning problem is to find the weights vector tv e such that the computed output of the unit, R, is as close as possible, if not equal, to the desired output, e R, for all the available input vectors x. The activation 

Other variations of the perceptron use an identical activation function, that is, f x) = X. The mathematical interpretation of the original perceptron is the problem of linear separability of two classes. The original condition of linear separability is to find a weight vector w such that x 0 for all points x in one class, and w x 0 for all points x in the other class. With a suitable linear transformation, the separability condition becomes w x 0 for all the points correctly classified, and otherwise. 

This means that in the case of the Rosenblatt perceptron, the desired output value y is always 1. The training procedure is the following  

Repeat Step 2 for all the test vectors available, until no improvement is done for all the test vectors, or until a certain maximal number of iterations has been reached. 

The correction realized at Step 2(b) is determined from the error minimization condition, and is called the perceptron learning rule  

---

In the Neural Spectrum Classifier (NSC) a multi-layer perceptron (MLP) has been used for classifying spectra. Although the MLP can perform feature extraction, an optional preprocessor was included for this purpose (see Figure 1). 

Artificial Neural Networks (ANNs) attempt to emulate their biological counterparts. McCulloch and Pitts (1943) proposed a simple model of a neuron, and Hebb (1949) described a technique which became known as Hebbian learning. Rosenblatt (1961), devised a single layer of neurons, called a Perceptron, that was used for optical pattern recognition. 

The Back-Propagation Algorithm (BPA) is a supervised learning method for training ANNs, and is one of the most common forms of training techniques. It uses a gradient-descent optimization method, also referred to as the delta rule when applied to feedforward networks. A feedforward network that has employed the delta rule for training, is called a Multi-Layer Perceptron (MLP). 

Rosenblatt, F. (1961) Principles of Neurodynamics Perceptrons and the Theory of Brain Mechanisms, Spartan Press, Washington, DC. 

Chapter 10 covers another important field with a great overlap with CA neural networks. Beginning with a short historical survey of what is really an independent field, chapter 10 discusses the Hopfield model, stochastic nets, Boltzman machines, and multi-layered perceptrons. 

Notice that the only real dynamics going on here takes place between the A-units and R-units. The dynamical flow of information proceeds directly from input to output layers with no hidden units. We will follow the convention of calling perceptron models that have only input and output layers simple perceptrons. As we will shortly see, the absence of a hidden layer dramatically curtails the simple perceptron s problem solving ability. 

Other rules are of course possible. One popular choice called the Delta Rule was introduced in 1960 by Widrow and Hoff ([widrowGO], [widrow62]). Their idea was to adjust the weights in proportion to the error (= A) between the desired output (= D(t)) and the actual output (= y(t)) of the perceptron ... 

To summarize, the simple perceptron learning algorithm consists of the following four steps ... 

Pseudo-Code Implementation of Perceptron Learning Algorithm ... 

An appropriate perceptron model for this problem is one that has two input neurons (corresponding to inputs X and X2) and one output neuron, whose value... 

To say that Minsky and Papert s stinging, but not wholly undeserved, criticism of the capabilities of simple perceptrons was taken hard by perceptron researchers, would be an understatement. They were completely correct in their assessment of the limited abilities of simple perceptrons and they were correct in pointing out that XOR-like problems require perceptrons with more than one decision layer. Where Minsky and Papert erred - and erred strongly - was in their conclusion that since no learning rule for multi-layered nets was then known and will never be found, perceptrons represent a dead end field of research. ... 

While, as mentioned at the close of the last section, it took more than 15 years following Minsky and Papert s criticism of simple perceptrons for a bona-fide multilayered variant to finally emerge (see Multi-layeved Perceptrons below), the man most responsible for bringing respectability back to neural net research was the physicist John J, Hopfield, with the publication of his landmark 1982 paper entitled Neural networks and physical systems with emergent collective computational abilities [hopf82]. To set the stage for our discussion of Hopfield nets, we first pause to introduce the notion of associative memory. 

Consider the Boolean exclusive-OR (or XOR) function that we used as an example of a linearly inseparable problem in our discussion of simple perceptrons. In section 10.5.2 we saw that if a perceptron is limited to having only input and output layers (and no hidden layers), and is composed of binary threshold McCulloch-Pitts neurons, the value y of its lone output neuron is given by... 

As we mentioned above, however, linearly inseparable problems such as the XOR-problem can be solved by adding one or more hidden layers to the perceptron. Figure 10.9, for example, shows a solution to the XOR-problem using a perceptron that has one hidden layer added to it. The numbers appearing by the links are the values of the synaptic weights. The numbers inside the circles (which represent the hidden and output neurons) are the required thresholds r. Notice that the hidden neuron takes no direct input but acts as just another input to the output neuron. Notice also that since the hidden neuron s threshold is set at r = 1.5, it does not fire unless both inputs are equal to 1. Table 10.3 summarizes the perceptron s output. 

Table 10.3 Output of the one-hidden-layer perceptron shown in figure 10.9.

Being able to construct an e xplicit solution to a nonlinearly separable problem such as the XOR-problem by using a multi-layer variant of the simple perceptron does not, of course, guarantee that a multi-layer perceptron can by itself learn the XOR function. We need to find a learning rule that works not just for information that only propagates from an input layer to an output layer, but one that works for information that propagates through an arbitrary number of hidden layers as well. 

As we will discuss a bit later on in this section, adding hidden layers but generalizing the McCulloch-Pitts step-function thresholding to a linear function yields a multi-layer perceptron that is fundamentally no more powerful than a simple perceptron that has no hidden layers and uses the McCulloch-Pitts step-function threshold. 

---