Perceptrons

In the previous chapter a simple two-layer artificial neural network was illustrated. Such two-layer, feed-forward networks have an interesting history and are commonly called perceptrons. Similar networks with more than two layers are called multilayer perceptrons, often abbreviated as MLPs. In this chapter the development of perceptrons is sketched with a discussion of particular applications and limitations. Multilayer perceptron concepts are developed applications, limitations and extensions to other kinds of networks are discussed. 

The field of artificial neural networks is a new and rapidly growing field and, as such, is susceptible to problems with naming conventions. In this book, a perceptron is defined as a two-layer network of simple artificial neurons of the type described in Chapter 2. The term perceptron is sometimes used in the literature to refer to the artificial neurons themselves. Perceptrons have been around for decades (McCulloch Pitts, 1943) and were the basis of much theoretical and practical work, especially in the 1960s. Rosenblatt coined the term perceptron (Rosenblatt, 1958). Unfortunately little work was done with perceptrons for quite some time after it was realized that they could be used for only a restricted range of linearly separable problems (Minsky Papert, 1969). 

The simplest perceptron can be used to classify patterns into one of two classes. Training perceptrons and other networks is a numerical, iterative process that will be discussed in Chapter 5. It has been rigorously proven that training perceptrons for classification problems will converge to a solution in a finite number of steps, if a solution exists. 

Since this is greater than 0, the output is 1.0, Le., Fh(0.05) = 1.0 in this case the perception classifies the patient as depressed. The reader is encouraged to try other values and see if they agree with the diagnosis he/she would infer from the graph. 

For an example of a perception used in nucleic acid research, see Stormo et al. (1982). In this study, a perceptron was used to find a weighting function, which distinguishes E. coli 

Coirsider a particular model of a perception called the photoreceptron, used to respond to optical patterns, shown schematically in figure 10.2. 

The photoreceptron consists of three basic parts (1) the retina, which is made 

Gon Neumann himself showed how to use such nets to reliably carry out conventional arithmetic calculations [vonN63]. He was driven by the problem of how to implement the theoretical design in practice, where one demands that a net continues to function correctly even when one or more of its components malfunctions. 

Notice that the only real dynamics going on here takes place between the A-units and R-units. The dynamical flow of information proceeds directly from input to output layers with no hidden units. We will follow the convention of calling perceptron models that have only input and output layers simple perceptrons. As we will shortly see, the absence of a hidden layer dramatically curtails the simple perceptron s problem solving ability. 

---

In the Neural Spectrum Classifier (NSC) a multi-layer perceptron (MLP) has been used for classifying spectra. Although the MLP can perform feature extraction, an optional preprocessor was included for this purpose (see Figure 1). 

Artificial Neural Networks (ANNs) attempt to emulate their biological counterparts. McCulloch and Pitts (1943) proposed a simple model of a neuron, and Hebb (1949) described a technique which became known as Hebbian learning. Rosenblatt (1961), devised a single layer of neurons, called a Perceptron, that was used for optical pattern recognition. 

The Back-Propagation Algorithm (BPA) is a supervised learning method for training ANNs, and is one of the most common forms of training techniques. It uses a gradient-descent optimization method, also referred to as the delta rule when applied to feedforward networks. A feedforward network that has employed the delta rule for training, is called a Multi-Layer Perceptron (MLP). 

Rosenblatt, F. (1961) Principles of Neurodynamics Perceptrons and the Theory of Brain Mechanisms, Spartan Press, Washington, DC. 

Chapter 10 covers another important field with a great overlap with CA neural networks. Beginning with a short historical survey of what is really an independent field, chapter 10 discusses the Hopfield model, stochastic nets, Boltzman machines, and multi-layered perceptrons. 

Other rules are of course possible. One popular choice called the Delta Rule was introduced in 1960 by Widrow and Hoff ([widrowGO], [widrow62]). Their idea was to adjust the weights in proportion to the error (= A) between the desired output (= D(t)) and the actual output (= y(t)) of the perceptron ... 

To summarize, the simple perceptron learning algorithm consists of the following four steps ... 

Pseudo-Code Implementation of Perceptron Learning Algorithm ... 

An appropriate perceptron model for this problem is one that has two input neurons (corresponding to inputs X and X2) and one output neuron, whose value... 

To say that Minsky and Papert s stinging, but not wholly undeserved, criticism of the capabilities of simple perceptrons was taken hard by perceptron researchers, would be an understatement. They were completely correct in their assessment of the limited abilities of simple perceptrons and they were correct in pointing out that XOR-like problems require perceptrons with more than one decision layer. Where Minsky and Papert erred - and erred strongly - was in their conclusion that since no learning rule for multi-layered nets was then known and will never be found, perceptrons represent a dead end field of research. ... 

While, as mentioned at the close of the last section, it took more than 15 years following Minsky and Papert s criticism of simple perceptrons for a bona-fide multilayered variant to finally emerge (see Multi-layeved Perceptrons below), the man most responsible for bringing respectability back to neural net research was the physicist John J, Hopfield, with the publication of his landmark 1982 paper entitled Neural networks and physical systems with emergent collective computational abilities [hopf82]. To set the stage for our discussion of Hopfield nets, we first pause to introduce the notion of associative memory. 

Consider the Boolean exclusive-OR (or XOR) function that we used as an example of a linearly inseparable problem in our discussion of simple perceptrons. In section 10.5.2 we saw that if a perceptron is limited to having only input and output layers (and no hidden layers), and is composed of binary threshold McCulloch-Pitts neurons, the value y of its lone output neuron is given by... 

As we mentioned above, however, linearly inseparable problems such as the XOR-problem can be solved by adding one or more hidden layers to the perceptron. Figure 10.9, for example, shows a solution to the XOR-problem using a perceptron that has one hidden layer added to it. The numbers appearing by the links are the values of the synaptic weights. The numbers inside the circles (which represent the hidden and output neurons) are the required thresholds r. Notice that the hidden neuron takes no direct input but acts as just another input to the output neuron. Notice also that since the hidden neuron s threshold is set at r = 1.5, it does not fire unless both inputs are equal to 1. Table 10.3 summarizes the perceptron s output. 

Table 10.3 Output of the one-hidden-layer perceptron shown in figure 10.9.

Being able to construct an e xplicit solution to a nonlinearly separable problem such as the XOR-problem by using a multi-layer variant of the simple perceptron does not, of course, guarantee that a multi-layer perceptron can by itself learn the XOR function. We need to find a learning rule that works not just for information that only propagates from an input layer to an output layer, but one that works for information that propagates through an arbitrary number of hidden layers as well. 

As we will discuss a bit later on in this section, adding hidden layers but generalizing the McCulloch-Pitts step-function thresholding to a linear function yields a multi-layer perceptron that is fundamentally no more powerful than a simple perceptron that has no hidden layers and uses the McCulloch-Pitts step-function threshold. 

---