The Perceptron

As an approximation to the Bayes rule, the linear discriminant function provides the basis for the most common of the statistical classification schemes. 

The perceptron is a learning algorithm and can be considered as a simple model of a biological neuron. It is worth examining here not only as a classifier in its own right, but also as providing the basic features of modem artificial neural networks. 

The training of the perceptron as a linear classifier then follows the following 

This process is repeated until all objects are correctly classified. 

For our first object, Al, the product of x and Nt is positive and the output is 1, which is a correct result. 

As an approximation to the Bayes rule, the linear discriminant function provides the basis for the most common of the statistical classification schemes, but there has been much work devoted to the development of simpler linear classification rules. One such method, which has featured extensively in spectroscopic pattern recognition studies, is the perceptron algorithm. 

---

Other rules are of course possible. One popular choice called the Delta Rule was introduced in 1960 by Widrow and Hoff ([widrowGO], [widrow62]). Their idea was to adjust the weights in proportion to the error (= A) between the desired output (= D(t)) and the actual output (= y(t)) of the perceptron ... 

As we mentioned above, however, linearly inseparable problems such as the XOR-problem can be solved by adding one or more hidden layers to the perceptron. Figure 10.9, for example, shows a solution to the XOR-problem using a perceptron that has one hidden layer added to it. The numbers appearing by the links are the values of the synaptic weights. The numbers inside the circles (which represent the hidden and output neurons) are the required thresholds r. Notice that the hidden neuron takes no direct input but acts as just another input to the output neuron. Notice also that since the hidden neuron s threshold is set at r = 1.5, it does not fire unless both inputs are equal to 1. Table 10.3 summarizes the perceptron s output. 

Neural networks have been introduced in QSAR for non-linear Hansch analyses. The Perceptron, which is generally considered as a forerunner of neural networks has been developed by the Russian school of Rastrigin and coworkers [62] within the context of QSAR. The learning machine is another prototype of neural network which has been introduced in QSAR by Jurs et al. [63] for the discrimination between different types of compounds on the basis of their properties. 

S.A. Hiller, V.E. Golender, A.B. Rosenblit, LA. Rastrigin and A.B. Glaz, Cybernetic methods of drug design. 1. Statement of the problem—The Perceptron approach. Comp. Biomed. Res., 6(1972)411-421. 

The linear learning machine and the perceptron network 44.4.1 Principle... 

The perceptron-like linear networks were the first networks that were developed [3,4], They are described in an intuitive way in Chapter 33. In this section we explain their working principle as an introduction to that of the more advanced MLF networks. We explain the principle of these early networks by means of the Linear Learning Machine (LLM) since it is the best known example in chemistry. 

Just as in the perceptron-Iike networks, an additional column of ones is added to the X matrix to accommodate for the offset or bias. This is sometimes explicitly depicted in the structure (see Fig. 44.9b). Notice that an offset term is also provided between the hidden layer and the output layer. 

The signal propagation in the MLF networks is similar to that of the perceptron-like networks, described in Section 44.4.1. For each object, each unit in the input layer is fed with one variable of the X matrix and each unit in the output layer is intended to provide one variable of the Y table. The values of the input units are passed unchanged to each unit of the hidden layer. The propagation of the signal from there on can be summarized in three steps. 

F. Rosenblatt, The perceptron a probabilistic model for information storage and organization in the brain. Psycholog. Rev., 65 (1958) 386-408. 

The relationship between the summed inputs to a neuron and its output is an important characteristic of the network, and it is determined by a transfer function (or squashing function or activation function). The simplest of neurons, the perceptron, uses a step function for this purpose, generating an output of zero unless the summed input reaches a critical threshold (Figure 7) for a total input above this level, the neuron fires and gives an output of one. 

It is easy to construct a network of perceptrons by bolting them together so that the outputs of some of them form the inputs of others, but in truth it is hardly worth the effort. The perceptron is not just simple, it is too simple. A network of perceptrons constructed manually can perform a few useful tasks, but it cannot learn anything worthwhile, and since learning is the key to a successful neural network, some modification is needed. 

The problem with the behavior of the perceptron lies in the transfer function if a neuron is to be part of a network capable of genuine learning, the step function used in the perceptron must be replaced by an alternative function that is slightly more sophisticated. The most widely used transfer function is sigmoidal in shape (Figure 8, Eq. [2]), although a linear relationship between input and output signals is used occasionally. 

The failure of the perceptron to handle real-world scientific problems first suggested a wrong model for the brain, and the use of the perceptron was abandoned. Years later, someone realized that the brain can process several independent streams of information simultaneously—it is referred to as a parallel device. Therefore, more than one perceptron may be used in order to accomplish a similar effect. This can be done in two different ways first giving the perceptrons neighbours to form a layer of units which share inputs from the environment and secondly by introducing further layers, each taking as their input, the output from the previous layer. 

Three commonly used ANN methods for classification are the perceptron network, the probabilistic neural network, and the learning vector quantization (LVQ) networks. Details on these methods can be found in several references.57,58 Only an overview of them will be presented here. In all cases, one can use all available X-variables, a selected subset of X-variables, or a set of compressed variables (e.g. PCs from PCA) as inputs to the network. Like quantitative neural networks, the network parameters are estimated by applying a learning rule to a series of samples of known class, the details of which will not be discussed here. 

The perceptron network is the simplest of these three methods, in that its execution typically involves the simple multiplication of class-specific weight vectors to the analytical profile, followed by a hard limit function that assigns either 1 or 0 to the output (to indicate membership, or no membership, to a specific class). Such networks are best suited for applications where the classes are linearly separable in the classification space. 

The functional unit of ANNs is the perceptron. This is a basic unit able to generate a response as a funtion of a number of inputs received from others perceptrons. For example, the response value can be obtained as follows ... 

The perceptron network for this problem is shown in Figure 5.1 below. Input from xo, the bias unit, is always -1. The length of the input vector is 2, and there are 18 input vectors in the training set. 

There are modifications to the perceptron learning rule to help effect faster convergence. The Widrow-Hoff delta rule (Widrow Hoff, 1960) multiplies the delta term by a number less than 1, called the learning rate, tv This effectively causes smaller changes to be made at each step. There are heuristic rules to decrease T] as training time increases the idea is that big changes may be taken at first and as the final solution is approached, smaller changes may be desired. 

The perceptron delta rule can be seen as one technique to achieve the function minimization of equation 5.1. There are more effective methods that will be discussed in later sections. 

---