Perceptron learning

To summarize, the simple perceptron learning algorithm consists of the following four steps ... 

Pseudo-Code Implementation of Perceptron Learning Algorithm ... 

There are modifications to the perceptron learning rule to help effect faster convergence. The Widrow-Hoff delta rule (Widrow Hoff, 1960) multiplies the delta term by a number less than 1, called the learning rate, tv This effectively causes smaller changes to be made at each step. There are heuristic rules to decrease T] as training time increases the idea is that big changes may be taken at first and as the final solution is approached, smaller changes may be desired. 

The correction realized at Step 2(b) is determined from the error minimization condition, and is called the perceptron learning rule ... 

The quantity y in Theorem 2.1 determines how well the two classes can be separated and consequently how fast the perceptron learning algorithm converges. We call the quantity a margin and give a more formal definition as follows. 

The Perceptron learning algorithm possesses a rate monitored, which means that it requires that the results are evaluated and, if necessary, make the corresponding modifications of the system. 

Artificial Neural Networks (ANNs) attempt to emulate their biological counterparts. McCulloch and Pitts (1943) proposed a simple model of a neuron, and Hebb (1949) described a technique which became known as Hebbian learning. Rosenblatt (1961), devised a single layer of neurons, called a Perceptron, that was used for optical pattern recognition. 

The Back-Propagation Algorithm (BPA) is a supervised learning method for training ANNs, and is one of the most common forms of training techniques. It uses a gradient-descent optimization method, also referred to as the delta rule when applied to feedforward networks. A feedforward network that has employed the delta rule for training, is called a Multi-Layer Perceptron (MLP). 

To say that Minsky and Papert s stinging, but not wholly undeserved, criticism of the capabilities of simple perceptrons was taken hard by perceptron researchers, would be an understatement. They were completely correct in their assessment of the limited abilities of simple perceptrons and they were correct in pointing out that XOR-like problems require perceptrons with more than one decision layer. Where Minsky and Papert erred - and erred strongly - was in their conclusion that since no learning rule for multi-layered nets was then known and will never be found, perceptrons represent a dead end field of research. ... 

Being able to construct an e xplicit solution to a nonlinearly separable problem such as the XOR-problem by using a multi-layer variant of the simple perceptron does not, of course, guarantee that a multi-layer perceptron can by itself learn the XOR function. We need to find a learning rule that works not just for information that only propagates from an input layer to an output layer, but one that works for information that propagates through an arbitrary number of hidden layers as well. 

We are now ready to introduce the backpropagation learning rule (also called the generalized delta rule) for multidayercd perceptrons, credited to Rumelhart and McClelland [rumel86a]. Figure 10.12 shows a schematic of the multi-layered per-ceptron s structure. Notice that the design shown, and the only kind we will consider in this chapter, is strictly feed-forward. That is to say, information always flows from the input layer to each hidden layer, in turn, and out into the output layer. There are no feedback loops anywhere in the system. 

Just as was the case with simple perceptrons, the multi-layer perceptron s fundamental problem is to learn to associate given inputs with desired outputs. The input layer consists of as many neurons as are necessary to set up some natural... 

In our discussion of Hopfield nets in section 10.6, we found that the maximal number of patterns that can be stored before their stability is impaired is some linear function of the size of the net n, ax aN, where 0 < a < 1 and N is the number of neurons in the net (see sections 10.6.6 and 10.7). A similar question can of course be asked of perceptroiis How many input-output fact pairs can a perceptron of given size learn ... 

Neural networks have been introduced in QSAR for non-linear Hansch analyses. The Perceptron, which is generally considered as a forerunner of neural networks has been developed by the Russian school of Rastrigin and coworkers [62] within the context of QSAR. The learning machine is another prototype of neural network which has been introduced in QSAR by Jurs et al. [63] for the discrimination between different types of compounds on the basis of their properties. 

The linear learning machine and the perceptron network 44.4.1 Principle... 

The perceptron-like linear networks were the first networks that were developed [3,4], They are described in an intuitive way in Chapter 33. In this section we explain their working principle as an introduction to that of the more advanced MLF networks. We explain the principle of these early networks by means of the Linear Learning Machine (LLM) since it is the best known example in chemistry. 

It is easy to construct a network of perceptrons by bolting them together so that the outputs of some of them form the inputs of others, but in truth it is hardly worth the effort. The perceptron is not just simple, it is too simple. A network of perceptrons constructed manually can perform a few useful tasks, but it cannot learn anything worthwhile, and since learning is the key to a successful neural network, some modification is needed. 

The problem with the behavior of the perceptron lies in the transfer function if a neuron is to be part of a network capable of genuine learning, the step function used in the perceptron must be replaced by an alternative function that is slightly more sophisticated. The most widely used transfer function is sigmoidal in shape (Figure 8, Eq. [2]), although a linear relationship between input and output signals is used occasionally. 

---