Training Perceptrons

The principle behind training perceptrons is simple try a set of weights. If the output from the network, in response to a given input vector, matches the target value for that 

1) Initialize all weights, including threshold values, to a small random number, say, between 0 and 1  

2) Apply an input vector with components (xi, X2,..Xi) to the network, calculate o, the output of the network for this vector. For notational simplicity, assume that there is just one input vector X with i elements, and its associated target value t  

4) Repeat 2 and 3 for the new weights, until there are no more changes to be made for each input vector. The symbol = means replace the old value with a new value. 

This is called the perception learning rule and it has been proven to converge to a solution, for linearly separable problems, in a finite number of iterations. The weight adjustment rule can be restated as 

---

The simplest perceptron can be used to classify patterns into one of two classes. Training perceptrons and other networks is a numerical, iterative process that will be discussed in Chapter 5. It has been rigorously proven that training perceptrons for classification problems will converge to a solution in a finite number of steps, if a solution exists. 

Schneider and others (Schneider et al., 1993) applied the perception approach to identifying cleavage sites in protein sequences. Again, a matrix approach was used, with 12 rows representing a sequence of 12 amino acid residues (one row per residue) and four columns representing physico-chemical features of each residue hydrophobicity, hydrophilicity, polarity and volume. The trained perceptron predicted cleavage sites correctly in 100% of test cases. 

The Back-Propagation Algorithm (BPA) is a supervised learning method for training ANNs, and is one of the most common forms of training techniques. It uses a gradient-descent optimization method, also referred to as the delta rule when applied to feedforward networks. A feedforward network that has employed the delta rule for training, is called a Multi-Layer Perceptron (MLP). 

Alvarez et al. [73] compared the performance of LDA and ANNs to classify different classes of wines. Metal concentrations (Ca, Mg, Sr, Ba, K, Na, P, Fe, Al, Mn, Cu and Zn) were selected as chemical descriptors for discrimination because of their correlation with soil nature, geographical origin and grape variety. Both LDA and ANNs led to a perfect separation of classes, especially when multi-layer perceptron nets were trained by error back-propagation. 

The most widely used solution to this problem is backpropagation of errors. Let us represent as opk the certain instant output of perceptron k for a sample p in the training set, calculated with an expression like Eq. 

The rest of the paper is organized as follows. The Section 2 describes attack classification and training data set. In the Section 3 the intrusion detection system is described, based on neural network approach. Section 4 presents the nonlinear PCA neural network and multilayer perceptron for identification and classification of computer network attack. In Section 5 the results of experiments are presented. Conclusion is given in Section 6. 

The characters are first normalized by rotating the original scanned image to correct for scanning error and by combinations of scaling under sampling and contrast and density adjustments of the scanned characters. In operation, the normalized characters are then presented to a multilayer perceptron neural network for recognition the network was trained on exemplars of characters form numerous serif and sans serif fonts to achieve font invariance. Where the output from the neural network indicates more than one option, for example 5 and s, the correct interpretation is determined from context. 

After the Minsky and Papert book in 1969 (Minsky Papert, 1969) which clarified the linearity restrictions of perceptrons, little work was done with perceptrons. However, in 1986 McClelland and Rumelhart (McClelland Rumelhart, 1986) revived the field with multilayer perceptrons and an intuitive training algorithm called back-propagation (discussed in Chapter 5). 

Networks based on radial basis functions have been developed to address some of the problems encountered with training multilayer perceptrons radial basis functions are guaranteed to converge and training is much more rapid. Both are feed-forward networks with similar-looking diagrams and their applications are similar however, the principles of action of radial basis function networks and the way they are trained are quite different from multilayer perceptrons. 

The only difficult part is finding the values for p. and o for each hidden unit, and the weights between the hidden and output layers, Le., training the network. This will be discussed later, in Chapter 5. At this point, it is sufficient to say that training radial basis function networks is considerably faster than training multilayer perceptrons. On the other hand, once trained, the feed-forward process for multilayer perceptrons is faster than for radial basis function networks. 

The perceptron network for this problem is shown in Figure 5.1 below. Input from xo, the bias unit, is always -1. The length of the input vector is 2, and there are 18 input vectors in the training set. 

Figure 5.1 Simple perceptron network for training example.

There are modifications to the perceptron learning rule to help effect faster convergence. The Widrow-Hoff delta rule (Widrow Hoff, 1960) multiplies the delta term by a number less than 1, called the learning rate, tv This effectively causes smaller changes to be made at each step. There are heuristic rules to decrease T] as training time increases the idea is that big changes may be taken at first and as the final solution is approached, smaller changes may be desired. 

One of the early problems with multilayer perceptrons was that it was not clear how to train them. The perception training rule doesn t apply directly to networks with hidden layers. Fortunately, Rumelhart and others (Rumelhart et al 1986) devised an intuitive method that quickly became adopted and revolutionized the field of artificial neural networks. The method is called back-propagation because it computes the error term as described above and propagates the error backward through the network so that weights to and from hidden units can be modified in a fashion similar to the delta rule for perceptions. 

The number of nodes or perceptrons in the hidden layer(s) characterizes the network architecture. Consequently, papers in the literature will state that they used one hidden layer with five nodes, for example. Frequently, users tune the network by selecting different numbers of nodes in the hidden layer, to find which gives the best fit to data. Again, when more nodes are added, this increases the risk of overtraining since more parameters will be used in the fit. The aim should be to use the minimum number of nodes that will provide a fit to the training data. 

The training of the perceptron as a linear classifier then follows the following... 

---