Training multilayer perceptions

Not all neural networks are the same their connections, elemental functions, training methods and applications may differ in significant ways. The types of elements in a network and the connections between them are referred to as the network architecture. Commonly used elements in artificial neural networks will be presented in Chapter 2. The multilayer perception, one of the most commonly used architectures, is described in Chapter 3. Other architectures, such as radial basis function networks and self organizing maps (SOM) or Kohonen architectures, will be described in Chapter 4. 

A multilayer perception with two hidden units is shown in Figure 3.7, with the actual weights and bias terms included, after training. This network can solve the nonlinear depression classification problem, presented in Figure 3.4. 

The same limitations that apply for multilayer perception networks, in general, hold for radial basis function networks. Training time is enhanced with radial basis function networks, but application is slower, due to the complexity of the calculations. Radial basis function networks require supervised training and hence are limited to those applications for which training data and answers are available. Several books are listed in the reference section with excellent descriptions of radial basis function networks and applications (Beale Jackson, 1991 Fu, 1994 Wasserman, 1993). 

Perceptions, multilayer perceptions and radial basis function networks require supervised training with data for which the answers are known. Some applications require the automatic clustering of data, data for which the clusters and clustering criteria are not known. One of the best known architectures for such problems is the Kohonen selforganizing map (SOM), named after its inventor, Teuvo Kohonen (Kohonen, 1997). In this section the rationale behind such networks is described. 

In Part II of this book we have encountered three network architectures that require supervised learning perceptions, multilayer perceptions and radial basis function networks. Training for perceptions and multilayer perceptions is similar. The goal of... 

One of the early problems with multilayer perceptrons was that it was not clear how to train them. The perception training rule doesn t apply directly to networks with hidden layers. Fortunately, Rumelhart and others (Rumelhart et al 1986) devised an intuitive method that quickly became adopted and revolutionized the field of artificial neural networks. The method is called back-propagation because it computes the error term as described above and propagates the error backward through the network so that weights to and from hidden units can be modified in a fashion similar to the delta rule for perceptions. 

Since multilayer perceptions use neurons that have differentiable functions, it was possible, using the chain rule of calculus, to derive a delta rule for training similar in form and function to that for perceptions. The result of this clever mathematics is a powerful and relatively efficient iterative method for multilayer perceptions. The rule for changing weights into a neuron unit becomes... 

Radial basis function networks and multilayer perceptions have similar functions but their training algorithms are dramatically different. Training radial basis function networks proceeds in two steps. First the hidden layer parameters are determined as a function of the input data and then the weights between the hidden and output layers are determined from the output of the hidden layer and the target data. 

Bayesian techniques A method of training and evaluating neural networks that is based on a stochastic (probabilistic) approach. The basic idea is that weights have a distribution before training (a prior distribution) and another (posterior) distribution after training. Bayesian techniques have been applied successfully to multilayer perception networks. 

Initially, networks were trained from data obtained from the experimental design conditions given in Figure 7.3. These were radial basis function (RBF) networks, multilayer perception (MLP) networks, probabilistic neural networks (PNNs), and generalized regression neural networks (GRNNs), as well... 

The one out of two multilayer perceptions which better approximates the unknown dependence (or the one out of the set of MLPs which approximates that dependence best) is indicated by the corresponding MSE values on test data, not by the values on training data. 

We have used both these methods to check the impression given by the above figures. The results are listed in Table 8.2. They clearly confirm that the order of errors of approximations computed with the trained multilayer perceptions for catalytic materials from the seventh generation and the order of the mean cross-validation errors of their architectures correlate. That correlation is substantially more significant if the errors of approximations are measured in the same way as during the architecture search, i.e., using MSB. 

Artificial neural networks (ANNs) are computer-based systems that can learn from previously classified known examples and can perform generalized recognition - that is, identification - of previously unseen patterns. Multilayer perceptions (MLPs) are supervised neural networks and, as such, can be trained to model the mapping of input data (e.g. in this study, morphological character states of individual specimens) to known, previously defined classes. 

---