Perceptron function

Consider the Boolean exclusive-OR (or XOR) function that we used as an example of a linearly inseparable problem in our discussion of simple perceptrons. In section 10.5.2 we saw that if a perceptron is limited to having only input and output layers (and no hidden layers), and is composed of binary threshold McCulloch-Pitts neurons, the value y of its lone output neuron is given by... 

Being able to construct an e xplicit solution to a nonlinearly separable problem such as the XOR-problem by using a multi-layer variant of the simple perceptron does not, of course, guarantee that a multi-layer perceptron can by itself learn the XOR function. We need to find a learning rule that works not just for information that only propagates from an input layer to an output layer, but one that works for information that propagates through an arbitrary number of hidden layers as well. 

As we will discuss a bit later on in this section, adding hidden layers but generalizing the McCulloch-Pitts step-function thresholding to a linear function yields a multi-layer perceptron that is fundamentally no more powerful than a simple perceptron that has no hidden layers and uses the McCulloch-Pitts step-function threshold. 

The basic backpiopagation algorithm described above is, in practice, often very slow to converge. Moreover, just as Hopfield nets can sometimes get stuck in undesired spurious attractor states (i.e. local minima see section 10.6.5), so to can multilayer perceptrons get trapped in some undesired local minimum state. This is an unfortunate artifact that plagues all energy (or cost-function) minimization schemes. 

In our discussion of Hopfield nets in section 10.6, we found that the maximal number of patterns that can be stored before their stability is impaired is some linear function of the size of the net n, ax aN, where 0 < a < 1 and N is the number of neurons in the net (see sections 10.6.6 and 10.7). A similar question can of course be asked of perceptroiis How many input-output fact pairs can a perceptron of given size learn ... 

A graph of this function shows that it is not until the number of points n is some sizable fraction of 2( V + 1) that an (N - l)-dimensional hyperplane becomes over constrained by the requirement to correctly separate out (N + 1) or fewer points. In therefore turns out that the capacity of a simple perceptron is given by a rather simple expression if the number of output neurons is small and independent of N, then, as —> oo, the maximum number of input-output fact pairs that can be... 

The basic component of the neural network is the neuron, a simple mathematical processing unit that takes one or more inputs and produces an output. For each neuron, every input has an associated weight that defines its relative importance, and the neuron simply computes the weighted sum of all the outputs and calculates an output. This is then modified by means of a transformation function (sometimes called a transfer or activation function) before being forwarded to another neuron. This simple processing unit is known as a perceptron, a feed-forward system in which the transfer of data is in the forward direction, from inputs to outputs, only. 

Although the minimization of the objective function might run to convergence problems for different NN structures (such as backpropagation for multilayer perceptrons), here we will assume that step 3 of the NN algorithm unambiguously produces the best, unique model, g(x). The question we would like to address is what properties this model inherits from the NN algorithm and the specific choices that are forced. 

The transfer function of the hidden units in MLF networks is always a sigmoid or related function. As can be seen in Fig. 44.5b, 0, represents the offset, and has the same function as in the simple perceptron-like networks. P determines the slope of the transfer function. It is often omitted in the transfer function since it can implicitly be adjusted by the weights. The main function of the transfer function is modelling the non-linearities in the data. In Fig. 44.11 it can be seen that there are five different response regions in the sigmoidal function ... 

The region from A to D is called the dynamic range. The regions 2 and 4 constitute the most imfwrtant difference with the hard delimiter transfer function in perceptron networks. These regions rather than the near-linear region 3 are most important since they assure the non-linear response properties of the network. It may... 

All of the studies above have used back propagation multilayer perceptrons and many other varieties of neural network exist that have been applied to PyMS data. These include minimal neural networks,117119 radial basis functions,114120 self-organizing feature maps,110121 and autoassociative neural networks.122123... 

Fig. 6.18. Schematic representation of a multilayer perceptron with two input neurons, three hidden neurons (with sigmoid transfer functions), and two output neurons (with sigmoid transfer functions, too)...

The relationship between the summed inputs to a neuron and its output is an important characteristic of the network, and it is determined by a transfer function (or squashing function or activation function). The simplest of neurons, the perceptron, uses a step function for this purpose, generating an output of zero unless the summed input reaches a critical threshold (Figure 7) for a total input above this level, the neuron fires and gives an output of one. 

The problem with the behavior of the perceptron lies in the transfer function if a neuron is to be part of a network capable of genuine learning, the step function used in the perceptron must be replaced by an alternative function that is slightly more sophisticated. The most widely used transfer function is sigmoidal in shape (Figure 8, Eq. [2]), although a linear relationship between input and output signals is used occasionally. 

Mathematically, each output represents just a specific linear combination with specific weights from each preceding perceptron, though passage from layer to layer is also modulated by the transfer function used (usually a given transfer function for all nodes in a given layer) ... 

The perceptron network is the simplest of these three methods, in that its execution typically involves the simple multiplication of class-specific weight vectors to the analytical profile, followed by a hard limit function that assigns either 1 or 0 to the output (to indicate membership, or no membership, to a specific class). Such networks are best suited for applications where the classes are linearly separable in the classification space. 

Among the most wide-spread neural networks are feedforward networks, namely multilayer perceptron (MLP). This network type has been proven to be universal function approximators [11], Another important feature of MLP is the ability to generalization. Therefore MLP can be powerful tool for design of intrusion detection systems. 

The functional unit of ANNs is the perceptron. This is a basic unit able to generate a response as a funtion of a number of inputs received from others perceptrons. For example, the response value can be obtained as follows ... 

For an example of a perception used in nucleic acid research, see Stormo et al. (1982). In this study, a perceptron was used to find a weighting function, which distinguishes E. coli... 

There are literally dozens of kinds of neural network architectures in use. A simple taxonomy divides them into two types based on learning algorithms (supervised, unsupervised) and into subtypes based upon whether they are feed-forward or feedback type networks. In this chapter, two other commonly used architectures, radial basis functions and Kohonen self-organizing architectures, will be discussed. Additionally, variants of multilayer perceptrons that have enhanced statistical properties will be presented. 

Networks based on radial basis functions have been developed to address some of the problems encountered with training multilayer perceptrons radial basis functions are guaranteed to converge and training is much more rapid. Both are feed-forward networks with similar-looking diagrams and their applications are similar however, the principles of action of radial basis function networks and the way they are trained are quite different from multilayer perceptrons. 

---