Neurons transfer function

Neural networks can also be classified by their neuron transfer function, which typically are either linear or nonlinear models. The earliest models used linear transfer functions wherein the output values were continuous. Linear functions are not very useful for many applications because most problems are too complex to be manipulated by simple multiplication. In a nonlinear model, the output of the neuron is a nonlinear function of the sum of the inputs. The output of a nonlinear neuron can have a very complicated relationship with the activation value. 

Practically all forms of neuron transfer functions include the summation operation, i.e., the sum of all inputs into the neuron (multiplied by their connection strength or weights) is calculated. In mathematical terms,... 

Figure 9-13. Artificial neuron the signals x, are weighted (with weights IV,) and summed to produce a net signal Net. This net signal is then modified by a transfer function and sent as an output to other neurons,...

The net signal is then modified by a so-called transfer function and sent as output to other neurons. The most widely used transfer function is sigmoidal it has two plateau areas having the values zero and one. and between these an area in which it is increasing nonlinearly. Figure 9-15 shows an example of a sigmoidal transfer function. 

The basic model of a single artificial neuron consists of a weighted summer and an activation (or transfer) function as shown in Figure 10.20. Figure 10.20 shows a neuron in the yth layer, where... 

A sigmoid (s-shaped) is a continuous function that has a derivative at all points and is a monotonically increasing function. Here 5,p is the transformed output asymptotic to 0 < 5/,p I and w,.p is the summed total of the inputs (- 00 < Ui p < -I- 00) for pattern p. Hence, when the neural network is presented with a set of input data, each neuron sums up all the inputs modified by the corresponding connection weights and applies the transfer function to the summed total. This process is repeated until the network outputs are obtained. 

Kolmogorov s Theorem (Reformulated by Hecht-Nielson) Any real-valued continuous function f defined on an N-dimensional cube can be implemented by a three layered neural network consisting of 2N -)-1 neurons in the hidden layer with transfer functions from the input to the hidden layer and (f> from all of... 

Kolmogorov s theorem thus effectively states that a three-layer net with N 2N -)-1) neurons using continuously increasing nonlinear transfer functions can compute any continuous function of N variables. Unfortunately, the theorem tells us nothing about how to select the required transfer functions or set the weights in our net. 

Neurons are not used alone, but in networks in which they constitute layers. In Fig. 33.21 a two-layer network is shown. In the first layer two neurons are linked each to two inputs, x, and X2- The upper one is the one we already described, the lower one has w, = 2, W2 = 1 and also 7= 1. It is easy to understand that for this neuron, the output )>2 is 1 on and above line b in Fig. 33.22a and 0 below it. The outputs of the neurons now serve as inputs to a third neuron, constituting a second layer. Both have weight 0.5 and 7 for this neuron is 0.75. The output yfi j, of this neuron is 1 if E = 0.5 y, + 0.5 y2 > 0.75 and 0 otherwise. Since y, and y2 have as possible values 0 and 1, the condition for 7 > 0.75 is fulfilled only when both are equal to 1, i.e. in the dashed area of Fig. 33.22b. The boundary obtained is now no longer straight, but consists of two pieces. This network is only a simple demonstration network. Real networks have many more nodes and transfer functions are usually non-linear and it will be intuitively clear that boundaries of a very complex nature can be developed. How to do this, and applications of supervised pattern recognition are described in detail in Chapter 44 but it should be stated here that excellent results can be obtained. 

Fig. 44.2. An artificial neuron x,. ..Xp are the incoming signals wi... Wp are the corresponding weight factors and F is the transfer function.

Fig. 6.18. Schematic representation of a multilayer perceptron with two input neurons, three hidden neurons (with sigmoid transfer functions), and two output neurons (with sigmoid transfer functions, too)...

The relationship between the summed inputs to a neuron and its output is an important characteristic of the network, and it is determined by a transfer function (or squashing function or activation function). The simplest of neurons, the perceptron, uses a step function for this purpose, generating an output of zero unless the summed input reaches a critical threshold (Figure 7) for a total input above this level, the neuron fires and gives an output of one. 

The problem with the behavior of the perceptron lies in the transfer function if a neuron is to be part of a network capable of genuine learning, the step function used in the perceptron must be replaced by an alternative function that is slightly more sophisticated. The most widely used transfer function is sigmoidal in shape (Figure 8, Eq. [2]), although a linear relationship between input and output signals is used occasionally. 

The first layer transmits the value of the predictors to the second— hidden—layer. All the neurons of the input layer are connected to the / neurons of the second layer by means of weight coefficients, meaning that the / elements of the hidden layer receive, as information, a weighted sum S of the values from the input layer. They transform the information received (S) by means of a suitable transfer function, frequently a sigmoid. 

A biological neuron can be active (excited) or inactive (not excited). Similarly, the artificial neurons can also have different activation status. Some neurons can be programmed to have only two states (active/inactive) as the biological ones, but others can take any value within a certain range. The final output or response of a neuron (let us call it a) is determined by its transfer function, f, which operates on the net signal (Netj) received by the neuron. Hence the overall output of a neuron can be summarised as ... 

The numerical value of aj determines whether the neuron is active or not. The bias, 0j, should also be optimised during training [8]. The activation function, ranges currently from 0 to 1 or from — 1 to +1 (depending on the mathematical transfer function, /). When a, is 0 or — 1 the neuron is totally inactive,... 

Before training the net, the transfer functions of the neurons must be established. Here, different assays can be made (as detailed in the previous sections), but most often the hyperbolic tangent function tansig function in Table 5.1) is selected for the hidden layer. We set the linear transfer function purelin in Table 5.1) for the output layer. In all cases the output function was the identity function i.e. no further operations were made on the net signal given by the transfer function). 

And the firing of the neuron happens if the threshold, as defined by a transference function used, is surpassed ... 

Our choice for the non-linear system approach to PARC is the ANN. The ANN is composed of many neurons configured in layers such that data pass from an input layer through any number of middle layers and finally exit the system through a final layer called the output layer. In Fig. 4 is shown a diagram of a simple three-layer ANN. The input layer is composed of numeric scalar data values, whereas the middle and output layers are composed of artificial neurons. These artificial neurons are essentially weighted transfer functions that convert their inputs into a single desired output. The individual layer components are referred to as nodes. Every input node is connected to every middle node, and every middle node is connected to every output node. 

So, the basic neuron can be seen as having two operations, summation and thresholding, as illustrated in Figure 2.5. Other forms of thresholding and, indeed, other transfer functions are commonly used in neural network modeling some of these will be discussed later. For input neurons, the transfer function is typically assumed to be unity, i.e., the input signal is passed through without modification as output to the next layer F(x) = 1.0. 

---