Connection weights

A sigmoid (s-shaped) is a continuous function that has a derivative at all points and is a monotonically increasing function. Here 5,p is the transformed output asymptotic to 0 < 5/,p I and w,.p is the summed total of the inputs (- 00 < Ui p < -I- 00) for pattern p. Hence, when the neural network is presented with a set of input data, each neuron sums up all the inputs modified by the corresponding connection weights and applies the transfer function to the summed total. This process is repeated until the network outputs are obtained. 

There are three inputs in this application we choose to feed in the Cartesian coordinates X and Y of a data point through two of the inputs the third input is provided by a bias node, which produces a constant signal of 1.0. The values shown beside each connection are the connection weights. 

As we shall see in the next section, the output of a node is computed from its total input the bias provides a threshold in this computation. Suppose that a node follows a rule that instructs it to output a signal of +1 if its input is greater than or equal to zero, but to output zero otherwise. If the input signal from the bias node, after multiplication by the connection weight, was +0.1, the remaining inputs to the node would together have to sum to a value no smaller than -0.1 in order to trigger a... 

How did the network do this Encoded in the network s connection weights is the equation of the line that separates the two groups of points this is... 

In the network in Figure 2.11, the connection weight on the X input signal and the bias node are equal to the coefficients of X1 and X° in equation (2.10), while the connection weight on the Y input is -1.0. When the node calculates the sum of the weighted inputs, it is computing ... 

Indeed, if the problem is simple enough that the connection weights can be found by a few moments work with pencil and paper, there are other computational tools that would be more appropriate than neural networks. It is in more complex problems, in which the relationships that exist between data points are unknown so that it is not possible to determine the connection weights by hand, that an ANN comes into its own. The ANN must then discover the connection weights for itself through a process of supervised learning. 

The ability of an ANN to learn is its greatest asset. When, as is usually the case, we cannot determine the connection weights by hand, the neural network can do the job itself. In an iterative process, the network is shown a sample pattern, such as the X, Y coordinates of a point, and uses the pattern to calculate its output it then compares its own output with the correct output for the sample pattern, and, unless its output is perfect, makes small adjustments to the connection weights to improve its performance. The training process is shown in Figure 2.13. 

To improve performance, the connection weights are now adjusted by an amount that is related to how good the match was between target response and network output. This is measured by the error, 8, which, if the network contains just a single output node, is the difference between the output of the node, y, and the target response, t. 

If the actual output and the target output are identical, the network has worked perfectly for this sample, therefore, no learning need take place. Another sample is drawn from the database and fed into the network and the process is continued. If the match is not perfect, the network needs to learn to do better. This is accomplished by adjusting the connection weights so that the next time the network is shown the same sample it will provide an output that is closer to the target response. 

Suppose that Figure 2.7 shows the initial connection weights for a network that we wish to train. The first sample taken from the database is X = 0.16, Y = 0.23, with a target response of 0.27. The node uses the identity function to determine its output, which is therefore ... 

Once the network geometry and the type of activation function have been chosen, the network will be ready to use as soon as the connection weights have been determined, which requires a period of training. 

Backpropagation has two phases. In the first, an input pattern is presented to the network and signals move through the entire network from its inputs to its outputs so the network can calculate its output. In the second phase, the error signal, which is a measure of the difference between the target response and actual response, is fed backward through the network, from the outputs to the inputs and, as this is done, the connection weights are updated. 

To reduce the chance that the network will be trapped by a set of endlessly oscillating connection weights, a momentum term can be added to the update of the weights. This term adds a proportion of the update of the weight in the previous epoch to the weight update in the current epoch ... 

As the network learns, connection weights are adjusted so that the network can model general rules that underlie the data. If there are some general rules that apply to a large proportion of the patterns in the dataset, the network will repeatedly see examples of these rules and they will be the first to be learned. Subsequently, it will turn its attention to more specialized rules of which there are fewer examples in the dataset. Once it has learned these rules as well, if training is allowed to continue, the network may start to learn specific samples within the data. This is undesirable for two reasons. Firstly, since these particular patterns may never be seen when the trained network is put to use, any time spent learning them is wasted. Secondly,... 

We saw in chapter 1 that Artificial Intelligence algorithms incorporate a memory. In the ANN the memory of the system is stored in the connection weights, but in the SOM the links are inactive and the vector of weights at each node provides the memory. This vector is of the same length as the dimensionality of points in the dataset (Figure 3.6). 

---