The Training of Artificial Neural Networks

Typical choices for the activation function are, among others, the sigmoid function  

The computed output of the sigmoid and Gaussian function can only fall between 0 and 1 and therefore, the output data used to train the neural network needs to be scaled to the range of 0-1 (-1 and 1 for the hyperbohc tangent function). Since the extremes of 0 to 1 occur only when the input to the sigmoid function is to +°o, the output data are often scaled to be between approximately 0.1 to 0.9. However, non-scaled data may be used if a linear activation function is used for the output layer. 

When the neuron is situated in an input or hidden layer its output will be sent to all the neurons in the next layer. They will use it, after it is multiplied with the weight of the link, as input. When the neuron is situated in the output layer, its output is (along with other outputs from neurons in the same layer) the output of the network. 

From the structure of a neural network it can be stated that the information learned is stored in the weights of the links. For different problems different stmctures can be used. The links do not have to point in the same direction, recurrent networks, in which delayed outputs are used as inputs, are also very common for process modehng since it can describe time dependency. Also all neurons in a network can be coimected with each other. For this type of network, the input, hidden and output neurons have to be strictly defined. However, the basic principle as explained above is valid for all these networks. 

Hertz et al. (1995) described in their work that in most cases a neural network consisting of three layers will give good results. This statement is supported by the hterature where neural networks were applied in process control. In all of these cases a network consisting of an input layer, one hidden layer and an output layer was used. 

---