Neural network Neuron activation

In theory one hidden layer neural network is sufficient to describe all input/output relations. More hidden layers can be introduced to reduce the number of neurons compared to the number of neurons in a single layer neural network. The same argument holds for the type of activation function and the choice of the optimisation algorithm. However, the emphasis of this work is not directed on the selection of the best neural network structure, activation function and training protocol, but to the application of neural networks as a means of non-linear function fit. 

Neural networks can also be classified by their neuron transfer function, which typically are either linear or nonlinear models. The earliest models used linear transfer functions wherein the output values were continuous. Linear functions are not very useful for many applications because most problems are too complex to be manipulated by simple multiplication. In a nonlinear model, the output of the neuron is a nonlinear function of the sum of the inputs. The output of a nonlinear neuron can have a very complicated relationship with the activation value. 

The basic component of the neural network is the neuron, a simple mathematical processing unit that takes one or more inputs and produces an output. For each neuron, every input has an associated weight that defines its relative importance, and the neuron simply computes the weighted sum of all the outputs and calculates an output. This is then modified by means of a transformation function (sometimes called a transfer or activation function) before being forwarded to another neuron. This simple processing unit is known as a perceptron, a feed-forward system in which the transfer of data is in the forward direction, from inputs to outputs, only. 

The recurrent network models assume that the structure of the network, as well as the values of the weights, do not change in time. Moreover, only the activation values (i.e., the output of each processor that is used in the next iteration) changes in time. In the biochemical network one cannot separate outputs and weights. The outputs of one biochemical neurons are time dependent and enter the following biochemical neurons as they are. However, the coefficients involved in these biochemical processes are the kinetic constants that appear in the rate equations, and these constants are real numbers. The inputs considered in biochemical networks are continuous analog numbers that change over time. The inputs to the recurrent neural networks are sets of binary numbers. 

There are many similarities between recurrent neural networks and the biochemical networks presented in this work. However, the dissimilarities reviewed here are very closely related to the inherent characteristics of biochemical systems, such as their kinetic equations. Thus, in future work one may consider recurrent neural networks that are similar to biochemical networks— having the same activation function and the same connections between neurons—and this approach will allow one to assess their computational capabilities. 

Figure 8.2 A motor neuron (a) and small artificial neural network (b). A neuron collects signals from other neurons via its dendrites. If the neuron is sufficiently activated, it sends a signal to other neurons via its axon. Artificial neural network are often grouped into layers. Data is entered through the input layer. It is processed by the neurons of the hidden layer and then fed to the neurons of the output layer. (Illustration of motor neuron from Life ART Collection Images 1989-2001 by Lippincott Williams Wilkins used by permission from SmartDraw.com.)...

The neurons weight all inputs and provide an output via the activation function. The complexity of the neural networks used will be determined by the number of nodes in the hidden layer (2,3,5 or 7). The activation applied in this application is a hyperbolic tangent function. In mathematical terms, the output of neuron j is defined by n With yj output of neuron j... 

Hidden neurons communicate only with other neurons. They are part of the large internal pattern that determines a solution to the problem. The information that is passed from one processing element to another is continued within a set of weights. Some of the interconnections are strengthened and some are weakened, so that a neural network will output a more corrected answer. The activation of a neuron is defined as the sum of the weighted input signals to that neuron ... 

---