More Flexible Activation Functions

The creation of a layered structure that includes one or more hidden layers of nodes as well as input and output layers takes us halfway along the road to a more versatile network the journey is completed by the introduction of a more responsive activation function. 

Although the linear activation function passes more information from the input to a node to its output than a binary function does, it is of limited value in layered networks as two nodes in succession that both use a linear activation function are equivalent to a single node that employs the same function, thus adding an extra layer of nodes does not add to the power of the network. This limitation is removed by the use of curved activation functions. 

---

Although some problems in more than two dimensions are linearly separable (in three dimensions, the requirement for linear separability is that the points are separated by a single plane, Figure 2.17), almost all problems of scientific interest are not linearly separable and, therefore, cannot be solved by a one-node network thus more sophistication is needed. The necessary additional power in the network is gained by making two enhancements (1) the number of nodes is increased and (2) each node is permitted to use a more flexible activation function. 

NMR is unique in that it can provide detailed and specific information on molecular dynamics in addition to structural information. The use of relaxation time measurements allows the relative mobility of individual atomic positions within a macromolecule to be determined. The d3mamic information obtained includes not only the rates or frequencies of internal motions but also their amplitudes. Such amplitudes are often expressed by order parameters. Not surprisingly, it is observed in many cases that the termini of proteins are more flexible than internal regions. More interestingly, NMR has provided a number of examples where internal loops in proteins have been shown to have dynamics that may be associated with their function. A good example of this is HIV protease, where NMR studies have identified reduced-order parameters in the flap region of the molecule that may reflect flexibility to allow entry of substrates or inhibitors into the active site. 

---