Overfitting networks

In all modeling techniques, and neural networks in particular, care must be taken not to overtrain or overfit the model. 

Zhang et al.14 develop a neural network approach to bacterial classification using MALDI MS. The developed neural network is used to classify bacteria and to classify culturing time for each bacterium. To avoid the problem of overfitting a neural network to the large number of channels present in a raw MALDI spectrum, the authors first normalize and then reduce the dimensionality of the spectra by performing a wavelet transformation. 

Overfitting arises when the network learns for too long. For most students, the longer they are trained the more they learn, but artificial neural networks are different. Since networks grow neither bored nor tired, it is a little odd that their performance can begin to degrade if training is excessive. To understand this apparent paradox, we need to consider how a neural network learns. 

Overfitting is a potentially serious problem in neural networks. It is tackled in two ways (1) by continually monitoring the quality of training as it occurs using a test set, and (2) by ensuring that the geometry of the network (its size and the way the nodes are connected) is appropriate for the size of the dataset. 

This method for preventing overfitting requires that there are enough samples so that both training and test sets are representative of the dataset. In fact, it is desirable to have a third set known as a validation set, which acts as a secondary test of the quality of the network. The reason is that, although the test set is not used to train the network, it is nevertheless used to determine at what point training is stopped, so to this extent the form of the trained network is not completely independent of the test set. 

A quite different way to reduce overfitting is to use random noise. A random signal is added to each data point as it is presented to the network, so that a data pattern ... 

However, the choice of experiments is still important for the artificial neural network approach, and it is best selected in a regular pattern. The central composite design, in which each factor takes five levels, is a generally a good compromise. " Great care must be taken not to overfit, and, in general, more experiments are required than for the classic RSM approach. 

If the ANNs present numerous advantages, they also present some weaknesses. Indeed they are versatile tools, but they require some experience to be correctly used to correctly scale the data, tune the network, and avoid overtraining and overfitting. More complete critical reviews of the ANN models can be found elsewhere [2,97-100],... 

Figure 15-7 Neural networks provide ultimate freedom of discrimination, with the additional danger of overfitting a solution.The curved line separating the two populations for this specific set of data may not work well in a subsequent sample since it is custom molded to separate this test set.

---