Chance effects

One of the problems with regression analysis, and other supervised learning methods, is that they seek to fit a model. This may seem like a curious statement to make, to criticize a method for doing just what it is intended to do. The reason that this is a problem is that given sufficient opportunity to fit a model then regression analysis will find an equation to fit a data set. What is meant by sufficient opportunity It has been 

Plot of mean versus the number of screened variables for regression equations generated for sets of random numbers (from Topliss and Edwards 1979, copyright (1979) 

---

This approach employs statistical methods that use no obvious theory-derived basis, but which derive usable relationships from realistic inputs. It is beyond the scope of this review to describe the methods and their validation in detail. Useful reviews are available (Livingstone, 2000 2003) and more details are provided in Chapter 3. The methods may be divided into two classes, often referred to as those derived from supervised and unsupervised learning. In the latter, the techniques used are more free to explore relationships between variables, and are therefore less likely to produce chance effects. 

Generation of the PCs is an unsupervised learning process that may help to reduce the possibility of chance effects. 

Livingstone, D.J. and Manallack, D.T., Statistics using neural networks chance effects, J. Med. Chem., 36, 1295-1297, 1993. 

Manallack, D.T. and Livingstone, D.J., Artificial neural networks application and chance effects for QSAR data analysis, Med. Chem. Res., 2, 181-190, 1992. 

There may be as great as a 1,000-fold variation in bacterial counts from leaf to leaf assayed from apparently similar leaves in similar canopy positions at the same sampling period (35,100). Most data sets obtained from plate counts of bean and corn leaves are described by a log normal distribution, indicating that on individual leaves or leaHets the spatial distribution of bacterial cells are not uniform within a canopy (35,100). A log normal distribution tends to reflect the fact that chance effects of random variables are acting on a large and diverse collection of objects (in this case bacteria). Interestingly, a similar study on wheat leaves indicated that bacterial counts were best described by a log normal distribution, but counts of yeasts (mainly Cryptococcus spp. and Sporobolomyces spp.) and Cladosporium spp. colonies appeared normally distributed (101). Much more work needs to be done, particularly with regard to host, host age, environmental factors, and position of the sample within a canopy, before the spatial distribution of these microbes can adequately be described. 

In evaluating correlations for chance effects attention should also be paid to whether some observations have been dropped from the set in order to obtain a better data fit. This is a fairly common occurrence and usually gives much higher r2 values. However, sometimes the Justification given for dropping the poorly fit observations are questionable. 

A major source of confusion contributing to the debate on the safety of pesticide residues in food, and low-level exposure to environmental chemicals, is the relationship between a chemical exposure or dose and the observed effect. Some pesticides to which we are exposed in our fruit and vegetables are capable of causing harm to humans if given in high enough quantities. This is usually limited to occupational exposure where high toxic doses may actually be encountered. If the need for statistical analysis to rule out chance effects is the first scientific tenet that forms the cornerstone of our modern science-based medicine, the dose-response relationship is the second. 

No assumptions concerning mechanism and may identify different mechanisms Unsupervised learning so chance effects unlikely Does not require biological data Non-linear... 

Multiple regression—robustness, chance effects, and the comparison of models... 

This chapter has described some of the more commonly used supervised learning methods for the analysis of data discriminant analysis and its relatives for classified dependent data, variants of regression analysis for continuous dependent data. Supervised methods have the advantage that they produce predictions, but they have the disadvantage that they can suffer from chance effects. Careful selection of variables and test/training sets, the use of more than one technique where possible, and the application of common sense will all help to ensure that the results obtained from supervised learning are useful. 

AI research has already provided the concepts of supervised and unsupervised learning to data analysis, and these have proved useful in the classification of analytical methods and to alert us to the potential danger of chance effects. But what of the application of AI techniques themselves... 

Liu et al. described the functional-link net architecture (FUNCLINK) (158). They claimed that, compared with methods such as adaptive least squares and back-propagation neural nets, FUNCLINK exhibited good recognition and high predictivity. However, Manallack and Livingstone found two disadvantages to the FUNCLINK technique (159). First, the natural ability of neural networks to develop nonlinear relationships is removed with FUNCLINK as these must be specified. Second, the large number of enhanced parameters produced by FUNCLINK increases the possibility of chance effects. They concluded that FUNCLINK adds little to the field of QSAR data analysis. 

---