Assessment of Predictivity

The OECD guidelines stipulate that be calculated according to the following equation  

Where the for external validation should be calculated with the sum of squared deviations referring to the training set mean. The use of the training set mean is pragmatic as it provides a fixed reference value, enabling comparison of for differing test sets. But this formalism has been recently been questioned 

However, using the test set mean can significantly underestimate the predictive ability of a model. Ambiguity occurs with the statistic when the test set data is not evenly distributed over the range of the training set. As the variance of the external test set approaches the RMSE of the fitted model, the measure would approach zero, even though it would appear that the predictions are in accordance with the model. Consonni defined a new statistic that expresses the mean predicted error sum of squared deviations between the observed and predicted values for the test set, over the mean training set sum of squared deviations from the mean value  

PRESS is predicted error sum of squares deviations between the observed and measured y values 

TSS is the training set sum of squared deviations from the mean Consonni demonstrated that this formulation of is stable with test sets of difierent variances. 

---

Marshal G, Grover FL, Henderson WG, Hammermeister KE. Assessment of predictive models for binary outcomes an empirical approach using operative death from cardiac surgery. Stat Med 1994 13 1501-11. 

Every medical research project involving human subjects should be preceded by careful assessment of predictable risks and burdens in comparison with foreseeable benefits to the subject or to others. This does not preclude the participation of healthy volunteers in medical research. The design of all studies should be publicly available. 

For illustrative purposes, it can be assumed that the following predictivity criteria were established before undertaking the independent assessment of predictive capacity ... 

Tong W, Xie Q, Hong H, Shi LM, Fang H, Perkins R. Assessment of prediction confidence and domain extrapolation of two structure-activity relationship models for predicting estrogen receptor binding activity. Environ Health Perspect 2004 112 1249-54. 

E. Sternberg. Recognition of analogous and homologous protein folds assessment of prediction success and associated alignment accuracy using empirical substitution matrices. 

A state-of-the-art assessment of predictions requires an up-to-date set of secondary structure assignments and a measure to compare predictions with the correct assignments. Evaluation sets of non-homologous domains with secondary structure assignments have been proposed by Barton et al. [166, 326], This dataset is an extension of other proposed test sets [155, 167]. It contains 496 non-homologous (SD cutoff of 5, which is more stringent than mutually less than 25% sequence identity) domains. The database contains 82 847 residues, 28678 alpha-helix (H), 17 741 beta-strand (E), and 36 428 coil (C) residues. 

Russell, R. B., et al., Recognition of analogous and homologous protein folds - assessment of prediction success and associated alignment accuracy using empirical substitution matrices. Protein Eng, 1998. 11(1) p. 1-9. 

Refining Assessments of Predicted Operator Exposure to Pesticides... 

CAPRI Critical Assessment of PRedicted Interactions, http //capri.ebi.ac.uk. 

In this chapter we will follow the structure suggested by the OECD recommendations. We will review the development of measures of goodness of fit, robustness, assessment of predictability for continuous models and models for classified endpoints, and the definition of domains of applicability. The aim is not to provide a comprehensive summary of all papers in this field, but to provide key references that have influenced the author s continuing journey to becoming a better QSAR scientist. 

Once the descriptors have been selected, investigators need to select the statistical approach for developing the QSAR model. This can involve a number of techniques, such as multiple linear regression, partial least squares analysis, neural networks, and a variety of others [9]. These techniques need to be applied to both the training set (model development) and the validation set (assessment of predictability). 

Zuegge J, Schneider G, Coassolo P, Lave T. Prediction of hepatic metabolic clearance comparison and assessment of prediction models. Clin Pharmacokinet 2001 40 553-563. 

Figure 10.3 Large observed sample split into three sets training for model building, validation for model selection, and test for assessment of prediction accuracy. (See color insert.)...

Jones RDO, Jones HM, Rowland M, Gibson CR, Yates JWT, Chien JY, Ring BJ, Adkison KK, Ku MS, He H, Vuppugalla R, Marathe P, Fischer V, Dntta S, Sinha VK, Bjornsson T, Lave T, Ponlin P. 2011a. PhRMA CPCDC initiative on predictive models of hnman pharmacokinetics. 2. Comparative assessment of prediction methods of hnman volnme of distribution. J Pharm Sci 100 4074-4089. 

---