Quality of Prediction

A key issue in calibration is to determine how well the data have been modelled. We have used only one indicator above, but it is important to appreciate that there are many other potential statistics. 

Most look at how well the concentration is predicted, or the c (or according to some authors y) block of data. 

The simplest method is to determine the sum of square of residuals between the true and predicted concentrations  

Often the error is reported as a root mean square error  

If the data are centred, a further degree of freedom is lost, so the sum of square residuals is divided by I — a — 1. 

---

Many excellent computer programs are available for predicting log P a from two-dimensional structures. The quality of predictions has risen over the years to the point that rouhne log P a measurements are not regularly done at some pharmaceutical companies, but rather, calculated values are used. It is worth noting that log Port values of newly synthesized classes of drug-like compounds sometimes are still poorly predicted and probably there will be the need for judicious log Port measurements for years to come. 

Reasonably good testing methods exist for acute toxicity. The tests use surrogate animals, and the correlation to humans is the weakest element. The quality of predictive modehng for acute effects based on SAR (Structure Activity Relationships), is only modest. For chronic effects, testing with surrogates for humans is modestly good, particularly for cancer. Tests for chronic toxicity in animals are only fair and for... 

While the n = 160 database provided the best statistical model, visually there was little difference between n = 199 and n = 160 dock databases. This is not of major significance as both models contain a large number of molecular examples and, therefore, might be expected to offer similar visual clues after deletion of a relatively small percentage of the data. However, the quality of predictions of these visually comparable models is not the same. 

To build a QSPR model, one should carefully select available experimental data, and choose the initial pool descriptors (from which the program selects the most appropriate ones) as well as a mathematical approach linking those descriptors with a given property. Then, a suitable strategy of model validation should be applied in order to obtain a quantitative assessment of the quality of predictions. Finally, some rules should be established in order to prevent the application of the models to compounds too different from those used for obtaining the models. 

The quality of prediction can be determined by the residuals (or errors) i.e. the difference between the observed and predicted, i.e. x — x the less this is the better. Generally the root mean error is calculated,... 

Using a test set to determine the quality of predictions is a form of validation. The test set could be obtained, experimentally, in a variety of ways, for example 60 orange juices might be analysed in the first place, and then randomly divided into 30 for the training set and 30 for the test set. Alternatively, the test set could have been produced in an independent laboratory. 

Relative to the standard deviation of the centred data it is even higher. Hence the .v and V blocks are modelled in different ways and it is important to recognise that the percentage error of prediction in concentration may diverge considerably from the percentage error of prediction of the spectra. It is sometimes possible to reconstruct spectral blocks fairly well but still not predict concentrations very effectively. It is best practice to look at errors in both blocks simultaneously to gain an understanding of the quality of predictions. 

The errors using 10 PLS components are summarised in Table 5.15, and are better than PCR in this case. It is important, however, not to get too excited about the improved quality of predictions. The c or concentration variables may in themselves contain errors, and what has been shown is that PLS forces die solution to model the apparent c block better, but it does not necessarily imply that the other mediods are worse at discovering die truth. If, however, we have a lot of confidence in the experimental procedure for determining c (e.g. weighing, dilution, etc.), PLS will result in a more faithful reconstruction. 

The third chapter maintains the theme of determining conformations by describing how to generate initial structures of organic and bioorganic molecules and how to model experimental NMR data. Andrew Torda and Wilfred van Gunsteren also discuss refinement methods, force fields, systematic errors and biases, and the quality of predicted structures. 

The small value of RMS %6P showirfor 363.15 K (90°C) iirdicates both the suitability of the van Laar equatioir for correlation of the VLE data and the capability of the equation of state to reproduce the data. A direct fit of these data with the van Laar equation by the ganmra/phi procedure yields RMS % 6 P = 0.19. The results at 423.15 to473.15K(150 and 200°C) are based oirly on vapor-pressuredata for the pure species and on mixture data at lower temperatures. Tlre quality of prediction is indicated by the P-x-y diagram of Fig. 14.10, wlrich reflects the uncertainty of the data as well. 

Results for the higher temperatures indicate the quality of predictions based only on vapor-pressure data for the pure species and on mixture data at 323.15 K. Extrapolations based on the same data to still higher temperatures can be expected to become progressively less accurate. 

Figures 1 and 2 show the results of the parametric studies for methane and propane, respectively. From the results of this study we observe that (i) there is a strong dependence of the predicted hydrate equilibrium pressure on the energy parameter, s/k, and the distance parameter, cr, (ii) there is a less significant dependence on the reference chemical potential difference, Ajn, (iii) there is a minor dependence on the core radius, a, and the reference enthalpy difference, Ah, and (iv) the quality of predictions is not satisfactory for higher...

It is evident that the quality of a design matrix for RSM is a question of the quality of prediction of the response within the domain. It is described by the variance function. We have already seen that ... 

The stringency and quality of predicted transcriptional networks in macrophages (or any other system) can be increased by acquiring and integrating additional pieces of information ... 

An alternative approach to using traditional parametric statistical methods to calculate the significance of fitted correlations, would be to directly assess the model based on its ability to predict, rather than merely to assess how well the model fits the training set. When the quality of the model is assessed by the prediction of a test set, rather than the fit of the model to its training set, a statistic related to r or can be defined, and denoted q or q, to indicate that the quality measure is assessed in prediction. A q may be calculated by internal cross-validation techniques, or by the quality of predictions of an independent test set, in which case an upper-case is used. The equation to calculate q (or Q ) is shown in equation 9.3. 

---