Judging model quality

The possibility to include judgments of quality is a great advantage with the principles of BES. This is possible since a BES is designed as a framework to combine multiple source of evidence for meta-analysis. Also, new kind of sensitivity analysis can be made. Spiegelhalter and Best (2003) consider the influence of expert judged qualities in three data sources through the observation process models. 

The predictive quality of the models is judged according to the cross-validated R2, known as q2, obtained using the leave-one-out (LOO) approach, which is calculated as follows ... 

The quality of the mathematical model can only be judged by its ability to predict the likely experimental results over a wide range of conditions. The goodness of the agreement between the predictions of our proposed model and experimental observations is very much dependent on the key parameters in the model being clearly defined and well understood as mentioned previously. One of the difficulties we have encountered in attempting to compare the model behaviour and experimental GPC traces has been in obtaining reliable estimates of Pe and D. ... 

Checking the data quality is strongly recommended inspection of the Wilson plot and data reduction statistics is very useful in judging the extent of the resolution to which the data can realistically be used. Pathologically bad data, for example those from a split crystal, twinned data, systematically incomplete data, low resolution overloads will always make model building and refinement hard if not impossible. 

Judging convergence and assessing model quality are overlapping tasks. I will discuss criteria of convergence here. In Chapter 8,1 will discuss some of the criteria further, particularly as they relate to the quality and usefulness of the final model. 

As a user of macromolecular models, you are faced with judging whether each model really supports the insights it appears to offer. The principles presented in Chapter 7, on how to judge the quality of models, apply to models obtained from all types of diffraction experiments. But today s structural databases also contain a growing number of models obtained by methods other than diffraction. In the next chapter, I will describe the origin of the major types of non-diffraction models and provide some guidance on how to use them wisely. 

In this section, I provide a simplified physical picture of pulse NMW spectroscopy, including a simple conceptual model to help you understand multidimensional NMR. Then I briefly discuss the problems of assigning resonances and determining distance restraints for molecules as large and complex as proteins, and the methods for deriving a structure from this information. Finally, I discuss the contents of coordinate files from NMR structure determination and provide some hints on judging the quality of models. 

The perceptual model as developed in this chapter is used to map the input and output of the audio device onto internal representations that are as close as possible to the internal representations used by the subject to judge the quality of the audio device. It is shown that the difference in internal representation can form the basis of a perceptual audio quality measure (PAQM) that has a high correlation with the subjectively perceived audio quality. Furthermore it is shown that with a simple cognitive module that interprets the difference in internal representation the correlation between objective and subjective results is always above 0.9 for both wideband music and telephone-band speech signals. For the measurement of the quality of telephone-band speech codecs a simplified version of the PAQM, the perceptual speech quality measure (PSQM), is presented. 

Figure 1.9 Relation between the mean opinion score (MOS) and the perceptual audio quality measure (PAQM) for the 50 items of the ISO/MPEG 1991 codec test [IS091, 1991] in loudspeaker presentation. The filled circles are items whose quality was judged significantly lower by the model than by the subjects.

A mathematical model of a plant or a section of a plant can be judged only by comparison with actual plant data. The model may be considered as good when the simulated variables can predict with some level of confidence the plant parameters which are important in determining the cost and quality of the finished product. Failures of the model are likely to be a result of (1) oversimplification of the equations that constitute the model, (2) inadequacy of the numerical solution of the equations. 

---