The Role of Independent Validation

Given the powerful analysis tools which are readily available today, there is the danger of over- fitting the data and/or over-interpreting the results. While much of the early work in biomedical vibrational spectroscopy was purely explorative, it is important to note that nowadays, i.e. as the field is approaching application, any claim on suitability for diagnostic applications needs to be accompanied by an independent validation. 

Frequently the efforts are hampered by the lack of a sufficiently large number of samples. In our experience, it appears to be helpful if the number of samples exceeds the number of parameters (wavenumber intervals, principal components, latent variables, etc.) by at least a factor of five. This finding is supported by calculating this ratio between the number of teaching samples and the number of parameters both in the field of diagnostic pattern recognition (see the column ratio in Table 6.2) as well as in the quantitative analysis of serum. 

Study Method teach N para Ratio Nval SE SP Ref.  

Another indication for overfitting can be seen in Figvue 6.7 at very high noise levels, i.e. at very low SNR, the algorithm tends to fit the noise, which gives superb results if evaluated with the same data (open circles) but which 

Although we have, so far, not been able to mathematically justify this rule of thumb of the minimum ratio of Ateach/Apara 5, it has been our experience, and that of others, that consistent and dependable data can hardly be achieved at significantly lower ratios. 

---

This chapter is intended to assist the procedure of biomedical vibrational spectroscopy in the context of synchrotron IR spectroscopy. It starts with the recording of spectra in standardised settings and continues through the assessment of spectral (and system) quality, spectral pre-processing up to data analysis (both quantitative and classification analysis). For the latter part, the main emphasis is placed on the role of independent validation. 

The technique of mid-IR microspectroscopy and imaging has great potential for the rapid and reliable identification of tissue structures not only for scientific research purposes but also in a real clinical set-up. The standardisation of the data acquisition and the assessment of the quality of the spectra constitute key factors for the successful transfer to elinical application. Furthermore the problem of overfitting and the role of independent validation have been discussed. In this chapter we also exemplified the question of standardisation of the hyperspectral image acquisition protocol and demonstrated how the... 

---