Validation sample

The data in the training set are used to derive the calibration which we use on the spectra of unknown samples (i.e. samples of unknown composition) to predict the concentrations in those samples. In order for the calibration to be valid, the data in the training set which is used to find the calibration must meet certain requirements. Basically, the training set must contain data which, as a group, are representative, in all ways, of the unknown samples on which the analysis will be used. A statistician would express this requirement by saying, "The training set must be a statistically valid sample of the population... 

The data in the validation set are used to challenge the calibration. We treat the validation samples as if they are unknowns. We use the calibration developed with the training set to predict (or estimate) the concentrations of the components in the validation samples. We then compare these predicted concentrations to the actual concentrations as determined by an independent referee method (these are also called the expected concentrations). In this way, we can assess the expected performance of the calibration on actual unknowns. To the extent that the validation samples are a good representation of all the unknown samples we will encounter, this validation step will provide a reliable estimate of the calibration s performance on the unknowns. But if we encounter unknowns that are significantly different from the validation samples, we are likely to be surprised by the actual performance of the calibration (and such surprises are seldom pleasant). 

The best protection we have against placing an inadequate calibration into service is to challenge the calibration as agressively as we can with as many validation samples as possible. We do this to uncover any weaknesses the calibration might have and to help us understand the calibration s limitations. We pretend that the validation samples are unknowns. We use the calibration that we developed with the training set to predict the concentrations of the validation samples. We then compare these predicted concentrations to the known or expected concentrations for these samples. The error between the predicted concentrations vs. the expected values is indicative of the error we could expect when we use the calibration to analyze actual unknown samples. 

Ideally, we validate a calibration with a great number of validation samples. Validation samples are samples that were not included in the training set. They should be as representative as possible of all of the unknown samples which the calibration is expected to successfully analyze. The more validation samples we use, and the better they represent all the different kinds of unknowns we might see, the greater the liklihood that we will catch a situation or a sample where the calibration will fail. Conversely, the fewer validation samples we use, the more likely we are to encounter an unpleasant surprise when we put the calibration into service— especially if these relatively few validation samples we are "easy cases" with few anomalies. 

We often cannot afford to assemble large numbers of validations samples with concentrations as accurate as the training set concentrations. But since the validation samples are used to test the calibration rather than produce the calibration, errors in validation sample concentrations do not have the same detrimental impact as errors in the training set concentrations. Validation set... 

Generally speaking, the more validation samples the better. It is nice to have at least as many samples in the validation set as were needed in the training set. It is even better to have considerably more validation samples than calibration samples. 

Ideally, the validation concentrations should be as accurate as the training concentrations. However, validation samples with poorer concentration accuracy are still useful. In general, we would prefer that validation concentrations would not have errors greater than 5%. Samples with concentrations errors of around 10% can still be useful. Finally, validation samples with concentration errors approaching 20% are better than no validation samples at all. 

Sometimes it is just not feasible to assemble any validation samples. In such cases there are still other tests, such as cross-validation, which can help us do a certain amount of validation of a calibration. However, these tests do not provide the level of information nor the level of confidence that we should have before placing a calibration into service. More about this later. 

Next, we create a concentration matrix containing mixtures that we will hold in reserve as validation data. We will assemble 10 different validation samples into a concentration matrix called C3. Each of the samples in this validation set will have a random amount of each component determined by choosing numbers randomly from a uniform distribution of random numbers between 0 and 1. 

PRESS for validation data. One of the best ways to determine how many factors to use in a PCR calibration is to generate a calibration for every possible rank (number of factors retained) and use each calibration to predict the concentrations for a set of independently measured, independent validation samples. We calculate the predicted residual error sum-of-squares, or PRESS, for each calibration according to equation [24], and choose the calibration that provides the best results. The number of factors used in that calibration is the optimal rank for that system. 

Cross-validation. We don t always have a sufficient set of independent validation samples with which to calculate PRESS. In such instances, we can use the original training set to simulate a validation set. This approach is called cross-validation. The most commom form of cross-validation is performed as follows ... 

Fortunately, since we also have concentration values for our samples, We have another way of deciding how many factors to keep. We can create calibrations with different numbers of basis vectors and evaluate which of these calibrations provides the best predictions of the concentrations in independent unknown samples. Recall that we do this by examing the Predicted Residual Error Sum-of Squares (PRESS) for the predicted concentrations of validation samples. 

When we examine the plots in Figure 56 we see that the PRESS decreases each time we add another factor to the basis space. When all of the factors are included, the PRESS drops all the way to zero. Thus, these fits cannot provide us with any information about the dimensionality of the data. The problem is that we are trying to use the same data for both the training and validation data. We lose the ability to assess the optimum rank for the basis space because we do not have independent validation samples that contain independent noise. So, the more factors we add, the better the calibration is able to model the particular noise in these samples. When we use all of the factors, we are able to model the noise completely. Thus, when we predict the concentrations for... 

The point is that it is usually advisable to generate calibrations using both PCR and PLS. We can then evaluate each calibration validation samples and choose whichever one works best in the particular application. Fortunately, most of the software packages available today make it an easy matter to quickly generate both calibrations. 

Just as we did for PCR, we must determine the optimum number of PLS factors (rank) to use for this calibration. Since we have validation samples which were held in reserve, we can examine the Predicted Residual Error Sum of Squares (PRESS) for an independent validation set as a function of the number of PLS factors used for the prediction. Figure 54 contains plots of the PRESS values we get when we use the calibrations generated with training sets A1 and A2 to predict the concentrations in the validation set A3. We plot PRESS as a function of the rank (number of factors) used for the calibration. Using our system of nomenclature, the PRESS values obtained by using the calibrations from A1 to predict A3 are named PLSPRESS13. The PRESS values obtained by using the calibrations from A2 to predict the concentrations in A3... 

It is apparent that these results are essentially identical to the results obtained from this data using PCR. We even do extremely well with the overrange validation samples in A4. But, it would be dangerous to assume that we can routinely get away with extrapolation of this kind. Sometimes it works well, sometimes it doesn t. There is no simple rule that can tell us which situation we might be facing. It is very dependent on the particular data and... 

CU-ASSAYLdat (Fig. 4.40) A tablet production process was being validated samples were pulled from the beginning, the middle, and the end of the production run Components A and B were analyzed. The requirement is that the means do not significantly differ and that the CV remains below 6%. 

Figure 5. Predicted versus measxired plot of %DE using the FTNIR IS ensemble (C=cross validated samples and T= test set samples).

The reviewers recommended that another study of a valid sample of the preschool population be conducted ... 

The third and last phase of the trial is the analysis of the validation samples. All data collected are reported. No results are discarded unless a determinate error can be identified. Any request to repeat the assay of a sample should be approved by... 

In summary, official German analytical methods for pesticide residues are always validated in several laboratories. These inter-laboratory studies avoid the acceptance of methods which cannot readily be reproduced in further laboratories and they do improve the ruggedness of analytical procedures applied. The recently introduced calibration with standards in matrix improves the trueness of the reported recovery data. Other aspects of validation (sample processing, analyte stability, extraction efficiency) are not considered. 

The first one we mention is the question of the validity of a test set. We all know and agree (at least, we hope that we all do) that the best way to test a calibration model, whether it is a quantitative or a qualitative model, is to have some samples in reserve, that are not included among the ones on which the calibration calculations are based, and use those samples as validation samples (sometimes called test samples or prediction samples or known samples). The question is, how can we define a proper validation set Alternatively, what criteria can we use to ascertain whether a given set of samples constitutes an adequate set for testing the calibration model at hand ... 

Even so, at best any of these answers treat only one aspect of the larger question, which includes not only how many samples, but which ones A properly taken random sample is indeed representative of the population from which it comes. So one subquestion here is, how should we properly sample The answer is randomly but how many workers select their validation samples in a verifiably random manner How can someone then tell if their test set is then valid, and against what criteria ... 

Reference values, n - the component concentrations or property values for the calibration or validation samples which are measured by the reference analytical method. 

Validation samples, n - a set of samples used in validating a calibration model. Validation samples are not generally part of the set of calibration samples. Reference component concentrations or property values are known (measured using a reference method), and are compared to those estimated using the model. 

Validation test, n - a test performed on a validation sample that demonstrates that the result produced by the instrument or analytical method and the result produced by the reference method are equivalent to within statistical tests. 

In each of the aforementioned studies, qualitative IR spectroscopy was used. It is important to realize that IR is also quantitative in nature, and several quantitative IR assays for polymorphism have appeared in the literature. Sulfamethoxazole [35] exists in at least two polymorphic forms, which have been fully characterized. Distinctly different diffuse reflectance mid-IR spectra exist, permitting quantitation of one form within the other. When working with the diffuse reflectance IR technique, two critical factors must be kept in mind when developing a quantitative assay (1) the production of homogeneous calibration and validation samples, and (2) consistent particle size for all components, including subsequent samples for analysis. During the assay development for... 

One should never forget that all sampling investigations are subject to experimental error. Careful consideration of sampling can keep this error to a minimum. To assure valid sampling, major consideration should be given to the following [13] ... 

Sampling procedures are extremely important in the analysis of soils, sediments and sludges. It is essential to ensure that the composition of the portion of the sample being analysed is representative of the material being analysed. This fact is even more evident when it is conceded that the size of the portion of sample being analysed is in many modern methods of analysis extremely small. It is therefore essential to ensure before the analysis is commenced that correct statistically validated sampling procedures are used to ensure as far as is possible that the portion of the sample being analysed is representative of the bulk of material from which the sample was taken. 

---