Test-set Validation

But to pose the question more directly how can we tell if any set of samples constitute a valid test set Even if they were chosen in a proper random manner, are there any independent tests for their validity What characteristics should the criteria for deciding be based on, and what are the criteria to use ... 

The model was claimed to compute 5000-6000 molecules per min. The predictive ability of the model was validated by four approaches. In the first approach, a set of 20 compounds was randomly selected as an initial validation test set. A model was developed from the remaining 86 compounds with an MAE of 0.33, from which the test set values were then predicted. The results of this test prediction were very good and provided momentum for support of the three structure descriptors. In the second approach, a full cross-validation test of the model was investigated. The data set of 102 compounds was divided... 

For assessment of the potential to predict granule moisture content, a large 1032-object data set recorded dnring 5 months of urea production was used. The first 900 objects were used for calibration and the last 132 as a validation test set [2]. The data matrix was resampled to allow acoustic data to be calibrated against laboratory tests of moisture content, which were only available with a relatively low sampling rate however, plenty of results were at hand to allow a full assessment of the multivariate prediction performance for granule moisture. The validated prediction results can be seen in Figure 9.11. 

Figure 9.23 Prediction resuits for ammonia, validated with two-segment cross validation (test set switch). Slope = 0.96. RMSEP = 0.48% ammonia.

The full-scale industrial experiment demonstrated the feasibility of a convenient, nonintrusive aconstic chemometric facility for reliable ammonia concentration prediction. The training experimental design spanned the industrial concentration range of interest (0-8%). Two-segment cross-validation (test set switch) showed good accnracy (slope 0.96) combined with a satisfactory RMSEP. It is fully possible to further develop this pilot study calibration basis nntil a fnll industrial model has been achieved. There wonld appear to be several types of analogous chemical analytes in other process technological contexts, which may be similarly approached by acoustic chemometrics. 

The validation (test) set should be independent of the training set, if possible. 

For data classification, the spectra were partitioned into training and validation (test) sets. The four differently preprocessed sets of H MR brain spectra were subjected to two classification methods LDA and a noise-augmented artificial neural net (NN). All classifier training was cross-validated via the LOO method. The two classifiers (LDA and NN) were used on three-class (E, M and A) data. CCD was then implemented based on stacked generalization.61... 

In the neural net example above, it was clear that a method that is created and performs well on a training set of data may not perform well when another data sample is encountered. Hence a validation (test set) of data is needed to estimate how the classifier would perform in real life. 

Figure 9 Comparison of NIR-predicted serum analyte levels to reference analytical results (see also NIR B in Table 4). Open circles correspond to the calibration (training) set, solid circles to the validation (test) set and the solid line is the line of identity. (Adapted from K.H. Hazen, M.A. Arnold, G.W. Small, Measurement of Glncose and Other Analytes in Undiluted Human Serum with Near-infrared Transmission Spectroscopy , Analytica Chimica Acta, 255-267, Vol. 371, 1998, with permission from Elsevier Science.)...

Initial Data Chemical space IJ representation, training set, validation/test set... 

The CamuS system is currently in the form of a laboratory prototype and is undergoing a series of validation tests using an extensive set of test-pieces covering a range of geometries and classes of defect which has been manufactured for the purpose. 

Another method of detection of overfitting/overtraining is cross-validation. Here, test sets are compiled at run-time, i.e., some predefined number, n, of the compounds is removed, the rest are used to build a model, and the objects that have been removed serve as a test set. Usually, the procedure is repeated several times. The number of iterations, m, is also predefined. The most popular values set for n and m are, respectively, 1 and N, where N is the number of the objects in the primary dataset. This is called one-leave-out cross-validation. 

Oui recommendation is that one should use n-leave-out cross-validation, rather than one-leave-out. Nevertheless, there is a possibility that test sets derived thus would be incompatible with the training sets with respect to information content, i.e., the test sets could well be outside the modeling space [8]. 

The validation of the prediction equation is its performance in predicting properties of molecules that were not included in the parameterization set. Equations that do well on the parameterization set may perform poorly for other molecules for several different reasons. One mistake is using a limited selection of molecules in the parameterization set. For example, an equation parameterized with organic molecules may perform very poorly when predicting the properties of inorganic molecules. Another mistake is having nearly as many fitted parameters as molecules in the test set, thus fitting to anomalies in the data rather than physical trends. 

Apart from inspection and test records (clause 4.10.5), the standard does not require records to be authenticated, certified, or validated. A set of results without being endorsed with the signature of the person who captured them lacks credibility. Facts that have been obtained by whatever means should be certified for four reasons ... 

Divide the available data into training and test data sets (1/3). Test sets are used to validate the trained network and insure accurate generalization. 

We have said that every time the calibration analyzes a new unknown sample, this amounts to an additional validation test of the calibration. It can be a major mistake to believe that, just because a calibration worked well when it was being developed, it will continue to produce reliable results from that point on. When we discussed the requirements for a training set, we said that collection of samples in the training set must, as a group, be representative in all ways of the unknowns that will be analyzed by the calibration. If this condition is not met, then the calibration is invalid and cannot be expected to produce reliable results. Any change in the process, the instrument, or the measurement procedure which introduces changes into the data measured on an unknown will violate this condition and invalidate the method If this occurs, the concentration values that the calibration predicts for unknown samples are completely unreliable We must therefore have a plan and procedures in place that will insure that we are alerted if such a condition should arise. 

The validity of set I can be tested by ascertaining whether Eq. (39) is itself valid similar procedures are followed for the other sets of properties. 

When applied to QSAR studies, the activity of molecule u is calculated simply as the average activity of the K nearest neighbors of molecule u. An optimal K value is selected by the optimization through the classification of a test set of samples or by the leave-one-out cross-validation. Many variations of the kNN method have been proposed in the past, and new and fast algorithms have continued to appear in recent years. The automated variable selection kNN QSAR technique optimizes the selection of descriptors to obtain the best models [20]. 

The FDA [51] has used the MDL QSAR software [19] to develop QSARs for the carcinogenic potential of pharmaceuticals and organic chemicals. These were validated using a test set of 108 compounds, with 72% correct prediction of carcinogens and 72% correct prediction of noncarcinogens. 

Figure 5. Predicted versus measxired plot of %DE using the FTNIR IS ensemble (C=cross validated samples and T= test set samples).

Validation of the classification rule, using an independent test set. This is described in more detail in Section 33.4. 

When not enough examples are available to make an independent monitoring set, the cross-validation procedure can be applied (see Chapter 10). The data set is split into C different parts and each part is used once as monitoring set. The network is trained and tested C times. The results of the C test sessions give an indication on the performance of the network. It is strongly advised to validate the network that has been trained by the above procedure with a second independent test set (see Section 44.5.10). 

---