Test sets independent

Instead of validating the predictions internally, it is possible to test the predictions against an independent data set, often called a test set. Computationally the procedure is similar to cross-validation. For example, a model is obtained using I samples, and then the predictions are calculated using an independent test set of L samples, to give 

The value of q is determined in exactly the same way as per cross-validation (see Section 5.6.2), but only one calibration model is formed, from the training set. 

Root mean square error using data in Table 5.1 as a training set and data in Table 5.20 as a test set, PLS1 (centred) and acenaphthylene 

The optimum size and representativeness of training and test sets for calibration modelling are a big subject. Some chemometricians recommend using hundreds or thousands of samples, but this can be expensive and time consuming. In some cases a completely orthogonal dataset is unlikely ever to occur and field samples with diese features cannot be found. Hence there is no perfect way of validating calibration 

---

Validation of the classification rule, using an independent test set. This is described in more detail in Section 33.4. 

When not enough examples are available to make an independent monitoring set, the cross-validation procedure can be applied (see Chapter 10). The data set is split into C different parts and each part is used once as monitoring set. The network is trained and tested C times. The results of the C test sessions give an indication on the performance of the network. It is strongly advised to validate the network that has been trained by the above procedure with a second independent test set (see Section 44.5.10). 

The basis of all performance criteria are prediction errors (residuals), yt - yh obtained from an independent test set, or by CV or bootstrap, or sometimes by less reliable methods. It is crucial to document from which data set and by which strategy the prediction errors have been obtained furthermore, a large number of prediction errors is desirable. Various measures can be derived from the residuals to characterize the prediction performance of a single model or a model type. If enough values are available, visualization of the error distribution gives a comprehensive picture. In many cases, the distribution is similar to a normal distribution and has a mean of approximately zero. Such distribution can well be described by a single parameter that measures the spread. Other distributions of the errors, for instance a bimodal distribution or a skewed distribution, may occur and can for instance be characterized by a tolerance interval. 

MSE Mean of squared errors, MSECV for cross validation, MSEtcst for an independent test set. 

In the case of an independent test set the file should contain 20-40% of the full data. The calibration and test set must cover the same population of samples as... 

Savolainen et al. investigated the role of Raman spectroscopy for monitoring amorphous content and compared the performance with that of NIR spectroscopy [41], Partial least squares (PLS) models in combination with several data pre-processing methods were employed. The prediction error for an independent test set was in the range of 2-3% for both NIR and Raman spectroscopy for amorphous and crystalline a-lactose monohydrate. The authors concluded that both techniques are useful for quantifying amorphous content however, the performance depends on process unit operation. Rantanen et al. performed a similar study of anhydrate/hydrate powder mixtures of nitrofurantoin, theophyllin, caffeine and carbamazepine [42], They found that both NIR and Raman performed well and that multivariate evaluation not always improves the evaluation in the case of Raman data. Santesson et al. demonstrated in situ Raman monitoring of crystallisation in acoustically levitated nanolitre drops [43]. Indomethazine and benzamide were used as model... 

The method error for Equation (6) is indicated by the correlation s standard error of 3.3 K. Cramer (1980b) also evaluated the equation with an independent test set of 138 compounds with the following results for the root mean square (rms) errors ... 

For the 4,426-compound data set used to develop this method, the predicted boiling points had an average absolute error of 15.5 K (3.2%). Stein and Brown (1994) also evaluated their method on an independent test set of 6,584 compounds and found the predicted boiling points had an average absolute error of 20.4 K (4.3%). 

We should, however, remark that in reality the data analyst will look for more samples, will probably try several cross-validation procedures, will test the classification functions using independent test sets, and so on. 

With the considerable choice of methods to calculate log Kow it is difficult even for an experienced researcher in this area to determine the best one to use. A number of comparative studies on the performance of calculation methods have been performed (Mannhold et al., 1990 Buchwald and Bodor, 1998). It is often difficult to use the results of comparative studies, however, as it is difficult to find suitable data to establish a truly independent test set (i.e., data for compounds that have not been included in the original training set). The choice of method often becomes a subjective decision based on criteria such as ease of entry of structure and handling of the predicted values, cost, and any personal conviction or opinion on the method. From the authors experience, methods including (but not limited to) ClogP for Windows, KOWWIN, and ACD/log P all have been shown to provide robust predictions for the most commonly encountered toxicants. 

In this step, it is necessary to identify or compile an independent test set of data. In other words, it is necessary to obtain a data set containing values of the descriptors and response variables for chemical structures that were not used in the training set. The selection of chemicals for the test set should take into account the predefined domain of applicability for the QSAR. 

The QSAR should make predictions of an independent test set with an r2 value > 0.8. 

The best way to detect the true predictive power of a QSAR model is external validation, using an independent test set. It is furthermore good to demonstrate the value of a model to predict compounds being made in the future and to look at a test set (called a temporal test set) for several months before implementing a new model for routine use. 

FIGURE 6.26 The Kohonen map of a network trained with 64 absorbance valnes from the initial range of atomization signals of different compounds shows a clear separation into four areas of atomization processes. The only conflict occurs with thermal dissociation of metal carbides and metal dimers. The map indicates the central neurons for metals from an independent test set. 

Finally, all these pruned subtrees will be subject to CV, in order to select the optimal tree size. The optimal tree (Fig. 13.11b) is selected as the simplest among those that have a CV error within one standard error deviation of the minimal CV error (26, 28). Another approach to determine the optimal tree size, preferred when a large number of training samples is available, is the use of an independent test set (26). After obtaining the final model, new samples can be classified by using the rules (split criteria) given by the model. 

By using the cross-validation statistical procedure and Kyte-Doolittle hydropathy scale, the prediction results for TMH in the training data base of 63 membrane proteins common to us and to Rost et al. [9] and also to Jones et al. [33] were similar in accuracy by all three methods. When training data base is enlarged to 168 proteins, we maintain the 95% accuracy for predicted transmembrane helices and almost 80% (78.6%) of proteins are predicted with 100% correct transmembrane topology. When 168 proteins are divided in the above mentioned training set of 63 proteins and an independent test set of 105 proteins, all performance parameters for TMH prediction associated with a set of 105 proteins exhibited a decrease which was smaller in our case than for Rost et al. [9]. 

Validation without an independent test set. Each application of the adaptive wavelet algorithm has been applied to a training set and validated using an independent test set. If there are too few observations to allow for an independent testing and training data set, then cross validation could be used to assess the prediction performance of the statistical method. Should this be the situation, it is necessary to mention that it would be an extremely computational exercise to implement a full cross-validation routine for the AWA. That is. it would be too time consuming to leave out one observation, build the AWA model, predict the deleted observation, and then repeat this leave-one-out procedure separately. In the absence of an independent test set, a more realistic approach would be to perform cross-validation using the wavelet produced at termination of the AWA, but it is important to mention that this would not be a full validation. 

Genetic Algorithm applied to the discussed data set leads to different subsets of selected variables. There are many different versions of GA, depending on the way reproduction, cross-over, etc., are performed. The algorithm used in our study, adapted from Leardi et al. [34,35], is particularly directed towards feature selection. In each GA run, a few subsets with similar responses are selected. Final solutions are then evaluated based on the RMSEP of an independent test set. Results are presented in Table 2 and in Figs. 5 and 6. 

Only for yl the predictive ability of the GA-MLR model is not satisfactory. For y2, y3, y4 and y5, the prediction of the models are excellent, but as one can notice, few of the selected variables are on the data baseline, which suggests that the models can prove unstable. In fact, this is a case with these models. If to the independent test set a small noise is added (simulated as randn(mt,nt) 0.001), the constructed models failed completely. Plots of ypredkted versus yobserved for the noisy test set, denoted as Xtn, are presented in Fig. 7. 

A discriminant model which is assessed using the same training data which was used to estimate the parameters in the model will usually reflect overly optimistic results. It can be appropriate to use an independent test set for assessing the validity of the model. Let X define the testing data which contains n objects with nj objects from class r such that n = n and y = (yj, , y n ) denotes the vector of true class labels of the testing data. 

We will test the performance of the adaptive wavelet algorithm for regression purposes using an independent test set. For this reason we have decided to formulate an R- measure for the test set which is denoted by R, ... 

---