Model validation residual testing

The first treated is test-set and cross-validation which are general tools useful for many purposes in validation. For example, cross-validation is often used in two-way analysis for establishing the number of components in PCA or PLS models. In essence, test-set and cross-validation simply provide methods for obtaining more realistic residuals than those obtained by ordinary residuals from fitted models. Thus, test-set and cross-validation are of general interest when residuals are used for reporting explained variance, assessing outliers etc. 

Test-set validation is performed by fitting the current model to new data. How this is done is explained in Chapter 6 for the different models. The residuals are simply obtained by subtracting the model of the new data from the actual data. Test-set validation is natural because it specifically simulates the practical use of the model on new data in the future. If the test-set is made independent of the calibration data (future samples measured by future technicians or in a future location etc.) then the test-set validation is the definitive validation because the new data are, in fact, made as a drawing from the future total population of all possible measurements. Small test-sets may provide uncertain results merely because of the small sample size. 

The main goal of the data analysis is usually to find X, but the residual E can give important clues to the quality of this model. Possibly, residuals obtained from a test-set or from cross-validation can be used instead of fitted residuals. Random noise or some symmetrical type of distribution for the elements of E is normally expected and this can be verified from plotting the residuals and by the use of diagnostics. A good description of the use of residuals in three-way analysis is given by Kroonenberg [1983],... 

Model validation consists of three parts (1) testing the residuals, (2) testing the adequacy of the model, and (3) taking corrective action. Each of these three parts will now be examined in greater detail. 

The basic principles of model validation, testing the residuals and the overall model, are the same as for regression analysis. The goal of this validation is to confirm that the residuals obtained are independent, normally distributed, white noise values and that the model captures a significant portion of the overall variability. The main tools for model validation are ... 

When performing model validation, it is important to bear in mind the final goal for which the model will be used. In time series analysis, the majority of the time, such models are used to forecast or predict future values of the system. In such cases, it is very important to not only test the performance of the model using the initial data set but also use another model validation data set. This validation data set can be obtained by splitting the original data set into parts. The first part is used for model estimation, while the second part is used for model validation. The residuals obtained using the data from the second part would then be used for model validation. The data set is often split Vs for estimation and % for validation. 

Once the model parameters have been determined, it is necessary to validate the model. As before, three different components need to be considered (1) testing the residuals, (2) testing the adequacy of the model, and (3) taking corrective action. The general details of these components are the same as for regression analysis (see Sect. 3.3.5 Model Validation). However, some specific details are needed for model validation in process system identification. 

However, when performing model validation using this approach, a few changes need to be made in the analysis due to correlation between the input, and the disturbance, e,. This correlation implies that the input will be correlated with past values of the disturbance (and hence the residuals). Therefore, the conditions for the cross-correlation test, mentioned previously for the open-loop case, need to be changed to state 95% of all cross-correlations should lie inside the 95% confidence interval for zero for lags greater than zero." ... 

The results of modeling the toxicity data indicates that the residual plots for the training, validation and test sets are not scattered and they do not warranty the stability of the models. There is a strong relationship between the residual and actual values which reflects that the obtained models have systematic error, therefore a correction scheme is done to correct this issue. The cross-validation parameters for the chosen models before and after correction are shown in Table 2. This table shows that the correction term improves the cross-validation parameters by lowering the RMSE and increasing the R CV values. Considering the number of variables entered to the regression model for the SR-PC-ANN... 

The qualitative validity of this model can be tested against the behavior of mutant RCs with the ionizable residues, Glu and Asp H substituted by neutral Gin and Asn. 

With cross-validation [28], the same objects are used both for model estimation and testing. A few objects are left out from the calibration data set and the model is calibrated on the remaining objects. Then the values for the left-out objects are predicted and the prediction residuals are computed. The process is repeated with another subset of the calibration set, and so on until every object has been left out once then all prediction residuals are combined to compute the validation residual variance and root mean square error in prediction (RMSEP). It is of utmost importance that the user is aware of which level of cross-validation one wants to validate. For example, if one physical sample is measured three times, and the objective is to establish a model across samples, the three replicates must be held out in the same cross-validation segment. If the objective is to validate the repeated measurement, keep out one replicate for all samples and generate three cross-validation segments. The calibration variance is always the same it is the validation... 

The validity of the Forster theory was tested and confirmed in a number of model studies with compounds that contained a donor and an acceptor separated by well-defined rigid spacers. This work has been reviewed 5] In a classical study, a naphthyl group (donor) was attached to the C-terminal and a dansyl group (acceptor) to the N-terminal of poly-L-proline oligomers (1-12 proline residues) 61 These proline oligomers assume a trans helical conformation in ethanol and thus represent spacers of well-defined length (12-46 A). A continuous decrease in the transfer efficiency from 100% at a donor-acceptor separation of... 

PLS is best described in matrix notation where the matrix X represents the calibration matrix (the training set, here physicochemical parameters) and Y represents the test matrix (the validation set, here the coordinates of the odor stimulus space). If there are n stimuli, p physicochemical parameters, and m dimensions of the stimulus space, the equations in Figure 6a apply. The C matrix is an m x p coefficient matrix to be determined and the residuals not explained by the model are contained in E. The X matrix is decomposed as shown in Figure 6b into two small matrices, an n x a matrix T and an a x p matrix B where a << n and a << p. F is the error matrix. The computation of T is such that it both models X and correlates with T and is accomplished with a weight matrix W and a set of latent variables U for Y with a corresponding loading matrix B. ... 

A simple and classical method is Wold s criterion [39], which resembles the well-known F-test, defined as the ratio between two successive values of PRESS (obtained by cross-validation). The optimum dimensionality is set as the number of factors for which the ratio does not exceeds unity (at that moment the residual error for a model containing A components becomes larger than that for a model with only A - 1 components). The adjusted Wold s criterion limits the upper ratio to 0.90 or 0.95 [35]. Figure 4.17 depicts how this criterion behaves when applied to the calibration data set of the working example developed to determine Sb in natural waters. This plot shows that the third pair (formed by the third and fourth factors) yields a PRESS ratio that is slightly lower than one, so probably the best number of factors to be included in the model would be three or four. 

The QSARs obtained were then tested by cross validation and visual examination of plots of fitted values against residuals. Cross-validation was performed by leave-one-out (LOO) testing. Each data point was omitted in turn from a regression, and the actual value of the omitted point compared to the value predicted by the revised model. The difference was referred to as a deletion residual. Q2 values (an analog of the summary statistic R2) were then calculated from the sum of squares of the deletion residuals. The Q2 statistic provides a measure of the predictive power of a regression, and is therefore more relevant for QSAR modeling than the R2 statistic (Damborsky and Schultz, 1997). 

---