Model validation data splitting

When performing model validation, it is important to bear in mind the final goal for which the model will be used. In time series analysis, the majority of the time, such models are used to forecast or predict future values of the system. In such cases, it is very important to not only test the performance of the model using the initial data set but also use another model validation data set. This validation data set can be obtained by splitting the original data set into parts. The first part is used for model estimation, while the second part is used for model validation. The residuals obtained using the data from the second part would then be used for model validation. The data set is often split Vs for estimation and % for validation. 

The selected subset cross-validation method is probably the closest internal validation method to external validation in that a single validation procedure is executed using a single split of subset calibration and validation data. Properly implemented, it can provide the least optimistic assessment of a model s prediction error. Its disadvantages are that it can be rather difficult and cumbersome to set it up so that it is properly implemented, and it is difficult to use effectively for a small number of calibration samples. It requires very careful selection of the validation samples such that not only are they sufficiently representative of the samples to be applied to the model during implementation, but also the remaining samples used for subset calibration are sufficiently representative as well. This is the case because there is only one chance given to test a model that is built from the data. 

In order to validate the final model, the data set can be split randomly into two parts. The model is developed with one part, the index data set. With this model and the demographic data of the second part, the validation data set, observations for the validation data set can be predicted. The difference of predicted data and observations is a measure of the accuracy of the model. An alternative is the bootstrap method (Efron 1981). 

Very often a test population of data is not available or would be prohibitively expensive to obtain. When a test population of data is not possible to obtain, internal validation must be considered. The methods of internal PM model validation include data splitting, resampling techniques (cross-validation and bootstrapping) (9,26-30), and the posterior predictive check (PPC) (31-33). Of note, the jackknife is not considered a model validation technique. The jackknife technique may only be used to correct for bias in parameter estimates, and for the computation of the uncertainty associated with parameter estimation. Cross-validation, bootstrapping, and the posterior predictive check are addressed in detail in Chapter 15. 

The resampling approaches of cross-validation (CV) and bootstrapping do not have the drawback of data splitting in that all available data are used for model development so that the model provides an adequate description of the information contained in the gathered data. Cross-validation and bootstrapping are addressed in Chapter 15. One problem with CV deserves attention. Repeated CV has been demonstrated to be inconsistent if one validates a model by CV and then randomly shuffles the data, after shuffling, the model may not be validated. 

Data splitting is fairly straightforward and covered in detail in the next section on validation. It simply implies that data to be modeled are partitioned based on differences in sampling (i.e., windows where suspect 0 are believed to be constant). The most common data splits to explore pharmacokinetic time dependencies would be single-dose, chronic non-steady-state, and steady-state conditions. Data subsets are modeled individually with all parameters and variability estimates along with any relevant covariate expressions compared in a manner similar to a validation procedure (see next section). Data can be combined in a leave-one-out strategy (see cross-validation description) to examine the uniformity of data windows. ... 

Cross-validation is a leave-one-out or leave-some-out validation technique in which part of the data set is reserved for validation. Essentially, it is a data-splitting technique. The distinction lies within the manner of the split and the number of data sets evaluated. In the strict sense a -fold cross-validation involves the division of available data into k subsets of approximately equal size. Models are built k times, each time leaving out one of the subsets from the build. The k models are evaluated and compared as described previously, and a hnal model is dehned based on the complete data set. Again, this technique as well as all validation strategies offers flexibility in its application. Mandema et al. successfully utilized a cross-validation strategy for a population pharmacokinetic analysis with oxycodone in which a portion of the data was reserved for an evaluation of predictive performance. Although not strictly a cross-validation, it does illustrate the spirit of the approach. 

Although the complete data set consisted of 417 measured IR spectra, it covered only 18 different rockets, i.e. it contained 399 replicates. These replicates were not used in the validation of the models. Instead, leave-one-out cross-validation (Hjorth, 1994) was used to assess the quality of the models, i.e. the set of n (= 18) independent samples was split into n-1 training samples, while the n h point was reserved for model validation. The training-validation split was repeated n times until each data point had been omitted once for validation. A validation set of n predictions on the unseen data was therefore derived from all the available data and a predicted residual estimate sum of squares (PRESS) was calculated on the validation set. 

The first system identification-specific detail is that the goal of most such models is to predict future values. Therefore, the model validation tests are often performed on a separate set of data that was not used for model parameter estimation. This is one major difference from standard regression analysis where the same data set is used for both cases. This means that the data set is split into two parts one is used for model parameter estimation and one is used for model validation. In general, the model creation part will consist of A of the data, while the model validation part will consist of % of the data. 

Finally, the prediction error model assumes that the parameter values do not change with respect to time, that is, they are time invariant. A quick and simple test of the invariance of the model is to split the data into two parts and cross validate the models using the other data set. If both models perform successfully, then the parameters are probably time invariant, at least over the time interval considered. 

There is another way to generate the prediction errors without actually having to split the data set. The idea is to set aside each data point, estimate a model using the rest of the data, and then evaluate the prediction error at the point that was removed. This concept is well known as the PRESS statistic in the statistical community (Myers, 1990) and is used as a technique for model validation of general regression models. However, to our knowledge, the system identification literature has not suggested the use of the PRESS for model structure selection. 

The statistics of prediction of an independent external test set provide the best estimate of the performance of a model. However, the splitting of the data set in training set (used to develop the model) and the test set (used to estimate how well the model predicts unseen data) is not a suitable solution for small-sized data sets and an extensive use of internal validation procedures is recommend. 

Leaving out one object at a time represents only a small perturbation to the data when the number (n) of observations is not too low. The popular LOO procedure has a tendency to lead to overfitting, giving models that have too many factors and a RMSPE that is optimistically biased. Another approach is k-fold cross-validation where one applies k calibration steps (5 < k < 15), each time setting a different subset of (approximately) n/k samples aside. For example, with a total of 58 samples one may form 8 subsets (2 subsets of 8 samples and 6 of 7), each subset tested with a model derived from the remaining 49 or 50 samples. In principle, one may repeat this / -fold cross-validation a number of times using a different splitting [20]. 

If data from many objects are available, a split into three sets is best into a Training set (ca. 50% of the objects) for creating models, a Validation set (ca. 25% of the objects) for optimizing the model to obtain good prediction performance, and a Test set (prediction set, approximately 25%) for testing the final model to obtain a realistic estimation of the prediction performance for new cases. The three sets are treated separately. Applications in chemistry rarely allow this strategy because of a too small number of objects available. 

CV or bootstrap is used to split the data set into different calibration sets and test sets. A calibration set is used as described above to create an optimized model and this is applied to the corresponding test set. All objects are principally used in training set, validation set, and test set however, an object is never simultaneously used for model creation and for test. This strategy (double CV or double bootstrap or a combination of CV and bootstrap) is applicable to a relatively small number of objects furthermore, the process can be repeated many times with different random splits resulting in a high number of test-set-predicted values (Section 4.2.5). 

In double CV, the CV strategy is applied in an outer loop (outer CV) to split all data into test sets and calibration sets, and in an inner loop (inner CV) to split the calibration set into training sets and validation sets (Figure 4.6). The inner loop is used to optimize the complexity of the model (for instance, the optimum number of PLS components as shown in Figure 4.5) the outer loop gives predicted values yjEST for all n objects, and from these data a reasonable estimation of the prediction performance for new cases can be derived (for instance, the SEPtest). It is important... 

It should be mentioned that another validation technique, called leverage correction [1], is available in some software packages. This method, unlike cross validation, does not involve splitting of the calibration data into model and test sets, but is simply an altered calculation of the RMSEE fit error of a model. This alteration involves the weighting of the contribution of the root mean square error from each calibration... 

After examining tiie training set classes with PCA, the next step is to construct and validate the SIMCA models. Ideally, tlie SIMCA models are constructed using the training set and validated using a completely separate test set. Tliis separate test set is usually not available and therefore it is necessary to use a cross-validation scheme. There are many procedures for splitting the data set and a few are discussed below. Keep in mind that the training set must be carefully... 

---