Bootstrap double

CV or bootstrap is used to split the data set into different calibration sets and test sets. A calibration set is used as described above to create an optimized model and this is applied to the corresponding test set. All objects are principally used in training set, validation set, and test set however, an object is never simultaneously used for model creation and for test. This strategy (double CV or double bootstrap or a combination of CV and bootstrap) is applicable to a relatively small number of objects furthermore, the process can be repeated many times with different random splits resulting in a high number of test-set-predicted values (Section 4.2.5). 

Comparison of the success of different classification methods requires a realistic estimation of performance measures for classification, like misclassification rates (% wrong) or predictive abilities (% correct) for new cases (Section 5.7)—together with an estimation of the spread of these measures. Because the number of objects with known class memberships is usually small, appropriate resampling techniques like repeated double CV or bootstrap (Section 4.2) have to be applied. A difficulty is that performance measures from regression (based on residuals) are often used in the development of classifiers but not misclassification rates. 

The double bootstrap was a method originally suggested by Efron (15) as a way to improve on the bootstrap bias correction of the apparent error rate of a linear discrimination rule. It is simply a bootstrap iteration (i.e., taking resamples from each bootstrap resample). The double bootstrap has been useful in improving the accuracy of confidence intervals but it substantially increases computation time and most likely increases the incidence of unsuccessfully terminated runs. It has been applied to linear models but not to PM modeling. 

---