Resampling techniques

Efron B (1982) The jackknife, the bootstrap and other resampling techniques. Society for Industrial and Applied Mathematics, Philadelphia, PA... 

Comparison of the success of different classification methods requires a realistic estimation of performance measures for classification, like misclassification rates (% wrong) or predictive abilities (% correct) for new cases (Section 5.7)—together with an estimation of the spread of these measures. Because the number of objects with known class memberships is usually small, appropriate resampling techniques like repeated double CV or bootstrap (Section 4.2) have to be applied. A difficulty is that performance measures from regression (based on residuals) are often used in the development of classifiers but not misclassification rates. 

Monte Carlo simulation An iterative resampling technique frequently used in uncertainty analysis in risk assessments to estimate the distribution of a model s output parameter. 

Very often a test population of data is not available or would be prohibitively expensive to obtain. When a test population of data is not possible to obtain, internal validation must be considered. The methods of internal PM model validation include data splitting, resampling techniques (cross-validation and bootstrapping) (9,26-30), and the posterior predictive check (PPC) (31-33). Of note, the jackknife is not considered a model validation technique. The jackknife technique may only be used to correct for bias in parameter estimates, and for the computation of the uncertainty associated with parameter estimation. Cross-validation, bootstrapping, and the posterior predictive check are addressed in detail in Chapter 15. 

Pharmacometric (PM) models have many and varied applications for drug development, regulation, and applied pharmacotherapy. Resampling techniques can be applied to model development, evaluation, and validation— most often resulting in an economy of effort once applied to these aspects of modeling (1-3). Models have been defined as either descriptive or predictive (see Chapter 8). While descriptive models require checks for reliability and stability, predictive models have the added requirement of validation (which resampling can do). 

Jackknife (IKK), cross-validation, and the bootstrap are the methods referred to as resampling techniques. Though not strictly classified as a resampling technique, the posterior predictive check is also covered in this chapter, as it has several characteristics that are similar to resampling methods. 

Gobburu, J. and Lawrence, J. Application of resampling techniques to estimate exact significance levels for covariate selection during nonlinear mixed effects model building Some inferences. Pharmaceutical Research 2002 19 92-98. 

STATISTICAL RESAMPLING TECHNIQUES FOR LARGE BIOLOGICAL DATA ANALYSIS... 

---