Resampling Methods

Molinaro AM, Simon R, Pfeiffer RM. Prediction error estimation A comparison of resampling methods. Bioinformatics 2005 21 3301-3307. 

The R-RMSECV values are rather time consuming because, for every choice of k, they require the whole RPCR procedure to be performed n times. Faster algorithms for cross validation are described [80], They avoid the complete recomputation of resampling methods, such as the MCD, when one observation is removed from the data set. Alternatively, one could also compute a robust R2-value [61], For q= 1 it equals ... 

Faber, K., Comment on a recently proposed resampling method, J. Chemom., 15, 169-188, 2001. 

Jaumot, J., Gargallo, R., and Tauler, R., Noise propagation and error estimations in multivariate curve-resolution alternating least squares using resampling methods, J. Chemom., 18, 324-340, 2004. 

Figure 6.2 A resampling method. A dataset is first randomly divided into two sets, such as 2/3 for training and 1/3 for testing. A model developed with the training set is accepted if it gives satisfactory predictions for the testing set. A set of predictive models is generated by repeating the procedure, and the predictions of these models are then combined when predicting a new chemical.

Wu CFJ. Jackknife, bootstrap and other resampling methods in regression analysis (with discussion). Ann Stat 1986 14 1261-95. 

Jackknife (IKK), cross-validation, and the bootstrap are the methods referred to as resampling techniques. Though not strictly classified as a resampling technique, the posterior predictive check is also covered in this chapter, as it has several characteristics that are similar to resampling methods. 

The role of resampling methods in PMM development and validation are explored. If applied, these techniques may bring efficiency to pharmacometric model development and result in models for which one s confidence level is very high. Patient pharmacotherapy will also be improved. One can expect to see resampling more extensively applied to modeling in the future. 

L. Sun and D. Muller-Schwarze, Statistical resampling methods in biology a case study of beaver dispersal patterns. Am J Math Manage Sci 16 463-502 (1996). 

H. Mager and G. Goller, Resampling methods in sparse sampling situations in prechnical pharmacokinetic studies. J Pharm Sci 87 372-378 (1008). 

External validation is applicable when either a large data set is available or a new data set has become available after generation of the model. In the former case, called the resampling method, we normally randomly sample a certain percentage of data for training and the rest for validation. Such a process can be repeated many times. It is noteworthy that the molecules included in the validation set have no role in model parameter estimation. 

Cross-validation is an internal resampling method much like the older Jackknife and Bootstrap methods [Efron 1982, Efron Gong 1983, Efron Tibshirani 1993, Wehrens et al. 2000]. The principle of cross-validation goes back to Stone [ 1974] and Geisser [ 1974] and the basic idea is simple ... 

Niedzwiecki, D. and Simonoff, J.S. Estimation and inference in pharmacokinetic models The effectiveness of model reformulation and resampling methods for functions of parameters. Journal of Pharmacokinetics and Biopharmaceutics 1990 18 361-377. 

Good, P. Permutation tests A practical guide to resampling methods for testing hypotheses. Springer-Verlag, New York, 2000. 

II. High-Dimensional Analysis Clustering analysis (Chapter 5), classification analysis (Chapter 6), multidimensional visualization (Chapter 7) in. Advanced Analysis Topics Statistical modeling (Chapter 8), experimental design (Chapter 9), statistical resampling methods (Chapter 10)... 

Resampling methods draw repeated samples from the observed sample itself to generate the sampling distribution of a statistic. The permutation method draws samples without replacement while the bootstrap method draws samples with replacement. These methods are useful for assessing the accuracy (e.g., bias and standard error) of complex statistics. 

In the case that the joint or marginal distribution of the test statistics is unknown, p-values can be estimated by resampling methods such as permutation and bootstrap. For example, consider a permutation algorithm to estimate p-values with large biological data in the following manner. First, permute the N sample columns of the data matrix and compute test statistics for each biomarker candidate. Let tij, be test statistics for the th permutation. When repeating this procedure many limes (e.g., B = 100 times), the permutation p-value for hypothesis Hj is... 

RESAMPLING METHODS FOR PREDICTION ERROR ASSESSMENT AND MODEL SELECTION ... 

To alleviate this biased estirrratiorL resampling methods, such as cross-validation and bootstrapping, can be employed to more accurately estimate prediction error. In the next sections, these techniques are described as well as the impUcations of their use in the framework of model selection and performance assessment. 

There are several considerations when selecting a resampling method. The foremost is the sample size n. Additional considerations are on the proportion of the observations reserved for the test set and the number of times the error estimate is calculated. We address these considerations in the following sections and also refer the reader to supplementary discussions in McLachlan (1992) and Davison and Hinkley (1997). 

---