Bootstrapping error estimates

Although this approach is still used, it is undesirable for statistical reasons error calculations underestimate the true uncertainty associated with the equations (17, 21). A better approach is to use the equations developed for one set of lakes to infer chemistry values from counts of taxa from a second set of lakes (i.e., cross-validation). The extra time and effort required to develop the additional data for the test set is a major limitation to this approach. Computer-intensive techniques, such as jackknifing or bootstrapping, can produce error estimates from the original training set (53), without having to collect data for additional lakes. 

Furthermore, when alternative approaches are applied in computing parameter estimates, the question to be addressed here is Do these other approaches yield similar parameter and random effects estimates and conclusions An example of addressing this second point would be estimating the parameters of a population pharmacokinetic (PPK) model by the standard maximum likelihood approach and then confirming the estimates by either constructing the profile likelihood plot (i.e., mapping the objective function), using the bootstrap (4, 9) to estimate 95% confidence intervals, or the jackknife method (7, 26, 27) and bootstrap to estimate standard errors of the estimate (4, 9). When the relative standard errors are small and alternative approaches produce similar results, then we conclude the model is reliable. 

Convergence was successful for 502 of the 505 bootstrap data sets. Three data sets persisted in terminating with rounding errors despite the application of several sets of starting parameters. The results of the bootstrap parameter estimates are presented in Table 15.2 and compared to the results from the PPK model building process. There is strong evidence that the model is without substantive deficiencies and should be accepted as the final irreducible model. 

More difficult is the estimation of errors for the nonlinear parameters, since no variance-covariance matrix exists. Frequently, the error estimations are restricted to a locally linear range. In the linearization range, the confidence bands for the parameters are then calculated as in the linear case (Eqs. (6.25)-(6.27)). An alternative consists in error estimations on the basis of Monte Carlo simulations or bootstrapping methods (cf. Section 8.2). 

Two estimators have been suggested to correct the bias observed in the leave-one-out bootstrap the 0.632 bootstrap and the 0.632+ estimator (Efron and Tibshirani, 1997). Both corrections adjust the overestimated leave-one-out bootstrap error, denoted eiTLooB by adding the underestimated resubstitution error, denoted err sE. This... 

Standard deviation, MSE, and bias of all methods. The small k chosen for two-fold CV and split sample with p = is due to the reduced training set size. For r-fold CV, a significant decrease in prediction error, bias, and MSE is seen as v increases from 2 to 10. Tenfold CV has a slightly decreased error estimate compared to LOOCV as well as a smaller standard deviation, bias, and MSE however, the LOOCV k is smaller than that of 10-fold CV. Repeated 5-fold CV decreases the standard deviation and MSE over 5-fold CV however, values for the bias and k are slightly larger. In comparison to 10-fold CV, the 0.632-1- bootstrap has a smaller standard deviation and MSE with a larger prediction error, bias, and k. 

Techniques to use for evaluations have been discussed by Cox and Tikvart (42), Hanna (43) and Weil et al. (44). Hanna (45) shows how resampling of evaluation data will allow use of the bootstrap and jackknife techniques so that error bounds can be placed about estimates. 

Thus, while the probability of a particular work value and the distribution of work values can give some estimate of the relative error in the free energy (for example by performing bootstrap or subsampling analysis over the full data set), there is no inherent way to extrapolate from the full finite data set to a larger (better converged) estimate. 

Vertzoni et al. (30) recently clarified the applicability of the similarity factor, the difference factor, and the Rescigno index in the comparison of cumulative data sets. Although all these indices should be used with caution (because inclusion of too many data points in the plateau region will lead to the outcome that the profiles are more similar and because the cutoff time per percentage dissolved is empirically chosen and not based on theory), all can be useful for comparing two cumulative data sets. When the measurement error is low, i.e., the data have low variability, mean profiles can be used and any one of these indices could be used. Selection depends on the nature of the difference one wishes to estimate and the existence of a reference data set. When data are more variable, index evaluation must be done on a confidence interval basis and selection of the appropriate index, depends on the number of the replications per data set in addition to the type of difference one wishes to estimate. When a large number of replications per data set are available (e.g., 12), construction of nonparametric or bootstrap confidence intervals of the similarity factor appears to be the most reliable of the three methods, provided that the plateau level is 100. With a restricted number of replications per data set (e.g., three), any of the three indices can be used, provided either non-parametric or bootstrap confidence intervals are determined (30). 

Cross validation and bootstrap techniques can be applied for a statistically based estimation of the optimum number of PCA components. The idea is to randomly split the data into training and test data. PCA is then applied to the training data and the observations from the test data are reconstmcted using 1 to m PCs. The prediction error to the real test data can be computed. Repeating this procedure many times indicates the distribution of the prediction errors when using 1 to m components, which then allows deciding on the optimal number of components. For more details see Section 3.7.1. 

Determination of the optimum complexity of a model is an important but not always an easy task, because the minimum of measures for the prediction error for test sets is often not well marked. In chemometrics, the complexity is typically controlled by the number of PLS or PCA components, and the optimum complexity is estimated by CV (Section 4.2.5). Several strategies are applied to determine a reasonable optimum complexity from the prediction errors which may have been obtained by CV (Figure 4.4). CV or bootstrap allows an estimation of the prediction error for each object of the calibration set at each considered model complexity. 

An approach that is sometimes helpful, particularly for recent pesticide risk assessments, is to use the parameter values that result in best fit (in the sense of LS), comparing the fitted cdf to the cdf of the empirical distribution. In some cases, such as when fitting a log-normal distribution, formulae from linear regression can be used after transformations are applied to linearize the cdf. In other cases, the residual SS is minimized using numerical optimization, i.e., one uses nonlinear regression. This approach seems reasonable for point estimation. However, the statistical assumptions that would often be invoked to justify LS regression will not be met in this application. Therefore the use of any additional regression results (beyond the point estimates) is questionable. If there is a need to provide standard errors or confidence intervals for the estimates, bootstrap procedures are recommended. 

Cross-validation is an alternative to the split-sample method of estimating prediction accuracy (5). Molinaro et al. describe and evaluate many variants of cross-validation and bootstrap re-sampling for classification problems where the number of candidate predictors vastly exceeds the number of cases (13). The cross-validated prediction error is an estimate of the prediction error associated with application of the algorithm for model building to the entire dataset. 

Figure 3. Reconstructions of (A) diatom-based and (B) chrysophyte-based monomeric Al for Big Moose Lake, and diatom-based monomeric Al for (C) Deep Lake, (D) Upper Wallface Pond, and (E) Windfall Pond in the Adirondack Mountains, New York. Reconstructions are bounded by bootstrapping estimates of the root mean-squared error of prediction for each sample. Bars to the right of each reconstruction indicate historical (H) and Chaoborus-based (C) reconstructions of fishery resources. The historical fish records are not continuous, unlike the paleolimnological records. Intervals older than 1884 are dated by extrapolation. (Reproduced with permission from reference 10.

Efron, B. (1981) Non parametric estimates of standard error the jack-knife, the bootstrap and other methods, Biometrika 68, 589-599. 

RF [29,30] is an ensemble of unpruned classification trees separately grown from bootstrap samples of the training data set. A subset of nitry input variables is randomly selected as candidates to determine the best possible split at each node during the tree induction. The final prediction is generally made by aggregating the outputs of all the ntree trees generated in the forest. The unbiased out-of-bag (OOB) estimate of the generalization error is used to internally evaluate the prediction performance ofRF. 

When a model is used for descriptive purposes, goodness-of-ht, reliability, and stability, the components of model evaluation must be assessed. Model evaluation should be done in a manner consistent with the intended application of the PM model. The reliability of the analysis results can be checked by carefully examining diagnostic plots, key parameter estimates, standard errors, case deletion diagnostics (7-9), and/or sensitivity analysis as may seem appropriate. Conhdence intervals (standard errors) for parameters may be checked using nonparametric techniques, such as the jackknife and bootstrapping, or the prohle likelihood method. Model stability to determine whether the covariates in the PM model are those that should be tested for inclusion in the model can be checked using the bootstrap (9). 

E. I. Ette and L. C. Onyiah, Estimating inestimable standard errors in population pharmacokinetic studies the bootstrap with winsorization. Ear J Drug Metab Pharmacokinet 27 213-224 (2002). 

---