Posterior predictive check

Yano, Y., Beal, S. L., Sheiner, L. B. Evaluating pharmacokinetic/ pharmacodynamic models using the posterior predictive check. J Pharmacokinet Pharmacodyn 2001, 28 171-192. 

Other model selection and/or discrimination tools include the posterior predictive check (PPG) and cross-validation (27,32,33). The PPG is useful for examination of the ability of the model to predict accurately certain features of the observed data (e.g., maximum concentration). Although PPG is not strictly a model discrimination technique, as it does not compare the predictive performance between models but rather evaluates the predictive performance of a single model, it does have useful characteristics that are discussed in more detail in Section 5.3.3. Gross-validation is considered accurate but is computer intensive and generally considered not to be suitable for small data sets (32). 

In contrast to the hypothesis testing style of model selection/discrimination, the posterior predictive check (PPC) assesses the predictive performance of the model. This approach allows the user to reformulate the model selection decision to be based on how well the model performs. This approach has been described in detail by Gelman et al. (27) and is only briefly discussed here. PPC has been assessed for PK analysis in a non-Bayesian framework by Yano et al. (40). Yano and colleagues also provide a detailed assessment of the choice of test statistics. The more commonly used test statistic is a local feature of the data that has some importance for model predictions for example, the maximum or minimum concentration might be important for side effects or therapeutic success (see Duffull et al. (6)) and hence constitutes a feature of the data that the model would do well to describe accurately. The PPC can be defined along the fines that posterior refers to conditioning of the distribution of the parameters on the observed values of the data, predictive refers to the distribution of future unobserved quantities, and check refers to how well the predictions reflect the observations (41). This method is used to answer the question Does the observed data look plausible under the posterior distribution This method is therefore solely a check of internal consistency of the model in question. 

Very often a test population of data is not available or would be prohibitively expensive to obtain. When a test population of data is not possible to obtain, internal validation must be considered. The methods of internal PM model validation include data splitting, resampling techniques (cross-validation and bootstrapping) (9,26-30), and the posterior predictive check (PPC) (31-33). Of note, the jackknife is not considered a model validation technique. The jackknife technique may only be used to correct for bias in parameter estimates, and for the computation of the uncertainty associated with parameter estimation. Cross-validation, bootstrapping, and the posterior predictive check are addressed in detail in Chapter 15. 

A recently proposed method, the posterior predictive check (PPC), may prove useful in determining whether important clinical features of present and future data sets are faithfully reproduced (31-33). The PPC is addressed in Chapter 15 and wiU not be discussed further here. 

Jackknife (IKK), cross-validation, and the bootstrap are the methods referred to as resampling techniques. Though not strictly classified as a resampling technique, the posterior predictive check is also covered in this chapter, as it has several characteristics that are similar to resampling methods. 

Step 7. Compare statistics from the replicated data sets to the original data set. A schematic representation of posterior predictive check is shown in Figure 15.3. 

FIGURE 15.3 Schematic representation of the posterior predictive check flow. 

The focus of this section is on some of the code that has been used to generate the data that has been used for the previous examples and on exploring how one might automate a posterior predictive check with or without a bootstrap step for the weight change example. 

FIGURE 28.15 Posterior predictive check with bootstrap step for weight change mixture problem. 

Another internal validation technique is the posterior predictive check (PPC), which has been used in the Bayesian literature for years, but only recently reported in the PopPK literature by Yano, Beal, and Sheiner (2001). The basic idea is an extension of the predictive check method just described but include hyperparameters on the model parameters. Data are then simulated, some statistic of the data that is not based on the model is calculated, e.g., half-life or AUC by noncompartmental method, and then compared to the observed statistic obtained with real data. The underlying premise is that the simulated data should be similar to the observed data and that any discrepancies between the observed and simulated data are due to chance. With each simulation the statistic of interest is calculated and after all the simulations are complete, a p-value is determined by... 

Figure 7.16 Histogram of posterior predictive check based on the observed data in Table 7.4. Concentration data were simulated for 26 subjects under the original experimental design and sampling times at each dose using population values and variance components randomly drawn from the bootstrap distribution of the final model parameter estimates (FOCE-I Table 7.5). The geometric mean concentration at 6-h postdose (top) and AUC to 12-h postdose (bottom) was calculated. This process was repeated 250 times.

Few published studies exist in which a posterior predictive check (PPC) has been employed as a validation technique for a population pharmacokinetic analysis. 

---