Bayesian Information Criterion

Akaike allowed for the existence of other AlC-like criteria that could be derived by making different assumptions regarding the distribution of the data. Schwarz (1978), in a Bayesian context, developed the Bayesian Information Criteria (BIC), which is also called the Schwarz Information Criteria (SIC) or Schwarz s criteria (SC), as... 

The MAICE approach involves computing AIC values for each candidate model and then selecting the model with the lowest AIC value. The model with the lowest AIC value contains the most information per estimated parameter. Similar criteria, such as the Sawa or Schwarz Bayesian information criteria, can also be applied. 

A second approach considered both the Akaike and Bayesian information criteria. The Akaike information criterion (AIC) is an operational way of considering both the complexity of a model and how well it hts the data (Burnham and Anderson, 1998). The AIC methodology attempts to hnd the model that best explains the data with a minimum of free parameters. When residuals are randomly distributed, the AIC is calculated as... 

Selecting the best distribution function is not a trivial task. A wide variety of statistical data can be used in this duty, including standard deviations, R, Akaike and Bayesian Information Criteria, and even CPU time, which are aU presented in Table 12.23. It is well accepted that correlation coefficients are not very useful in discriminating between models. In this study, the correlation coefficients were very close to unity (0.986-0.999) for all of the functions. To highlight this point, only the Alpha distribution function exhibited a value of R lower than 0.99. 

In order to choose the model that predicts most accurately for the test data, we need a new rule or a new information criterion. The usual criteria, the Akaike Information Criterion (AIC), the Bayesian Information Criterion (BIC), Leave One Out (LOO or Qsquared), and so on, are all insufficient for our needs. This motivated Kerry Bemis to propose a new measure which he called predictive R2 or pR2 (described below). 

Figures 11 and 12 illustrate the performance of the pR2 compared with several of the currently popular criteria on a specific data set resulting from one of the drug hunting projects at Eli Lilly. This data set has IC50 values for 1289 molecules. There were 2317 descriptors (or covariates) and a multiple linear regression model was used with forward variable selection the linear model was trained on half the data (selected at random) and evaluated on the other (hold-out) half. The root mean squared error of prediction (RMSE) for the test hold-out set is minimized when the model has 21 parameters. Figure 11 shows the model size chosen by several criteria applied to the training set in a forward selection for example, the pR2 chose 22 descriptors, the Bayesian Information Criterion chose 49, Leave One Out cross-validation chose 308, the adjusted R2 chose 435, and the Akaike Information Criterion chose 512 descriptors in the model. Although the pR2 criterion selected considerably fewer descriptors than the other methods, it had the best prediction performance. Also, only pR2 and BIC had better prediction on the test data set than the null model.

In the second solution phase (problem P2) the rninirnization of the Bayesian Information Criterion (BIC) is directly considered, subject to model constraints, using a recursive estimation of the variance of the data [10]. The optimization problems solved in this case correspond to mixed integer quadratic programs (MIQP). 

A common form of model selection is to maximize the likelihood that the data arose under the model. For non-Bayesian analysis this is the basis of the likelihood ratio test, where the difference of two -2LL (where LL denotes the log-Ukelihood) for nested models is assumed to be approximately asymptotically chi-squared distributed. A Bayesian approach— see also the Schwarz criterion (36)—is based on computation of the Bayesian information criterion (BIC), which minimizes the KuUback-Leibler KL) information (37). The KL information relates to the ratio of the distribution of the data given the model and parameters to the underlying true distribution of the data. The similarity of the KL information expression (Eq. (5.24)) and Bayes s formula (Eq. (5.1)) is easily seen ... 

Schwarz Bayesian Information Criterion regression parameters (0 Table Rl)... 

Model comparison plays a central role in statistical learning and chemometrics. Performances of models need to be assessed using a given criterion based on which models can be compared. To our knowledge, there exist a variety of criteria that can be applied for model assessment, such as Akaike s information criterion (AIC) [1], Bayesian information criterion (BIC) [2], deviance information criterion (DlC),Mallow s Cp statistic, cross validation [3-6] and so on. There is a large body of literature that is devoted to these criteria. With the aid of a chosen criterion, different models can be comp>ared. For example, a model with a smaller AIC or BIC is preferred if AIC or BIC are chosen for model assessment. 

For large data, this can be approximated as the so-called Bayesian information criterion (BIC) ... 

When fitting models, the MLE is used to find the optimal fit to the dataset. However, maximizing the log likelihood often results in fitting noise and parameter estimates that are unstable, particularly when the data set is relatively small. This is because MLE trusts too much the observed trends in the, often limited, data (Moons et al., 2004). In order to avoid possible over-fitting, the Bayesian Information Criterion (BIC) was utilized (Schwarz, 1978). BIC is a criterion for model selection, and includes a penalty term for the number of parameters in the model. The BIC is given by the following equation ... 

Structural identification, i.e. selection of the model type and structure, is always an arbitrary research decision. What is helpful is autocorrelation and spectrum analysis (detection of the intervals). Generally, the simplest possible model is chosen. A series of information criteria (algorithms) exist that may help in this process, usually defined as a combination of the model error and the number of model parameters, such as the AIC criterion (Akaike s information criterion), the criterion of the final error of the prediction, Ravelli Vulpiani criterion or Schwarz s BIC criterion (Bayesian information criterion comparison of log likelihood of specific models corrected by the number of estimated parameters and the number of observations). 

The expression to calculate the Bayesian information criterion for models with randomly distributed residuals is ... 

Distribution parameters Akaike information criterion Bayesian information criterion Bromine number (g Br/lOOg)... 

With GAM the data (covariate and individual Bayesian PM parameter estimates) would be subjected to a stepwise (single-term addition/deletion) modeling procedure. Each covariate is allowed to enter the model in any of several functional representations. The Akaike information criterion (AIC) is used as the model selection criterion (22). At each step, the model is changed by addition or deletion of a covariate that results in the largest decrease in the AIC. The search is stopped when the AIC reached a minimum value. 

The SIC is deduced from Bayesian arguments. It consistently estimates the true order of ARMA(p, q) processes and is probably the most widely used information criterion in univariate time series analysis. The HQIC is the most recent IC and especially designed for multivariate time series models. In practice, multiple ICs are simultaneously calculated which allows the analyst to cross-check the recommendations of the various ICs. Strongly deviating recommendations may indicate an inappropriate model structure. 

In Sections 2 to 4, we review the technology of synthetic oligonucleotide microarrays and describe some of the popular statistical methods that are used to discover genes with differential expression in simple comparative experiments. A novel Bayesian procedure is introduced in Section 5 to analyze differential expression that addresses some of the limitations of current procedures. We proceed, in Section 6, by discussing the issue of sample size and describe two approaches to sample size determination in screening experiments with microarrays. The first approach is based on the concept of reproducibility, and the second approach uses a Bayesian decision-theoretic criterion to trade off information gain and experimental costs. We conclude, in Section 7, with a discussion of some of the open problems in the design and analysis of microarray experiments that need further research. 

Haines et al. (47) suggested including the criterion Bayesian D-optimality, which maximizes some concave function of the information matrix, which in essence is the minimization of the generalized variance of the maximum likelihood estimators of the two parameters of the logistic regression. The authors underline that toxicity is recorded as an ordinal variable and not a simple binary variable, and that the present design needs to be extended to proportional odds models. 

---