Classification and prediction methods

In this section widely used classification methods are discussed. For description of each method, we use the following notation. Let the total sample size of each dataset be n, with m total predictors under consideration. The predictors are represented by an n X m matrix X = (Xjj), where Xy is the yth measurement for an independent observation i. The class label for the observation i is y where y = (yi, ya. . yn) and y, is defined to be 0 or 1 depending on class status. 

Bayesian decision theory is a fundamental statistical approach to the problem of classification. This approach is based on quantifying the trade-offs between various classification decisions using probability and the costs that accompany such decisions. It makes the assumption that the decision problem is posed in probabilistic terms and that all of the relevant probability values are known. 

A naive Bayes classifier is a simple probabilistic classifier based on the so-called Bayes theorem with strong independence assumptions and is particularly suited when the dimensionality of the inputs is high. The naive Bayes model assumes that, given a class r = j, the features X, are independent. Despite its simplicity, the naive Bayes classifier is known to be a robust method even if the independence assumption does not hold (Michalski and Kaufman, 2001). 

The probabUistic model for a naive Bayes classifier is a conditional model P(T Xi, X2. X ) over a dependent class variable F, conditional on features Xi, X2, X. Using Bayes s theorem, F(F Xj. X ) oc P(F) 7(Xi. X F). The prior probability F(F = j) can be calculated based on the ratio of the class j samples such as P(F = 7) = (number of class j samples)/(total number of samples). Having formulated the prior probabihties, the likelihood function p(Xi, X2. X F) can be written as ]/[ j p(Xi F) under the naive conditional independence assumptions of the feature X, with the feamre Xj for j i. A new sample is classified to a class with maximum posterior probability, which is argmaxr erF (r7)nr ( i 1 /)- If the independence assumption is correct, it is the Bayes optimal classifier for a problem. Extensions of the naive Bayes classifier can be found in Demichelis et al. (2006). 

Logistic regression is a model used for prediction of the probabihty of occurrence of an event. It makes use of several predictor variables that may be either numerical or 

---

CLASSIFICATION AND PREDICTION METHODS 133 categorical. Logistic regression analyzes binomiaUy distributed data of the form Yi -- Bin( ,-, Pi) i= 1,2,m,... 

Weiss, S. H. and Kulikowski, C. A. (1991) Computer Systems that Learn Classification and Prediction Methods from Statistics, Neural Networks, Machine Learning, and Expert Systems, Morgan Kaufmann, San Mateo, CA. 

Burger J, Gowen A. Classification and prediction methods. In Park B, editor. Hyperspectral imaging technology in food and agriculture. New York Springer 2012. 

The two methods have been compared by Moret et al. in the classification of three white wines they give about the same predictive ability, lower, of course, than the classification ability. The difference between classification and predictive ability becomes greatest when the number of objects is not much greater than the number of variables, or when there are very few objects in a category. A very low predicitive ability and a high classification ability reveal a bad experimental design. 

The category correlations can be cancelled only when all the objects of the training set are in the same category, and the method is used as a class modelling technique. However, the bayesian analysis in ARTHUR-BACLASS has b n compared with the usual BA in classification problems about winra and olive oils and about the same classification and prediction abilities were observe for both methods. 

Today, analysis and prediction methods have mostly a statistical flavor. They are trained on classified data and make new classifications or predictions based on statistical models. In some sense, all Chapters of Volume 1 present such methods. For instance, the methods for homology-based protein structure prediction in Chapter 5 and 6 of volume 1 learn from a set of observed structures rules that predict alike structures. 

Often the goal of a data analysis problem requites more than simple classification of samples into known categories. It is very often desirable to have a means to detect oudiers and to derive an estimate of the level of confidence in a classification result. These ate things that go beyond sttictiy nonparametric pattern recognition procedures. Also of interest is the abiUty to empirically model each category so that it is possible to make quantitative correlations and predictions with external continuous properties. As a result, a modeling and classification method called SIMCA has been developed to provide these capabihties (29—31). 

We will explore the two major families of chemometric quantitative calibration techniques that are most commonly employed the Multiple Linear Regression (MLR) techniques, and the Factor-Based Techniques. Within each family, we will review the various methods commonly employed, learn how to develop and test calibrations, and how to use the calibrations to estimate, or predict, the properties of unknown samples. We will consider the advantages and limitations of each method as well as some of the tricks and pitfalls associated with their use. While our emphasis will be on quantitative analysis, we will also touch on how these techniques are used for qualitative analysis, classification, and discriminative analysis. 

In Section 4.2, we treated the performance of regression models. For classification methods, these concepts remain valid and can be directly used. However, there is an important difference concerning the performance measures to be used. While for regression the basic information for the evaluation measures are the residuals, i.e., the difference between observed and predicted v-values, this would... 

QSAR methods are in part retrospective as well as predictive, since a training set of compounds of known pharmacological activity must first be established. The purpose of such methods is to increase the probability of finding active compounds among those evenmally synthesized, thus keeping synthetic and screening efforts within reasonable limits in relation to the success rate. There are three main classifications of QSAR methods ... 

Two methods are used to evaluate the predictive ability for LDA and for all other classification techniques. One method consists of dividing the objects of the whole data set into two subsets, the training and the prediction or evaluation set. The objects of the training set are used to obtain the covariance matrix and the discriminant scores. Then, the objects of the training set are classified, so obtaining the apparent error rate and the classification ability, and the objects of the evaluation set are classified to obtain the actual error rate and the predictive ability. The subdivision into the training and prediction sets can be randomly repeated many times, and with different percentages of the objects in the two sets, to obtain a better estimate of the predictive ability. 

---