Class separation, discriminant analysi

Discriminant emalysis is a supervised learning technique which uses classified dependent data. Here, the dependent data (y values) are not on a continuous scale but are divided into distinct classes. There are often just two classes (e.g. active/inactive soluble/not soluble yes/no), but more than two is also possible (e.g. high/medium/low 1/2/3/4). The simplest situation involves two variables and two classes, and the aim is to find a straight line that best separates the data into its classes (Figure 12.37). With more than two variables, the line becomes a hyperplane in the multidimensional variable space. Discriminant analysis is characterised by a discriminant function, which in the particular case of hnear discriminant analysis (the most popular variant) is written as a linear combination of the independent variables ... 

The surface that actually separates the classes is orthogonal to this discriminant function, as shown in Figure 12.37, and is chosen to maximise the number of compounds correctly classified. To use the results of a discriminant analysis, one simply calculates the appropriate value of the discriminant function, from which the class can be determined. 

A first distinction which is often made is that between methods focusing on discrimination and those that are directed towards modelling classes. Most methods explicitly or implicitly try to find a boundary between classes. Some methods such as linear discriminant analysis (LDA, Sections 33.2.2 and 33.2.3) are designed to find explicit boundaries between classes while the k-nearest neighbours (A -NN, Section 33.2.4) method does this implicitly. Methods such as SIMCA (Section 33.2.7) put the emphasis more on similarity within a class than on discrimination between classes. Such methods are sometimes called disjoint class modelling methods. While the discrimination oriented methods build models based on all the classes concerned in the discrimination, the disjoint class modelling methods model each class separately. 

Equation (33.10) is applied in what is called quadratic discriminant analysis (QDA). The equations can be shown to describe a quadratic boundary separating the regions where is minimal for the classes considered. 

In the class discrimination methods or hyperplane techniques, of which linear discriminant analysis and the linear learning machine are examples, the equation of a plane or hyperplane is calculated that separates one class from another. These methods work well if prior knowledge allows the analyst to assume that the test objects must... 

Because of the aforementioned EDA hypotheses, the ellipses of different categories present equal eccentricity and axis orientation they only differ for their location in the plane. By coimecting the intersection points of each couple of corresponding ellipses, a straight line is identified which corresponds to the delimiter between the two classes (see Eig. 2.15B). Eor this reason, this technique is called linear discriminant analysis. The directions which maximize the separation between classes are called EDA canonical variables. 

Quadratic discriminant analysis (QDA) is a probabilistic parametric classification technique which represents an evolution of EDA for nonlinear class separations. Also QDA, like EDA, is based on the hypothesis that the probability density distributions are multivariate normal but, in this case, the dispersion is not the same for all of the categories. It follows that the categories differ for the position of their centroid and also for the variance-covariance matrix (different location and dispersion), as it is represented in Fig. 2.16A. Consequently, the ellipses of different categories differ not only for their position in the plane but also for eccentricity and axis orientation (Geisser, 1964). By coimecting the intersection points of each couple of corresponding ellipses (at the same Mahalanobis distance from the respective centroids), a parabolic delimiter is identified (see Fig. 2.16B). The name quadratic discriminant analysis is derived from this feature. 

The MANOVA enables significant class separation with a multivariate scaled separation measure of 330.9. The sampling times 5 a.m. and 11 p.m. are well separable from the times 11 a.m. and 5 p.m. by the optimum separation set which consists in the features suspended material, iron, magnesium, nickel, and copper. The result of discriminant analysis is shown in the plane of the two strongest discriminant functions (Fig. 8-3). 

The principle of multivariate analysis of variance and discriminant analysis (MVDA) consists in testing the differences between a priori classes (MANOVA) and their maximum separation by modeling (MDA). The variance between the classes will be maximized and the variance within the classes will be minimized by simultaneous consideration of all observed features. The classification of new objects into the a priori classes, i.e. the reclassification of the learning data set of the objects, takes place according to the values of discriminant functions. These discriminant functions are linear combinations of the optimum set of the original features for class separation. The mathematical fundamentals of the MVDA are explained in Section 5.6. 

Figure 7.4 2D representation of discriminant analysis. The dotted line represents the discriminant function and the solid line represents a discriminant surface that separates the two classes of samples. (From Livingstone, D.J., Data Analysis for Chemists Applications to QSAR and Chemical Product Design, Oxford University Press, Oxford, 1995. Reproduced with permission of Oxford University Press.)... 

When using a linear method, such as LDA, the underlying assumption is that the two classes are linearly separable. This, of course, is generally not true. If linear separability is not possible, then with enough samples, the more powerful quadratic discriminant analysis (QDA) works better, because it allows the hypersurface that separates the classes to be curved (quadratic). Unfortunately, the clinical reality of small-sized data sets denies us this choice. 

Linear discriminant analysis (LDA) [41] separates two data classes of feature vectors by constructing a hyperplane defined by a linear discriminant function ... 

Linear discriminant analysis (LDA) is aimed at finding a linear combination of descriptors that best separate two or more classes of objects [100]. The resulting transformation (combination) may be used as a classifier to separate the classes. LDA is closely related to principal component analysis and partial least square discriminant analysis (PLS-DA) in that all three methods are aimed at identifying linear combinations of variables that best explain the data under investigation. However, LDA and PLS-DA, on one hand, explicitly attempt to model the difference between the classes of data whereas PCA, on the other hand, tries to extract common information for the problem at hand. The difference between LDA and PLS-DA is that LDA is a linear regression-like method whereas PLS-DA is a projection technique... 

Fisher suggested to transform the multivariate observations x to another coordinate system that enhances the separation of the samples belonging to each class tt [74]. Fisher s discriminant analysis (FDA) is optimal in terms of maximizing the separation among the set of classes. Suppose that there is a set of n = ni + U2 + + rig) m-dimensional (number of process variables) samples xi, , x belonging to classes tt, i = 1, , g. The total scatter of data points (St) consists of two types of scatter, within-class scatter Sw and hetween-class scatter Sb- The objective of the transformation proposed by Fisher is to maximize S while minimizing Sw Fisher s approach does not require that the populations have Normal distributions, but it implicitly assumes that the population covariance matrices are equal, because a pooled estimate of the common covariance matrix (S ) is used (Eq. 3.45). 

Discriminant plots were obtained for the adaptive wavelet coefficients which produced the results in Table 2. Although the classifier used in the AWA was BLDA, it was decided to supply the coefficients available upon termination of the AWA to Fisher s linear discriminant analysis, so we could visualize the spatial separation between the classes. The discriminant plots are produced using the testing data only. There is a good deal of separation for the seagrass data (Fig. 5), while for the paraxylene data (Fig. 6) there is some overlap between the objects of class I and 3. Quite clearly, the butanol data (Fig. 7) post a challenge in discriminating between the two classes. 

Discriminant analysis (Figure 31) [41,487, 577 — 581] separates objects with different properties, e.g. active and inactive compounds, by deriving a linear combination of some other features e.g. of different physicochemical properties), which leads to the best separation of the individual classes. Discriminant analysis is also appropriate for semiquantitative data and for data sets, where activities are only characterized in qualitative terms. As in pattern recognition, training sets are used to derive a model and its stability and predictive ability is checked with the help of different test sets. 

The adaptive least squares (ALS) method [396, 585 — 588] is a modification of discriminant analysis which separates several activity classes e.g. data ordered by a rating score) by a single discriminant function. The method has been compared with ordinary regression analysis, linear discriminant analysis, and other multivariate statistical approaches in most cases the ALS approach was found to be superior to categorize any numbers of classes of ordered data. ORMUCS (ordered multicate-gorial classification using simplex technique) [589] is an ALS-related approach which... 

In multi-class problems more than one discriminant function results (Q-1 for Q classes). A favourable aspeet of equation (34) and, thus, of non-elementary discriminant analysis is that the separating power of the diseriminant functions decreases as the eigenvalues decrease in the order 1> 2...> /...> q-1. This makes it possible to eliminate those last discriminant functions which fail to contribute significantly to class separation, thereby diminishing the dimensionality of the diseriminant problem. 

Once the classificator (wg and the w,) is known new compounds can be classified by inserting their values of x, into equation (45). Reclassification of the compounds of the training series is performed in the same way. The separating power of the classificator is judged in the usual way by the error of reclassification and the error of classification for a test set of compounds or for a simulated prediction using the leave- -out technique as already discussed in eoimection with discriminant analysis. In two-class problems sign and magnitude... 

---