Classification hyperplane

Figure 15 The classification hyperplane defines a region for class +1 and another region for class -1.

The last two experiments for the linearly separable dataset are performed with the Gaussian RBF kernel (a = 1 Figure 25a) and the B spline kernel (degree 1 Figure 25b). Although not optimal, the classification hyperplane for the Gaussian RBF kernel is much better than those obtained with the exponential RBF kernel and degree 10 polynomial kernel. On the other hand, SVM... 

In this section, we present the most used SVM kernels. As these functions are usually computed in a high-dimensional space and have a nonlinear character, it is not easy to derive an impression on the shape of the classification hyperplane generated by these kernels. Therefore, we will present several plots for SVM models obtained for the dataset shown in Table 5. This dataset is not separable with a linear classifier, but the two clusters can be clearly distinguished. 

An important question to ask is as follows Do SVMs overfit Some reports claim that, due to their derivation from structural risk minimization, SVMs do not overfit. However, in this chapter, we have already presented numerous examples where the SVM solution is overfitted for simple datasets. More examples will follow. In real applications, one must carefully select the nonlinear kernel function needed to generate a classification hyperplane that is topologically appropriate and has optimum predictive power. 

Using a descriptor selection procedure, we found that only three descriptors ( homo lumo nd Q ) are essential for the SVM model. To exemplify the shape of the classification hyperplane for polar and nonpolar narcotic pollutants, we selected 20 compounds (Table 7) as a test set (nonpolar compounds, class +1 polar compounds, class —1). 

In fact, b() + b xj gives the signed distance of an object jc, to the decision plane, and for classification only the sign is primarily important (although the distance from the decision plane may be used to measure the certainty of classification). If the two groups are linearly separable, one can find a hyperplane which gives a perfect group separation as follows ... 

Up to this point the methods of classification operate in the same way. They differ considerably, however, in the way that rules for classification are derived. In this regard the various methods are of three types 1) class discrimination or hyperplane methods, 2) distance methods, and 3) class modeling methods. 

Only one class modeling method is conmonly applied to analytical data and this is the SIMCA method ( ) of pattern recognition. In this method the class structure (cluster) is approximated by a point, line, plane, or hyperplane. Distances around these geometric functions can be used to define volumes where the classes are located in variable space, and these volumes are the basis for the classification of unknowns. This method allows the development of information beyond class assignment ( ). 

Feature selection is the process by which the data or variables liq>or-tant for class assignment are determined. In this step of a pattern recognition study the various methods differ considerably. In the hyperplane methods, the strategy is to begin with a block of variables for the classes, calculate a classification function, and test it for classification of the training set. In this initial phase, generally many more variables are included than are necessary. Variables are then detected in a stepwise process and a new rule is derived and tested. This process is repeated until a set of variables is obtained that will give an acceptable level of classification. 

Class assignment, the methods of classification discussed earlier, differs considerably. In the hyperplane methods, a plane or hyper plane is calculated that separates each class, and class assignment is based on the side of this discriminant plane on which the unknown falls. The limitation of this approach is that it requires prior knowledge (or an assumption) that the unknown be a member of one of the classes in the training sets. 

Support vector machines In addition to more traditional classification methods like clustering or partitioning, other computational approaches have recently also become popular in chemoinformatics and support vector machines (SVMs) (Warmuth el al. 2003) are discussed here as an example. Typically, SVMs are applied as classifiers for binary property predictions, for example, to distinguish active from inactive compounds. Initially, a set of descriptors is selected and training set molecules are represented as vectors based on their calculated descriptor values. Then linear combinations of training set vectors are calculated to construct a hyperplane in descriptor space that best separates active and inactive compounds, as illustrated in Figure 1.9. 

Although perceptrons are quite useful for a wide variety of classification problems, their usefulness is limited to problems that are linearly separable problems in which a line, plane or hyperplane can effect the desired dichotomy. As an example of a non-linearly separable problem, see Figure 3.4. This is just Figure 3.1 with an extra point added (measure 1 =. 8 and measure 2 =. 9) but this point makes it inpossible to find a line that can separate the depressed from non-depressed. This is no longer a linearly separable problem, and a simple perceptron will not be able to find a solution. However, note that a simple curve can effectively separate the two groups. Multilayer perceptrons, discussed in the next section, can be used for classification, even in the presence of nonlinearities. 

The essence of the differences between the operation of radial basis function networks and multilayer perceptrons can be seen in Figure 4.1, which shows data from the hypothetical classification example discussed in Chapter 3. Multilayer perceptrons classify data by the use of hyperplanes that divide the data space into discrete areas radial basis functions, on the other hand, cluster the data into a finite number of ellipsoid regions. Classification is then a matter of finding which ellipsoid is closest for a given test data point. 

The main advantage of SVM over other data analysis methods is its relatively low sensitivity to data overfitting, even with the use of a large number of redundant and overlapping molecular descriptors. This is due to its reliance on the structural risk minimization principle. Another advantage of SVM is the ability to calculate a reliability score, R-value, which provides a measure of the probability of a correct classification of a compound [70], The R-value is computed by using the distance between the position of the compound and the hyperplane in the hyperspace. The expected classification accuracy for the compound can then be obtained from the 7 -value by using a chart which shows the statistical relationship between them. As with other methods, SVM requires a sufficient number of samples to develop a classification system and irrelevant molecular descriptors may reduce the prediction accuracies of the SVM classification systems. 

SVMs were originally designed as a classification method using advanced mathematics to position a hyperplane to define and separate two or more classes. In later versions, it can also be used to predict continuous data. They are becoming increasingly popular in QSAR studies. 

The SVM method, introduced by Vapnik (32) in 1995, is applicable for both classification and regression problems. In case of classification, SVM are used to determine a boundary, a hyperplane, which separates classes independently of the probabilistic distributions of samples in the data set and maximizes the distance between these classes. The decision boundary is determined calculating a function f(x) = y(x) (32-34). The technique is gaining popularity fast in... 

With the LLM and discriminant analysis covered in this section, classification of an object is carried out strictly by assigning it to the class on either side of the separating plane (hyperplane). To deal with overlapping classes, one approach is to allow for some objects to be on the wrong side of the margin. 

Given the classification vector, y, in the interval [—1, +1], a function f(x) =x w+Wq withy, /( , ) > 0 can be found for all i. Then a hyperplane can be computed that creates the biggest margin between the training points for classes 1 and —1. The optimization problem is then given by... 

Support vector machine (SVM) is originally a binary supervised classification algorithm, introduced by Vapnik and his co-workers [13, 32], based on statistical learning theory. Instead of traditional empirical risk minimization (ERM), as performed by artificial neural network, SVM algorithm is based on the structural risk minimization (SRM) principle. In its simplest form, linear SVM for a two class problem finds an optimal hyperplane that maximizes the separation between the two classes. The optimal separating hyperplane can be obtained by solving the following quadratic optimization problem ... 

Support vector machine (SVM) is a widely used machine learning algorithm for binary data classification based on the principle of structural risk minimization (SRM) [21, 22] unlike the traditional empirical risk minimization (ERM) of artificial neural network. For a two class problem SVM finds a separating hyperplane that maximizes the width of separation of between the convex hulls of the two classes. To find the expression of the hyperplane SVM minimizes a quadratic optimization problem as follows ... 

In classification using linear procedures, the class borders are described as hyperplanes in an n-dimensional data space. These hyperplanes are obtedned either as in Subsection 6.1.1 by classification via regression, or by linear discriminant analysis (LDA). Both methods are described in detail in Chapter 4 of [117]. In the present work we will mostly use binary classification (see Section 7.6 and Subsection 8.5.2). 

Originally, SVMs were implemented to cope with two-class problems and, thus, their mathematical development considers two classes whose members are labelled as +1 and -1 (for instance, the circles in Figures 6.9 and 6.10 may be represented by +1 and the squares by -1). Let us depict how they work for classification before proceeding with regression. The simplest situation is given in Figure 6.10a. There, the two classes (+1 and -1, circles and squares) are obviously separable (this is termed the linear separable problem) and the solution is trivial. In fact, you can think of any line (hyperplane) situated between the two dashed lines as a useful one. However most of us would (unconsciously) visualise the eontinuous one as the best one, just because it is far enough from each class. That conclusion, which our brains reached... 

---