Linearly separable classification problems

In certain classification problems, a linear separation of two classes by only one decision plane is impossible. Figure 27 shows a two-modal class (+) consisting of two distinct clusters (subclasses). Evidently, this class should be represented by two prototypes (Wg/ w ) and a minimum distance classifier would be successful. In this way, the pattern space is partitioned by several decision planes (piecewise-linear separation). Classification of an unknown pattern requires the calculation of the scalar products with all weight vectors (prototype vectors). The unknown is assigned to the class with the largest scalar product (Chapter 2.1.5.). In the same way, a mu Iticategory classification is possible C89, 3963. 

Although perceptrons are quite useful for a wide variety of classification problems, their usefulness is limited to problems that are linearly separable problems in which a line, plane or hyperplane can effect the desired dichotomy. As an example of a non-linearly separable problem, see Figure 3.4. This is just Figure 3.1 with an extra point added (measure 1 =. 8 and measure 2 =. 9) but this point makes it inpossible to find a line that can separate the depressed from non-depressed. This is no longer a linearly separable problem, and a simple perceptron will not be able to find a solution. However, note that a simple curve can effectively separate the two groups. Multilayer perceptrons, discussed in the next section, can be used for classification, even in the presence of nonlinearities. 

To establish a correlation between the concentrations of different kinds of nucleosides in a complex metabolic system and normal or abnormal states of human bodies, computer-aided pattern recognition methods are required (15, 16). Different kinds of pattern recognition methods based on multivariate data analysis such as principal component analysis (PCA) (8), partial least squares (16), stepwise discriminant analysis, and canonical discriminant analysis (10, 11) have been reported. Linear discriminant analysis (17, 18) and cluster analysis were also investigated (19,20). Artificial neural network (ANN) is a branch of chemometrics that resolves regression or classification problems. The applications of ANN in separation science and chemistry have been reported widely (21-23). For pattern recognition analysis in clinical study, ANN was also proven to be a promising method (8). 

However, in real life, many nonseparable (linear or nonlinear) classification problems occur, which practically means that distributions between two classes are overlapping. This implies that misclassihcations should be tolerated. Therefore, a set of slack variables > 0) is introduced in the margin minimization approach used for the linearly separable case, allowing some samples inside the margin. For this purpose. Equation 13.11 is replaced by Equation 13.12. 

NonLinear SVM Classifiers. For nonlinear classification problems, the SVM basic idea is to project samples of the data set, initially defined in dimensional space, into another space 91 with a higher dimension (d < e), where samples then are separated by a linear separation (Fig. 13.16) (34). 

Linear/Non-Linear separation boundaries Here our attention is focused on the mathematical form of the decision boundary. Typical non-linear classification techniques are based on ANN and SVM, specially devoted to apply for classification problems of non-linear nature. It is remarkable that CAIMAN method seems not to suffer of nonlinear class separability problems. 

In a binary classification problem one has to distinguish only between two mutually exclusive classes (e.g. class 1 contains compounds with a certain chemical substructure and class 2 contains all other compounds). If the two classes form well separated clusters in the pattern space it is often possible to find a plane (decision plane) which separates the classes completely (Figure 3). In this case the data are called to be "linearly separable". The calculation of a suitable decision plane is often called the "training". 

A "committee machine uses several different classifiers for the same classification problem in a parallel manner. The classifiers may stem e.g. from different training processes with the same training set. The results (scalar products) are summarized to give the final classification. Threshold logical units are usually applied and the majority of votes determines the class membership. Correct classification for all members of the training set may be obtained by a committee machine even if the data set is not Linearly separable. Unfortunately, no exhaustive theory about the training of several parallel classifiers exists. 

One neuron can divide only linearly separated patterns. To select just one region in n-dimensional input space, more than + 1 neurons should be used. If more input clusters are to be selected, then the number of neurons in the input (hidden) layer should be properly multiplied. If the number of neurons in the input (hidden) layer is not limited, then all classification problems can be solved using the three-layer network. An example of such a neural network, classifying three clusters in the two-dimensional space, is shown in Fig. 19.19. Neurons in the first hidden layer create the separation Hnes between input clusters. Neurons in the second hidden layer perform the AND operation, as shown in Fig. 19.13(b). Output neurons perform the OR operation as shown in Fig. 19.13(a), for each category. The linear separation property of neurons makes some problems specially difficult for neural networks, such as exclusive OR, parity computation for several bits, or to separate patterns laying on two neighboring spirals. 

When complex classification problems arise (e.g. the different classes of sample overlap or distribute in a non-linearly separable shape) one can have resource either to ANNs (which implies that one must be aware of their stochastic nature and of the optimisation tasks that will be required) or increase the dimensionality of the data (i.e. the variables that describe the samples) in the hope that this will allow a better separation of the classes. How can this be possible Let us consider a trivial example where the samples were drawn/ projected into a two-dimensional subspace (e.g. two original variables, two principal components, etc.) and the groups could not be separated by a linear border (in the straight line sense. Figure 6.9a). However, if three variables were considered instead, the groups would be separated easily (Figure 6.9b). How to get this(these) additional dimension(s) is what SVM addresses. 

Originally, SVMs were implemented to cope with two-class problems and, thus, their mathematical development considers two classes whose members are labelled as +1 and -1 (for instance, the circles in Figures 6.9 and 6.10 may be represented by +1 and the squares by -1). Let us depict how they work for classification before proceeding with regression. The simplest situation is given in Figure 6.10a. There, the two classes (+1 and -1, circles and squares) are obviously separable (this is termed the linear separable problem) and the solution is trivial. In fact, you can think of any line (hyperplane) situated between the two dashed lines as a useful one. However most of us would (unconsciously) visualise the eontinuous one as the best one, just because it is far enough from each class. That conclusion, which our brains reached... 

The separation surface may be nonlinear in many classification problems, but support vector machines can be extended to handle nonlinear separation surfaces by using feature functions < )(x). The SVM extension to nonlinear datasets is based on mapping the input variables into a feature space of a higher dimension (a Hilbert space of finite or infinite dimension) and then performing a linear classification in that higher dimensional space. For example, consider the set of nonlinearly separable patterns in Figure 28, left. It is... 

The mathematical formulation of the hard margin nonlinear SVM classification is similar to that presented for the SVM classification for linearly separable datasets, only now input patterns x are replaced with feature functions, X —> 4)(x), and the dot product for two feature functions () (x,) i xj) is replaced with a kernel function K xi, Xj), Eq. [64]. Analogously with Eq. [28], the dual problem is... 

Sometimes an equation out of this classification can be altered to fit by change of variable. The equations with separable variables are solved with a table of integrals or by numerical means. Higher order linear equations with constant coefficients are solvable with the aid of Laplace Transforms. Some complex equations may be solvable by series expansions or in terms of higher functions, for instance the Bessel equation encountered in problem P7.02.07, or the equations of problem P2.02.17. In most cases a numerical solution Is possible. 

Support vector machine (SVM) is originally a binary supervised classification algorithm, introduced by Vapnik and his co-workers [13, 32], based on statistical learning theory. Instead of traditional empirical risk minimization (ERM), as performed by artificial neural network, SVM algorithm is based on the structural risk minimization (SRM) principle. In its simplest form, linear SVM for a two class problem finds an optimal hyperplane that maximizes the separation between the two classes. The optimal separating hyperplane can be obtained by solving the following quadratic optimization problem ... 

The electrochemical behaviour of a PFC is of prime interest for our purposes. When separating the total current into electronic and ionic parts, it is usually assumed that the ratio of these particular currents is equal to the ratio of electronic and ionic conductivities. We shall use this simple model of parallel connection of electronic and ionic resistors for qualitative classification of EFSs (see below). However, as it has been shown more recently, such model is only a rough approximation— the ratio does not remain constant with change of the applied current because of non-linear dependence of the partial conductivities on current [37, 38]. This problem will be discussed in Chap. 4. 

The methods for solving an optimization task depend on the problem classification. Since the maximum of a function / is the minimum of the function —/, it suffices to deal with minimization. The optimization problem is classified according to the type of independent variables involved (real, integer, mixed), the number of variables (one, few, many), the functional characteristics (linear, least squares, nonlinear, nondifferentiable, separable, etc.), and the problem. statement (unconstrained, subject to equality constraints, subject to simple bounds, linearly constrained, nonlinearly constrained, etc.). For each category, suitable algorithms exist that exploit the problem s structure and formulation. 

---