Optimizing the SVM model

Finding an SVM model with good prediction statistics is a trial-and-error task. The objective is to maximize the predictions statistics while keeping the model simple in terms of number of input descriptors, number of support vectors, patterns used for training, and kernel complexity. In this section, we present an overview of the techniques used in SVM model optimization. 

Selecting relevant input parameters is both important and difficult for any machine learning method. For example, in QSAR, one can compute thousands of structural descriptors with software like CODESSA or Dragon, or with various molecular field methods. Many procedures have been developed in QSAR to identify a set of structural descriptors that retain the important characteristics of the chemical compounds. These methods can be extended to SVM models. Another source of inspiration is represented by the algorithms proposed in the machine learning literature, which can be readily applied to cheminformatics problems. We present here several literature pointers for algorithms on descriptor selection. 

---

The SVM model was developed by using the LIBSVM software version 2.86 [50] with the RBF kernel function. The grid-search approach was adopted to select the optimal parameters C and y using the standard 5-fold cross validation within the training set. The optimal C and y values for the resulting SVM model were 128.0 and 0.03125, respectively, with the 5-fold cross validation training ER of 6.98 %. 

Aptula et al. used multiple linear regression to investigate the toxicity of 200 phenols to the ciliated protozoan Tetrahymena pyriformis Using their MLR model, they then predicted the toxicity of another 50 phenols. Here we present a comparative study for the entire set of 250 phenols, using multiple linear regression, artificial neural networks, and SVM regression methods. Before computing the SVM model, the input vectors were scaled to zero mean and unit variance. The prediction power of the QSAR models was tested with complete cross-validation leave-5%-out (L5%0), leave-10%-out (L10%O), leave-20%-out (L20%O), and leave-25%-out (L25%0). The capacity parameter C was optimized for each SVM model. 

NeuroSolutions NeuroSolutions (http //www.nd.com/) is a powerful commercial neural network modeling software that provides an icon-based graphical user interface and intuitive wizards to enable users to build and train neural networks easily [78], It has a large selection of neural network architectures, which includes FFBPNN, GRNN, PNN, and SVM. A genetic algorithm is also provided to automatically optimize the settings of the neural networks. 

These transformations are executed by using so-called kernel functions. The kernel functions can be both linear and nonlinear in nature. The most commonly used kernel function is of the latter type and called the radial basis function (RBF). There are a number of parameters, for example, cost functions and various kernel settings, within the SVM applications that will affect the statistical quality of the derived SVM models. Optimization of those variables may prove to be productive in deriving models with improved performance [97]. The original SVM protocol was designed to separate two classes but has later been extended to also handle multiple classes and continuous data [80]. 

The SVM and RF models for identifying the potential FXa inhibitors and noninhibitors were developed by using the training set of 423 FXa inhibitors and 494 noninhibitors. The models were optimized by using the equivalent cross-validation or OOB estimate strategy within the training set in the process of model development. 

In GA-SVM, the quality of SVM for regression depends on several parameters namely, kernel type k, which determines the sample distribution in the mapping space, and its corresponding parameter o, capacity parameter C, and s-insensitive loss function. The three parameters were optimized in a systematic grid search-way and the final optimal model was determined. Six general statistical parameters were selected to evaluate the prediction ability of the constructed model. These parameters are root mean square error of prediction... 

The first type of task is to solve the formabllity problems , i.e., to find some mathematical model or criterion for the stability of some unknown molecules or chemical substances. The second type of task is the property prediction , i.e., to make mathematical models for the structure-property relationships and use these models to predict the property of new materials (or the inverse problem to search the unknown new materials with some pre-assigned property). The third type of task is to solve the optimization problems , i.e., to find the conditions for optimizing some properties of certain materials. The fourth type of task is to solve the problem of control , i.e., is to find the mathematical model to control some index of materials within a desired range. And the fifth type of task is to find the multivariate relationships between the conditions of preparation and the properties of materials. Different SVM techniques should be used for these different purposes. In the following sections, we will use different examples of materials design tasks to demonstrate various strategies of solution by SVM technique. 

The use of nonlinear kernels provides the SVM with the ability to model complicated separation hyperplanes in this example. However, because there is no theoretical tool to predict which kernel will give the best results for a given dataset, experimenting with different kernels is the only way to identify the best function. An alternative solution to discriminate the patterns from Table 1 is offered by a degree 3 polynomial kernel (Figure 5a) that has seven support vectors, namely three from class +1 and four from class —1. The separation hyperplane becomes even more convoluted when a degree 10 polynomial kernel is used (Figure 5b). It is clear that this SVM model, with 10 support vectors (4 from class +1 and 6 from class —1), is not an optimal model for the dataset from Table 1. 

The optimization problem from Eq. [20] represents the minimization of a quadratic function under linear constraints (quadratic programming), a problem studied extensively in optimization theory. Details on quadratic programming can be found in almost any textbook on numerical optimization, and efficient implementations exist in many software libraries. However, Eq. [20] does not represent the actual optimization problem that is solved to determine the OSH. Based on the use of a Lagrange function, Eq. [20] is transformed into its dual formulation. All SVM models (linear and nonlinear, classification and regression) are solved for the dual formulation, which has important advantages over the primal formulation (Eq. [20]). The dual problem can be easily generalized to linearly nonseparable learning data and to nonlinear support vector machines. 

The parameter a controls the shape of the separating hyperplane, as one can see from the two SVM models in Figure 36, both obtained with a Gaussian RBF kernel (a, a = l b, a=10). The number of support vectors increases from 6 to 17, showing that the second setting does not generalize well. In practical applications, the parameter a should be optimized with a suitable cross-validation procedure. 

It is sometimes claimed that SVMs are better than artificial neural networks. This assertion is because SVMs have a unique solution, whereas artificial neural networks can become stuck in local minima and because the optimum number of hidden neurons of ANN requires time-consuming calculations. Indeed, it is true that multilayer feed-forward neural networks can offer models that represent local minima, but they also give constantly good solutions (although suboptimal), which is not the case with SVM (see examples in this section). Undeniably, for a given kernel and set of parameters, the SVM solution is unique. But, an infinite combination of kernels and SVM parameters exist, resulting in an infinite set of unique SVM models. The unique SVM solution therefore brings little comfort to the researcher because the theory cannot foresee which kernel and set of parameters are optimal for a... 

Journal of Machine Learning Research is an open-access journal that contains many papers on SVM, including new algorithms and SVM model optimization. All papers can be downloaded and printed for free. In the current context of widespread progress toward an open access to scientific publications, this journal has a remarkable story and is an undisputed success. 

FIGURE 5.28 Comparison of the test errors for the glass data using different classification methods. One hundred replications of the evaluation procedure (described in the text) are performed for the optimal parameter choices (if the method depends on the choice of a parameter). The methods are LDA, LR, Gaussian mixture models (Mix), fc-NN classification, classification trees (Tree), ANN, and SVMs. 

Like ANNs, SVMs can be useful in cases where the x-y relationships are highly nonlinear and poorly nnderstood. There are several optimization parameters that need to be optimized, including the severity of the cost penalty , the threshold fit error, and the nature of the nonlinear kernel. However, if one takes care to optimize these parameters by cross-validation (Section 12.4.3) or similar methods, the susceptibility to overfitting is not as great as for ANNs. Furthermore, the deployment of SVMs is relatively simpler than for other nonlinear modeling alternatives (such as local regression, ANNs, nonlinear variants of PLS) because the model can be expressed completely in terms of a relatively low number of support vectors. More details regarding SVMs can be obtained from several references [70-74]. 

Shape optimization of microfluidic structures is a challenging problem, where MOR is strongly desired to reduce the computational complexity during iterations. Utilization of reduced order models for shape optimization in microfluidic devices has been explored recently. Antil et al. [15] combined the POD and the balanced truncation MOR methods for shape optimization of capillary barriers in a network of microchannels. Ooi [9] developed a computationally efficient SVM surrogate model for optimization of a bioMEM microfluidic weir to enhance particle trapping. 

The classification performance of SVM is greatly decided by the model s parameters. Different problems have specific optimal parameters that have to be determined. Consequently, the selection of the model s parameters is an important requirement in SVM, because the kernel parameters and the cost parameter decide the performance of SVM in classification. 

Table 6.3 presents the predictions for the eight oils. It is observed that the predictions are good, especially for the two CRMs, and that there were no extreme errors. The RMSEP was 0.40pgCul while the RMSEC resulted in 0.38pgCul, which is in excellent agreement. It is seen that these results are similar to those from ANN and PLS (see Section 6.10), which reinforces the ability of SVM to produce useful predictive models. However in this particular case it did not outperform the optimal PLS model. 

Although SVM is indeed a useful tool of modeling for the industrial optimization and fault analysis, its results will be more believable if we can find the physical meaning of this mathematical model by some... 

---