Misclassification rate

The above example demonstrates that the choice of k is crucial. As mentioned above, k should be selected such that the smallest misclassification rate for the test data occurs. If no test data are available, an appropriate resampling procedure (CV or bootstrap) has to be used. Figure 5.14 shows for the example with three overlapping groups, used above in Figure 5.12, how k can be selected. Since we have independent training and test data available, and since their group membership is known, we... 

FIGURE 5.14 k-NN classification for the training and test data used in Figure 5.12. The left plot shows the misclassification rate for the test data with varying value of k for k-NN classification, and the right plot presents the result for k = 25. The misclassified objects are shown by dark symbols. 

One has to be careful with the use of the misclassification error as a performance measure. For example, assume a classification problem with two groups with prior probabilities pi = 0.9 and p2 = 0.1, where the available data also reflect the prior probabilities, i.e., nx k, npi and n2 np2. A stupid classification rule that assigns all the objects to the first (more frequent) group would have a misclassification error of about only 10%. Thus it can be more advisable to additionally report the misclassification rates per group, which in this case are 0% for the first group but 100% for the second group which clearly indicates that such a classifier is useless. 

Comparison of the success of different classification methods requires a realistic estimation of performance measures for classification, like misclassification rates (% wrong) or predictive abilities (% correct) for new cases (Section 5.7)—together with an estimation of the spread of these measures. Because the number of objects with known class memberships is usually small, appropriate resampling techniques like repeated double CV or bootstrap (Section 4.2) have to be applied. A difficulty is that performance measures from regression (based on residuals) are often used in the development of classifiers but not misclassification rates. 

Simon et al. (14) also showed that cross-validating the prediction rule after selection of differentially expressed genes from the full data set does little to correct the bias of the re-substitution estimator 90.2% of simulated data sets with no true relationship between expression data and class still result in zero misclassifications. When feature selection was also re-done in each cross-validated training set, however, appropriate estimates of mis-classification error were obtained the median estimated misclassification rate was approximately 50%. 

In Tab. 5-13 we report the results of both mentioned strategies for the selection process. In both procedures the WILKS lambda varies monotonously and each set has a significant meaning. We may, therefore, stop the selection process following the misclassification rate. In the forward strategy the first zero error rate appears with the feature set Ti, Mg, Ca in step 3 (Fig. 5-25) whereas in the backward strategy the zero error rate is obtained with the remaining elements Si, Ca, Al, Mg in step 3. Now it is up to the expert to decide which feature set to retain in the future. 

Fig. 9-9 demonstrates the results of MVDA for the three investigated territories in the plane of the computed two discriminant functions. The separation line corresponds to the limits of discrimination for the highest probability. The results prove that good separation of the three territories with a similar geological background is possible by means of discriminant analysis. The misclassification rate amounts to 13.0%. The scattering radii of the 5% risk of error of the multivariate analysis of variance overlap considerably. They demonstrate also that the differences in the multivariate data structure of the three territories are only small. 

Soil samples Discriminated samples in class Misclassification rate... 

When the time correlated HMM is introduced and the probabilities are re-calculated, the results show a significant improvement (Figure 7.9). The misclassification rate is reduced to 3.9%. 

A study that integrates SVM with genetic-quasi-Newton optimization algorithms reported the application of the methodology to rayon yarn data (two classes) and wine data (three classes) with very low misclassification rates (0.1%) [156]. 

The correct classification rate (CCR) or misclassification rate (MCR) are perhaps the most favoured assessment criteria in discriminant analysis. Their widespread popularity is obviously due to their ease in interpretation and implementation. Other assessment criteria are based on probability measures. Unlike correct classification rates which provide a discrete measure of assignment accuracy, probability based criteria provide a more continuous measure and reflect the degree of certainty with which assignments have been made. In this chapter we present results in terms of correct classification rates, for their ease in interpretation, but use a probability based criterion function in the construction of the filter coefficients (see Section 2.3). Whilst we speak of correct classification rates, misclassification rates (MCR == 1 - CCR) would equally suffice. The correct classification rate is typically formulated as the ratio of correctly classified objects with the total... 

Multiobjective recursive partitioning N= 161 with classification for metabolism by CYP1A2, CYP2C9, CYP2C19, CYP2E1 and CYP3A4. Leave out 10%, misclassification rate 12.6%. Descriptors generated with ADMET Predictor software. 200... 

Note that the set of criteria for deciding on function allocation was different for the three studies. The first used only the error probabilities, the second expanded this set to include performance measures such as misclassification rate, while the third also included performance speed. The issue of flexibility was addressed in the third study by comparing the competing systems across different contrast levels and board sizes. In none of these examples was there any measure of the cost of... 

The false positive rate for this model is 10.9% and the false negative rate, 7.9%. The original CROSSBOW keys (149 in number) were employed in the development of this equation an expanded set of keys is being developed by Craig and Enslein (79), and with the new set (more than 300 in number), it is anticipated that the misclassification rate and the number of indeterminate compounds will be lowered. 

Finally we combine these single steps to demonstrate automated structure elucidation via MS for two examples (Section 8.6). Given the known misclassification rates of MS classihers, the large size of structure spaces, and the deficiencies of candidate selection, an expert system based exclusively on low resolution EI-MS cannot, at present, work sufficiently reliably for practical use in an automatic mode. The incorporation of additional information into this automated workflow increases the success rate of automated CASE via MS and this is discussed in greater detail in Chapter 9. 

Fig. 8.28. Mean misclassification rates for learning set and test set, classification by CT.

---