Misclassification probability

To judge the performance of the discriminant functions and the classification procedure in respect of future samples one can calculate misclassification probabilities or error rates. But these probabilities cannot be calculated in general because they depend on the unknown density functions of the classes. Instead we can usually utilize a measure called apparent error rate. The value of this quantity is easily calculated from the classification or confusion matrix based on the samples of the training set. For example with two classes we can have the following matrix ... 

Misclassification Probabilities for RQDR and CQDR Applied to the Fruit Data Set... 

Friedman [12] introduced a Bayesian approach the Bayes equation is given in Chapter 16. In the present context, a Bayesian approach can be described as finding a classification rule that minimizes the risk of misclassification, given the prior probabilities of belonging to a given class. These prior probabilities are estimated from the fraction of each class in the pooled sample ... 

Next, Bayesian probabilities were computed and these produced a clear U-shaped pattern. The authors assigned cases with the probability of. 90 or above to the taxon, and found that 35 individuals were identified as taxon members by these rules, that is, the prevalence of this taxon was 3.3%. Note that the. 50 cutoff is generally associated with the lowest overall rate of misclassification, but the. 90 cutoff may be preferable under certain conditions. In epidemiological studies, however, accuracy—a low rate of misclassification—is the primary consideration. In fact, the actual prevalence of the taxon in the Waller and Ross study appears to be about 5% based on the non-Bayesian base rate estimates. Thus, it appears that the use of a conservative cutoff in this study may produce somewhat misleading findings. 

This preliminary assessment will need to be updated as and when further information becomes available. It should favor sensitivity over specificity so that a borderline possible-probable case is classified as probable rather than possible to make certain that the case is not lost when at a later stage the probable cases are picked out as a signal. A full assessment when all the information is available can then rectify any misclassifications. 

FIGURE 5.3 An optimal discriminant mle is obtained in the left picture, because the group covariances are equal and an adjustment is made for different prior probabilities. The linear mle shown in the right picture is not optimal—in terms of a minimum probability of misclassification—because of the different covariance matrices. 

We come back to the problem of selecting the optimum dimensions a, ..., ak of the PCA models. This can be done with an appropriate evaluation technique like CV, and the goal is to minimize the total probability of misclassification. The latter can be obtained from the evaluation set, by computing the percentage of misclassified objects in each group, multiplied by the relative group size, and summarized over all groups. 

One has to be careful with the use of the misclassification error as a performance measure. For example, assume a classification problem with two groups with prior probabilities pi = 0.9 and p2 = 0.1, where the available data also reflect the prior probabilities, i.e., nx k, npi and n2 np2. A stupid classification rule that assigns all the objects to the first (more frequent) group would have a misclassification error of about only 10%. Thus it can be more advisable to additionally report the misclassification rates per group, which in this case are 0% for the first group but 100% for the second group which clearly indicates that such a classifier is useless. 

FIGURE 5.26 ANNs applied to the glass data with six glass types. The optimal parameter choices are (probably) 20 hidden units and a weight decay of 0.2. The plots show the misclassification errors by fixing one of these parameters. Since the result is not unique, we obtain two answers for the test error 0.41 in the left plot and 0.37 in the right plot. 

Fig. 8.4. A priori probability density functions (pdfs) of two classes and their resulting assignments after weighting them with their corresponding class probabilities. For a given feature value along the x-axis, the higher of the corresponding y-axis values decides the class for that value. Two types of errors are possible with this scheme, namely the misclassification of class 1 as class 2 (horizontal stripes) and the misclassification of class 2 as class 1 (vertical stripes/shaded). The colored regions indicate the relative probabilities of such errors. Errors can be explicitly understood and the contribution of each feature to classification can be quantitatively measured...

Fig. 9-9 demonstrates the results of MVDA for the three investigated territories in the plane of the computed two discriminant functions. The separation line corresponds to the limits of discrimination for the highest probability. The results prove that good separation of the three territories with a similar geological background is possible by means of discriminant analysis. The misclassification rate amounts to 13.0%. The scattering radii of the 5% risk of error of the multivariate analysis of variance overlap considerably. They demonstrate also that the differences in the multivariate data structure of the three territories are only small. 

This conditional expected cost of misclassifying an event belonging to tti occurs with prior probability pi (the probability of tti). The conditional overall expected cost of misclassification is computed by multiplying each ECM tti) with its prior probability and summing over all classes... 

The generalized variance Si, the prior probability pi and the Mahalanobis distance contribute to the quadratic score dj (x). Using the discriminant scores, the minimum total probability of misclassification rule for Normal populations and unequal covariance matrices becomes [126] ... 

When the time correlated HMM is introduced and the probabilities are re-calculated, the results show a significant improvement (Figure 7.9). The misclassification rate is reduced to 3.9%. 

The correct classification rate (CCR) or misclassification rate (MCR) are perhaps the most favoured assessment criteria in discriminant analysis. Their widespread popularity is obviously due to their ease in interpretation and implementation. Other assessment criteria are based on probability measures. Unlike correct classification rates which provide a discrete measure of assignment accuracy, probability based criteria provide a more continuous measure and reflect the degree of certainty with which assignments have been made. In this chapter we present results in terms of correct classification rates, for their ease in interpretation, but use a probability based criterion function in the construction of the filter coefficients (see Section 2.3). Whilst we speak of correct classification rates, misclassification rates (MCR == 1 - CCR) would equally suffice. The correct classification rate is typically formulated as the ratio of correctly classified objects with the total... 

Probably the reason for this misclassification lies in the fact that compound 9 may not be "weU grouped" into one of the two classes. In fact when you analyze Fig. 2 you note that 9 is the compound classified as more active that is closer to compounds classified as less active. 

---