Decision boundaries

A network that is too large may require a large number of training patterns in order to avoid memorization and training time, while one that is too small may not train to an acceptable tolerance. Cybenko [30] has shown that one hidden layer with homogenous sigmoidal output functions is sufficient to form an arbitrary close approximation to any decisions boundaries for the outputs. They are also shown to be sufficient for any continuous nonlinear mappings. In practice, one hidden layer was found to be sufficient to solve most problems for the cases considered in this chapter. If discontinuities in the approximated functions are encountered, then more than one hidden layer is necessary. 

Thus, they share exactly the same solution (H) and performance criteria (y ) spaces. Furthermore, since their role is simply to estimate y for a given X, no search procedures S are attached to classical pattern recognition techniques. Consequently, the only element that dilfers from one classification procedure to another is the particular mapping procedure / that is used to estimate y(x) and/ or ply = j x). The available set of (x, y) data records is used to build /, either through the construction of approximations to the decision boundaries that separate zones in the decision space leading to different y values (Fig. 2a), or through the construction of approximations to the conditional probability functions, piy =j ). 

Since the denominator in Equation 5.1 is the same for each group, we can directly compare the posterior probabilities P(j x) for all groups. Observation x will be assigned to that group for which the posterior probability is the largest. Thus the decision boundary between two classes h and l is given by objects jt for which the posterior probabilities are equal, i.e., P(h x) P(l x). 

FIGURE 5.2 Visualization of the Bayesian decision mle in the univariate case, where the prior probabilities of the three groups are equal. The dashed lines are at the decision boundaries between groups 1 and 2 (x12) and between groups 2 and 3 ( 23). 

FIGURE 5.13 k-NN classification for two groups of two-dimensional data. The training data are shown with the symbol corresponding to the group membership. Any new data point would be classified according to the presented decision boundaries, where k = 1 in the left plot, and k= 15 in the right plot has been used. 

An algorithm for computing the decision boundary thus requires the choice of the kernel function frequently chosen are radial basis functions (RBFs). A further input parameter is the priority of the size constraint for used in the optimization problem (Equation 5.38). This constraint is controlled by a parameter that is often denoted by y. A large value of y forces the size of to be small, which can lead to an overht and to a wiggly... 

In the following example, the effect of these parameter choices will be demonstrated. We use the same example as in Section 5.5 with two overlapping groups. Figure 5.20 shows the resulting decision boundaries for different kernel functions... 

P(N0T OK) while for workplaces lying to the right of the unbiased decision contour, P(N0T OK) > P(0K). The second is that since our decision criteria have been derived to provide high confidence in the decisions which are made, it frequently happens that the most likely outcome from a trial involving one sample taken from a workplace lying close to the unbiased decision contour is that a decision cannot be made with sufficient confidence. Because of this, it is important to define additional decision boundaries. 

Bayes classifier [89] is an elementary probabilistic method to obtain decision boundaries based on using Bayes theorem. A decision rule based on probabilities is to assign K to class Cj if the probability of class Cj given the observation K, p Cj K) is greatest over all classes, i.e.,... 

The system of linear equations originating from the difference equation (2.308) has to be supplemented by the difference equations for the points around the boundaries where the decisive boundary conditions are taken into account. As a simplification we will assume that the boundaries run parallel to the x- and y-directions. Curved boundaries can be replaced by a series of straight lines parallel to the x- and y-axes. However a sufficient degree of accuracy can only be reached in this case by having a very small mesh size Ax. If the boundaries are coordinate lines of a polar coordinate system (r, differential equation and its boundary conditions are formulated in polar coordinates and then the corresponding finite difference equations are derived. 

Figure 15-6 Quadratic discriminant analysis removes the constraint of a straight and/or flat plane and allows more molding of the decision boundary than is possible with linear discriminant analysis.

Figure 15-8 The decision boundary derived using a neural net, fit from different data in Figure 15-7, no longer provides optimal separation, although in this case the separation by chance is still good.

The SVM method, introduced by Vapnik (32) in 1995, is applicable for both classification and regression problems. In case of classification, SVM are used to determine a boundary, a hyperplane, which separates classes independently of the probabilistic distributions of samples in the data set and maximizes the distance between these classes. The decision boundary is determined calculating a function f(x) = y(x) (32-34). The technique is gaining popularity fast in... 

If the detection process is publicly known, an attacker can perturb the public signal s in such a way that the attacked signal r exactly lies on the decision boundary between different quantizer points. For binary dither modulation these points are depicted by the short lines in Fig. 3 (left). After such an attack, the decoder can only randomly guess whether the received signal sample was originally quantized by Qo or Qi. Thus, the watermark information is completely lost. Note that no chaimel coding can help to recover information from the such attacked signal. 

A eorrect classifieation rate is a discrete measure whose calculation is based upon which side of a decision boundary the observations lie. It does not reflect how close or how far away the observations lie from the decision boundary and hence how clear the assignments are made. It is still possible to have a high classification rate, where many assignments have lied close to... 

A fundamental difficulty about this method of finding decision vectors is that the learning machine orients itself on the extreme points. The final position of the decision boundary is largely determined by spectra which are most atypical of their dass. Therefore useful results can only be expected from a carefully selected, self-consistent and error-free training set. 

Linear/Non-Linear separation boundaries Here our attention is focused on the mathematical form of the decision boundary. Typical non-linear classification techniques are based on ANN and SVM, specially devoted to apply for classification problems of non-linear nature. It is remarkable that CAIMAN method seems not to suffer of nonlinear class separability problems. 

To find a decision boundary that separates the two groups, the data vectors have to be augmented by adding a ( + 1) component equal to 1.0. This ensures that the boundary for separating the classes passes through the origin. If more than two categories... 

The boundary that separates the two categories is found iteratively by adjusting the elements of a weight vector, w, which is normal to the boundary, such that the dot product of w and any vector of the full circles is positive, while that of w and the empty circles is negative (Figure 5.25). The decision boundary s is expressed by... 

A decision boundary separates two or more groups of data. 

A more formal way of finding a decision boundary between different classes is based on linear discriminant analysis (LDA) as introduced by Fisher and Mahalanobis. The boundary or hyperplane is calculated such that the variance between the classes is maximized and the variance within the individual classes is minimized. There are several ways to arrive at the decision hyperplanes. In... 

---