Linearly separable problem

A natural question to ask is whether the basic model can be modified in some way that would enable it to correctly learn the XOR function or, more generally, any other non-linearly-separable problem. The answer is a qualified yes in principle, all that needs to be done is to add more layers between what we have called the A-units and R-units. Doing so effectively generates more separation lines, which when combined can successfully separate out the desired regions of the plane. However, while Rosenblatt himself considered such variants, at the time of his original analysis (and for quite a few years after that see below) no appropriate learning rule was known. 

The field of artificial neural networks is a new and rapidly growing field and, as such, is susceptible to problems with naming conventions. In this book, a perceptron is defined as a two-layer network of simple artificial neurons of the type described in Chapter 2. The term perceptron is sometimes used in the literature to refer to the artificial neurons themselves. Perceptrons have been around for decades (McCulloch Pitts, 1943) and were the basis of much theoretical and practical work, especially in the 1960s. Rosenblatt coined the term perceptron (Rosenblatt, 1958). Unfortunately little work was done with perceptrons for quite some time after it was realized that they could be used for only a restricted range of linearly separable problems (Minsky Papert, 1969). 

Although perceptrons are quite useful for a wide variety of classification problems, their usefulness is limited to problems that are linearly separable problems in which a line, plane or hyperplane can effect the desired dichotomy. As an example of a non-linearly separable problem, see Figure 3.4. This is just Figure 3.1 with an extra point added (measure 1 =. 8 and measure 2 =. 9) but this point makes it inpossible to find a line that can separate the depressed from non-depressed. This is no longer a linearly separable problem, and a simple perceptron will not be able to find a solution. However, note that a simple curve can effectively separate the two groups. Multilayer perceptrons, discussed in the next section, can be used for classification, even in the presence of nonlinearities. 

This is called the perception learning rule and it has been proven to converge to a solution, for linearly separable problems, in a finite number of iterations. The weight adjustment rule can be restated as... 

One-layer neural networks are relatively easy to train, but these networks can solve only linearly separated problems. One possible solution for nonlinear problems presented by Nilsson (1965) and elaborated by Pao (1989) using the functional link network is shown in Fig. 19.23. Using nonlinear terms with initially determined functions, the actualnum-ber of inputs supplied to the one-layer neural network is increased. In the simplest case, nonlinear elements are higher order terms of input patterns. 

Originally, SVMs were implemented to cope with two-class problems and, thus, their mathematical development considers two classes whose members are labelled as +1 and -1 (for instance, the circles in Figures 6.9 and 6.10 may be represented by +1 and the squares by -1). Let us depict how they work for classification before proceeding with regression. The simplest situation is given in Figure 6.10a. There, the two classes (+1 and -1, circles and squares) are obviously separable (this is termed the linear separable problem) and the solution is trivial. In fact, you can think of any line (hyperplane) situated between the two dashed lines as a useful one. However most of us would (unconsciously) visualise the eontinuous one as the best one, just because it is far enough from each class. That conclusion, which our brains reached... 

By this way, Aizermann successfully converted a linearly nonseparable problem to a very simple linearly separable problem to take the difference of electric fields at every point. It is easy to understand that potential functions other than Coulomb potential function are also applicable in this method. Aizermann also suggested the following function for the evaluation of the field strength around every sample point instead of Coulomb potential function ... 

To see what we mean by linear separability, we consider a simple problem that is not linearly separable the so-called XOR problem - to teach a neural net the exclusive-OR function (table 10.2). 

Furthermore, the pattern structures in a representation space formed from raw input data are not necessarily linearly separable. A central issue, then, is feature extraction to transform the representation of observable features into some new representation in which the pattern classes are linearly separable. Since many practical problems are not linearly separable (Minsky and Papert, 1969), use of linear discriminant methods is especially dependent on feature extraction. 

Problems that are not linearly separable are easy to generate. The classic example is the XOR function, Figure 2.15, in which Y(.t j, x2) equals one if just one of Xj and x2 equals 1, but is zero if x1 and x2 have the same value. In this... 

Although some problems in more than two dimensions are linearly separable (in three dimensions, the requirement for linear separability is that the points are separated by a single plane, Figure 2.17), almost all problems of scientific interest are not linearly separable and, therefore, cannot be solved by a one-node network thus more sophistication is needed. The necessary additional power in the network is gained by making two enhancements (1) the number of nodes is increased and (2) each node is permitted to use a more flexible activation function. 

In the first step, the adsorption isotherms of the compounds should be determined under non-linear chromatographic conditions, which can be done in several ways [11]. Afterwards, models should be implemented and used to simulate the chromatographic behavior and to find the optimum system parameters for a given separation problem. Different approaches for finding the optimum parameter are described in the literature [12-16] mainly for adsorption and ion exchange chromatography. 

One way of linearizing the problem is to use the method of least squares in an iterative linear differential correction technique (McCalla, 1967). This approach has been used by Taylor et al. (1980) to solve the problem of modeling two-dimensional electrophoresis gel separations of protein mixtures. One may also treat the components—in the present case spectral lines—one at a time, approximating each by a linear least-squares fit. Once fitted, a component may be subtracted from the data, the next component fitted, and so forth. To refine the overall fit, individual components may be added separately back to the data, refitted, and again removed. This approach is the basis of the CLEAN algorithm that is employed to remove antenna-pattern sidelobes in radio-astronomy imagery (Hogbom, 1974) and is also the basis of a method that may be used to deal with other two-dimensional problems (Lutin et al., 1978 Jansson et al, 1983). 

Occasionally a problem will be elevated to the status of trouble and interfere with the advantageous use of an integrator. These troublesome situations can be grouped into those that occur suddenly in the middle of what was thought to be a routine day s work and those that are encountered while working on a new separation problem. Alternatively, problems could be grouped in a more operational (even if overlapping) way accuracy, reproducibility, linearity, and "noise." Both viewpoints will be represented in this discussion. 

CV and valence calculations leads to almost insuperable linear dependence problems in the resulting basis set [97], We should note that these problems axe perhaps at their worst for the first row. For heavier elements the separation between the valence shell and the core (meaning the next innermost shell here) is not as great, so the disparities in correlating orbital occupation number are less. Suitable ANO sets can often be obtained by correlating all the desired electrons [48, 98]. 

Therefore, the assumption in v2-GBD holds true if separability and linearity hold which covers also the case of linear 0-1 y variables. This way under conditions Cl, C2, C3 the v2-GBD determined the global solution for separability in x andy and linearity iny problems. 

Yet another unsolved problem is the separation of two voices that contain closely spaced harmonics or overlapping harmonic and aharmonic components. The time-varying nature of sine-wave parameters, as well as the synchrony of movement of these parameters within a voice [Bregman, 1990], may provide the key to solving this more complex separation problem. Section 6.3 revealed, for example, the limitation of assuming constant sine-wave amplitude and frequency in analysis and as a consequence proposed a generalization of Equation (9.75) based on linear sine-wave amplitude and frequency trajectories as a means to aid separation of sine waves with closely-spaced frequencies. 

This linear evolution model may be even more important in the context of the signal separation problem described in section 6. 

For example, if a brominated polystyrene is treated with K2Se and hydrolyzed we obtain — starting from linear polystyrene — a polymer with about 30% Se, whilst the crosslinked polymer — even with the low amount of 0.5% DVB — gives about 20% Se under identical conditions (14). Separation problems are not always combined with linear polymers. So N-Chloro-nylons are readily soluble in many solvents and can be used for oxidation and chlorination (15). Unsubstituted nylon regenerated during this reaction is insoluble in most solvents therefore it precipitates during the reaction. 

---