Preprocessing of data

Feature variables may be defined on different scales. The nominal scale characterizes qualitative equivalence, for example, male and female. The ordinal scale describes ordering or ranking. The interval scale measures distances between values of the features. The ratio scale also enables quotients between feature values to be evaluated. 

For some problems, these data are divided columnwise into dependent and independent variables, for example, for calibrating concentrations on spectra. The dependent variables are then renamed, for example, by the character y. 

A class comprises a collection of objects that have similar features. The pattern of an object is its collection of characteristic features. For multivariate data evaluation, not all objects and features are necessarily used. On the other hand, some of the available data cannot be used as they are reported. Therefore, pretreatment of data is a prerequisite for efficient multivariate data analysis. 

In the first step, the data have to be reviewed with respect to completeness. Missing data do not hinder mathematical analysis. Of course, missing data should not be replaced by zeros. Instead, the vacancies should be filled up either by the column/row mean or, in the worst case, by generating a random number in the range of the considered column/row. Features and/or objects can be removed from the data set if they are highly correlated with each other, or if they are redundant or constant. 

To eliminate a constant offset, the data can be translated along the coordinate origin. The common procedure is mean centering, where each variable, Xy, is centered by subtracting the column 

Some learning methods require variables that fulfill particular conditions, while others will merely perform better following a data preprocessing step. A listing of preprocessing methods is found e.g. in [231]. Some examples include the removal of variables with only a constant value and only retaining one independent variable where two or more independent variables agree on each observation. 

There are several preliminary linear transformations for a dependent or independent continuous variable Z with values z, i e m  

After range scaling, the values of a variable span the interval [0,1 ]  

Data preprocessed by autoscaling will assume mean = 0 and variance = 1  

Depending on the distribution of values it may make sense to perform nonlinear transformations on the data, such as n-th root or logarithm. Nonlineeirly transformed independent variables may be used as additional predictors. Further, new variables may be obtained by applying arithmetic operators to pedrs or leirger subsets of predictors Xj, j n. This is called a base extension. Quadratic base extensions are often used, where the squares X, j e n, eind products X X , k + l,k,l e neire used as predictors along with X.  

---

Partial order ranking (POR) is based on elementary methods of discrete mathematics (e.g., Hasse diagrams) — if A < B and B < C, then A < C in the ranking procedures. POR does not assume linearity or any assumptions about distribution properties such as normality. The disadvantage is that often a preprocessing of data is needed to avoid the effects of stochastic noise. Combining POR with PCA may improve its usefulness. POR can only be applied for interpolation. 

Fig. 2 Schematic representation of RADPRE anaiysis pipeiine. (a) Preprocessing of data including background correction, across-array normaiization, probe fiitering and trimming, ratio generation, and log-transformation.

And, last but by far not least we must mention a very important part of data preprocessing. It is up to a researcher to decide when to employ these techniques. Figure 4-2 displays a step-by-step preparation of a dataset. 

The molecular mechanics force helds available are AMBER95 and CFIARMM. The molecular mechanics and dynamics portion of the code is capable of performing very sophisticated calculations. This is implemented through a large number of data hies used to hold different types of information along with keywords to create, use, process, and preprocess this information. This results in having a very hexible program, but it makes the input for simple calculations unnecessarily complex. QM/MM minimization and dynamics calculations are also possible. 

Computers, often combined with transputers, are used for three main functions when connected to a mass spectrometer. The foremost requirements involve the acquisition and preprocessing of basic data and the control of the instrument s scanning operations. Additional software programs are available to manipulate the preprocessed data in a wide variety of ways depending on what is required, e.g., a mass spectrum or a total ion chromatogram. 

The choice of which of the many preprocessing methods to apply depends on the type of data involved and the context and objectives of the problem. The importance of appropriate preprocessing caimot be overemphasized improper treatment of the data at this stage may make the rest of the analysis meaningless. 

After preprocessing of a raw data matrix, one proceeds to extract the structural features from the corresponding patterns of points in the two dual spaces as is explained in Chapters 31 and 32. These features are contained in the matrices of sums of squares and cross-products, or cross-product matrices for short, which result from multiplying a matrix X (or X ) with its transpose ... 

In the following section on preprocessing of the data we will show that column-centering of X leads to an interpretation of the sums of squares and cross-products in in terms of the variances-covariances of the columns of X. Furthermore, cos djj> then becomes the coefficient of correlation between these columns. 

E. de Noord, The influence of data preprocessing on the robustness nd parsimony of multivariate calibration models. Chemom. Intell. Lab. Systems, 23 (1994) 65-70,... 

The goal of EDA is to reveal structures, peculiarities and relationships in data. So, EDA can be seen as a kind of detective work of the data analyst. As a result, methods of data preprocessing, outlier selection and statistical data analysis can be chosen. EDA is especially suitable for interactive proceeding with computers (Buja et al. [1996]). Although graphical methods cannot substitute statistical methods, they can play an essential role in the recognition of relationships. An informative example has been shown by Anscombe [1973] (see also Danzer et al. [2001], p 99) regarding bivariate relationships. 

In data analysis, data are seldom used without some preprocessing. Such preprocessing is typically concerned with the scale of data. In this regard two main scaling procedures are widely used zero-centered and autoscaling. 

Helland, I. S., Naes, T., Isaksson, T. Chemom. Intell. Lab. Syst. 29, 1995, 233-241. Related versions of the multiplicative scatter correction method for preprocessing spectroscopic data. 

MSC Multiplicative signal/scatter correction (preprocessing of NIR data)... 

Air quality simulation models for photochemical pollutants were reviewed by Johnson et al. for a new edition of Air Pollution. Some of the models developed for simulating photochemical smog were reviewed from the viewpoints of module logic and evaluation. The Los Angeles-based developments were outlined, including the format and preprocessing of emission inventory data and meteorologic data. Lumped-param-... 

The fundamental elements of deterministic models involve a combination of chemical and meteorologic input, preprocessing with data transmission, logic that describes atmospheric processes, and concentration-field output tables or displays. In addition to deterministic models, there are statistical schemes that relate precursors (or emission) to photo-chemical-oxidant concentrations. Models may be classified according to time and space scales, depending on the purposes for which th are designed. 

The movement of a fiber-optic probe and/or the hbers leading to/from a probe or cell can also induce baseline shifts in the recorded spectra. These shifts are usually minor and can be eliminated with proper preprocessing of the data (e.g. derivatives). (A detailed description of preprocessing techniques can be found in Chapter 12)... 

---