Data transformation

Furthermore, one may need to employ data transformation. For example, sometimes it might be a good idea to use the logarithms of variables instead of the variables themselves. Alternatively, one may take the square roots, or, in contrast, raise variables to the nth power. However, genuine data transformation techniques involve far more sophisticated algorithms. As examples, we shall later consider Fast Fourier Transform (FFT), Wavelet Transform and Singular Value Decomposition (SVD). 

Wavelet transformation (analysis) is considered as another and maybe even more powerful tool than FFT for data transformation in chemoinetrics, as well as in other fields. The core idea is to use a basis function ("mother wavelet") and investigate the time-scale properties of the incoming signal [8], As in the case of FFT, the Wavelet transformation coefficients can be used in subsequent modeling instead of the original data matrix (Figure 4-7). 

The method has many applications among them arc Denoising Smoothing (DS), compression, and Feature Extraction (FE), which arc powerful tools for data transformations. See the "Selected Reading" section at the end of this chapter for further details. 

It may look weird to treat the Singular Value Decomposition SVD technique as a tool for data transformation, simply because SVD is the same as PCA. However, if we recall how PCR (Principal Component Regression) works, then we are really allowed to handle SVD in the way mentioned above. Indeed, what we do with PCR is, first of all, to transform the initial data matrix X in the way described by Eqs. (10) and (11). 

Now, one may ask, what if we are going to use Feed-Forward Neural Networks with the Back-Propagation learning rule Then, obviously, SVD can be used as a data transformation technique. PCA and SVD are often used as synonyms. Below we shall use PCA in the classical context and SVD in the case when it is applied to the data matrix before training any neural network, i.e., Kohonen s Self-Organizing Maps, or Counter-Propagation Neural Networks. 

The profits from using this approach are dear. Any neural network applied as a mapping device between independent variables and responses requires more computational time and resources than PCR or PLS. Therefore, an increase in the dimensionality of the input (characteristic) vector results in a significant increase in computation time. As our observations have shown, the same is not the case with PLS. Therefore, SVD as a data transformation technique enables one to apply as many molecular descriptors as are at one s disposal, but finally to use latent variables as an input vector of much lower dimensionality for training neural networks. Again, SVD concentrates most of the relevant information (very often about 95 %) in a few initial columns of die scores matrix. 

Widely used methods of data transformation are Fast Fourier and Wavelet Transformations or Singular Value Decomposition... 

Data Mining is the core of the more comprehensive process of knowledge dis-coveiy in data bases (KDD). However, the term data mining" is often used synonymously with KDD. KDD describes the process of extracting and storing data and also includes methods for data preparation such as data cleaning, data selection, and data transformation as well as evaluation, presentation, and visualization of the results after the data mining process. 

Substitution of the data transforms the beat and material balances into Eqs. (H) and (I) ... 

LEGEND NN linear regression without data transformation LL idem, using logarithmically transformed axes not interpretable... 

System Evaluation Subject Management Data Quality Assurance Treatment Dispensing Handling Unexpected Events Data Transformation... 

To support the application of the learning methodology, fix) was used to generate 500 (x, 2, w) records of simulated operational data, transformed by Eq. (26) into an equivalent number of (x, y) pairs. Finally, the following constraints were imposed to the search procedure, 5 ... 

Linearization. In preliminary screening of reaction mechanisms, it is very useful to construct plots of experimental data transformed in such a way that the plot of the dependent (transformed) variable versus the independent (transformed) variable is a straight line if the rate equation being the basis of transformation has been chosen properly. This is illustrated with the rate expression for a-th order kinetics ... 

The Fourier transform may be considered as a special case of the general data transform... 

PCR is based on a PCA input data transformation that by definition is independent of the Y-data set. The approach to defining the X-Y relationship is therefore accomplished in two steps. The first is to perform PCA on the. Y-data, yielding a set of scores for each measurement vector. That is, if xk is the fcth vector of d measurements at a time k, then zk is the corresponding kth vector of scores. The score matrix Z is then regressed onto the Y data, generating the predictive model... 

Now, what is interesting about this situation is that ordinary regression theory and the theory of PCA and PLS specify that the model generated must be linear in the coefficients. Nothing is specified about the nature of the data (except that it be noise-free, as our simulated data is) the data may be non-linear to any degree. Ordinarily this is not a problem because any data transform may be used to linearize the data, if that is desirable. 

Before applying chemometric analysis to data, environmental data tables need a previous arrangement, which is often a very laborious part of the whole data analysis. Data disposal entails different processes such as data arrangement, homogenization, and data transformation which, in each case, depend on the format in which data have been initially obtained, their volume, etc. 

A variety of units have been used for the assessment of exposures to ionizing radiation. The current international standard terminology is shown in Table 32.3. This chapter uses the new terminology exclusively this frequently necessitated data transformation of units from early published accounts into the currently accepted international terminology. 

In-Kwon Yeo received the PhD degree in Statistics from University of Wisconsin-Madison in 1997. He joined the Department of Control and Instrumentation Engineering, Kangwon National University as a visiting professor in 2000 and the Division of Mathematics and Statistical Informatics, Chonbuk National University as an assistant professor in Korea. He is currently an associate professor at the Department of Statistics, Sookmyung Women s University. His current research interests include data transformations, multivariate time series analysis and generalized additive models. 

To determine whether the skew was responsible for the taxonic findings, Gleaves et al. transformed the data using a square root or log transformation and were successful at reducing the skew of all but one indicator to less than 1.0. This is a fairly conservative test of the taxonic Conjecture, because data transformation not only reduces indicator skew, but it can also reduce indicator validities, and hence produce a nontaxonic result. Yet, this did not happen in this study. All but one plot originally rated as taxonic were still rated as taxonic after the transformation. MAMBAC base rate estimates were. 19 (SD =. 18) for transformed empirical indicators, and. 24 (SD =. 06) for transformed theoretical indicators. Nevertheless, these estimates are probably not as reliable as the original estimates because of the possible reduction in validity, which is likely to lower the precision of the estimates. 

Separating the data transformation into three distinct steps enforces a completely modular software design. In practice, the data transformation is executed via command shell scripts, using freely available software for both the XSLT transformation and XSD validation. The raw data contained in the 39 surveys in the New Brunswick compilation are exported into 7,000 individual KML files, which can be viewed online at http //gdr.nrcan.gc.ca/geochem. 

There is also a special subset of statistical techniques that is part of both the second and third functions of statistics. This is data transformation, which includes such things as the conversion of numbers to log or probit values. 

Also nonlinear methods can be applied to represent the high-dimensional variable space in a smaller dimensional space (eventually in a two-dimensional plane) in general such data transformation is called a mapping. Widely used in chemometrics are Kohonen maps (Section 3.8.3) as well as latent variables based on artificial neural networks (Section 4.8.3.4). These methods may be necessary if linear methods fail, however, are more delicate to use properly and are less strictly defined than linear methods. 

Data transformations can be applied to change the distributions of the values of the variables, for instance to bring them closer to a normal distribution. Usually, the data are mean centered (column-wise), often they are autoscaled (means of all... 

By looking at the data one can observe right-skewed distributions for some of the variables. Thus an appropriate data transformation (e.g., the log-transformation) can improve the quality of the cluster results. However, it turned out that the results changed only marginally for the transformed data, and thus they will not be presented in the following. 

We will describe an accurate statistical method that includes a full assessment of error in the overall calibration process, that is, (I) the confidence interval around the graph, (2) an error band around unknown responses, and finally (3) the estimated amount intervals. To properly use the method, data will be adjusted by using general data transformations to achieve constant variance and linearity. It utilizes a six-step process to calculate amounts or concentration values of unknown samples and their estimated intervals from chromatographic response values using calibration graphs that are constructed by regression. 

The model with Mean PLR as the dependent variable required no data transformations. In the Mean PLA model, we transformed the positively skewed dependent variable with a natural logarithm to achieve a normal distribution, creating the In Mean PLA. The data contain no influential outliers. 

---