Data preprocessing

For a data array X containing responses of M variables over N different samples, this method involves the calculation of the mean response of each of the M variables over the N samples, and, for each of the M variables, subsequent subtraction of this mean from the original responses. Mean-centering can be represented by the following equation  

The mean-centering operation effectively removes the absolute intensity information from each of the variables, thus enabling snbsequent modeling methods to focus on the response variations about the mean. In PAT instrument calibration applications, mean-centering is almost always nsefnl, because it is almost always the case that relevant analyzer signal is represented by variation in responses at different variables, and that the absolute values of the responses at those variables are not relevant to the problem at hand. 

This actually refers to a family of similar spectral pretreatment methods that involve the snbtraction of a 

Different baseline correction methods vary with respect to the both the properties of the baseline component d and the means of determining the constant k. One of the simpler options, baseline ojfset correction, nses a flat-line baseline component (d = vector of Is), where k can be simply assigned to a single intensity of the spectrum x at a specific variable, or the mean of several intensities in the spectrum. More elaborate baseline correction schemes allow for more complex baseline components, such as linear, quadratic or user-defined functions. These schemes can also utilize different methods for determining k, such as least-squares regression. 

This common variable-wise scaling method consists of mean-centering followed by division of the resnlting mean-centered intensities by the variable s standard deviation  

The Simplified Molecular Input Line Entry System (SMILES) strings ofthe structures in the data set were canonicalized, the charges were standardized, the additional fragments and salts were removed, and duplicate or invalid structures were identified and removed using the KNIME workflow environment [29]. Further data quality control was performed by the Eli Lilly AD ME group. 

The characteristics ofthe metabolic stability data set are summarized in Table 6.2. 

Property Size Mean Median StdDev Max Min Unit 

---

And, last but by far not least we must mention a very important part of data preprocessing. It is up to a researcher to decide when to employ these techniques. Figure 4-2 displays a step-by-step preparation of a dataset. 

E. de Noord, The influence of data preprocessing on the robustness nd parsimony of multivariate calibration models. Chemom. Intell. Lab. Systems, 23 (1994) 65-70,... 

Uncertainty in Process Measurements. Sensor measurements are always subject to noise, calibration error, and temporary signal loss, as well as various faults that may not be immediately detected. Therefore, data preprocessing will often be required to overcome the inherent limitations of... 

The goal of EDA is to reveal structures, peculiarities and relationships in data. So, EDA can be seen as a kind of detective work of the data analyst. As a result, methods of data preprocessing, outlier selection and statistical data analysis can be chosen. EDA is especially suitable for interactive proceeding with computers (Buja et al. [1996]). Although graphical methods cannot substitute statistical methods, they can play an essential role in the recognition of relationships. An informative example has been shown by Anscombe [1973] (see also Danzer et al. [2001], p 99) regarding bivariate relationships. 

In Chapter 2, we approach multivariate data analysis. This chapter will be helpful for getting familiar with the matrix notation used throughout the book. The art of statistical data analysis starts with an appropriate data preprocessing, and Section 2.2 mentions some basic transformation methods. The multivariate data information is contained in the covariance and distance matrix, respectively. Therefore, Sections... 

C. EXPERTISE INCLUDES DATA PREPROCESSING AND EVALUATION OF RESULTS... 

To build a calibration model, the software requires the concentration and spectral data, preprocessing options, the maximum rank (number of factors) to estimate, and the approach to use to choose the optimal number of factors to include in the model. This last option usually involves selection of the cross- i alidation technique or the use of a separate validation set. The maximum... 

Several very important accessory tools, for example for data preprocessing and variable selection, complete the chemometric pattern recognition arsenal. 

Evaluate carefully the most appropriate data preprocessing tools (signal corrections, transforms, compression) in order to minimize the amount of unwanted information. 

The SIMCA method has been developed to overcome some of these limitations. The SIMCA model consists of a collection of PCA models with one for each class in the dataset. This is shown graphically in Figure 10. The four graphs show one model for each excipient. Note that these score plots have their origin at the center of the dataset, and the blue dashed line marks the 95% confidence limit calculated based upon the variability of the data. To use the SIMCA method, a PCA model is built for each class. These class models are built to optimize the description of a particular excipient. Thus, each model contains all the usual parts of a PCA model mean vector, scaling information, data preprocessing, etc., and they can have a different number of PCs, i.e., the number of PCs should be appropriate for the class dataset. In other words, each model is a fully independent PCA model. 

Data pretreatment or data preprocessing is the mathematical manipulation of... 

The authors of this book have found LOTUS and EXCEL satisfactory for data preprocessing and STATGRAPH, SPSS, and STATISTICA suitable for the application of statistical methods. UNSCRAMBLER is recommended if applying PLS modeling. 

Obviously in Eq. 5-8 one cannot handle measurements of features with different units. It is, for example, not possible to subtract pH values from Fe content measured in %. Therefore one important aspect of data preprocessing is to ensure the comparability of the features. Even if no EUCLIDean measure is used one should keep this aspect in mind. 

The data processing can be divided into three phases. Phase 1 is the removal of poor quality spectra with an automated routine. Phase 2 is the data preprocessing of the spectra, which passed the quality test. This usually entails some type of baseline correction and normalization process. Phase 3 is multivariate image reconstruction where the spectra are classified and reproduced as color points... 

Phase 2 - data preprocessing. There are many ways to process spectral data prior to multivariate image reconstruction and there is no ideal method that can be generally applied to all types of tissue. It is usual practice to correct the baseline to account for nonspecific matrix absorptions and scattering induced by the physical or bulk properties of the dehydrated tissue. One possible procedure is to fit a polynomial function to a preselected set of minima points and zero the baseline to these minima points. However, this type of fit can introduce artifacts because baseline variation can be so extreme that one set of baseline points may not account for all types of baseline variation. A more acceptable way to correct spectral baselines is to use the derivatives of the spectra. This can only be achieved if the S/N of the individual spectra is high and if an appropriate smoothing factor is introduced to reduce noise in the derivatized spectra. Derivatives serve two purposes they minimize broad... 

---