Data pre-processing

The raw data, in units of absorbance, were truncated to the most diagnostic wave-number range, typically 800-1800cm, and corrected for a constant offset. The 

For PCA, the entire spectral data set, containing n spectra, is written as a matrix S in which each column represents one spectral vector S(v) of m intensity data points. The spectral vectors may be raw or smoothed intensities, or first or second derivatives. 

The intensity correlation matrix C is constructed from the spectral matrix S according to 

For spectral data sets of individual cells, it is found that a large fraction of the total spectral variance is contained in the first few loading vectors . Typically, five to eight loading vectors contain more than 99% of the variance, such that the Hnear 

The mean and standard deviation generally are not part of the important aspects of the data and may obscure the issue and complicate the networks task. A good method for removing both is the Z-score procedure, which involves subtracting the mean and dividing by the standard deviation. This removes all effects of offset and measurement scale. A simple linear mapping can be used to normalize the Z-scores to the bounds of the transfer function. 

No single spectrometer is the perfect input to Eq. (2). Each experimental plan has limitations and many of these are associated with accessible compositions, temperatures and pressures. In addition, there is the issue of having an overdetermined system as remarked previously. Taken together, we arrive at the conclusion  

Given adequate consideration of the experimental system, the spectroscopies used, and the experimental design, a good hut not perfect data set measured. Although the data set is necessarily incomplete, (like any truly 

Data pre-processing is commonly used prior to the solution of an inverse problem. In this section, five data pre-processing procedures are discussed. None of these procedures are absolutely necessary to obtain a solution to Eq. (2). However, all have been implemented, at one time or another by our group, in order to improve the inversion of in situ spectroscopic measurements of catalytic systems.  

---

First, we have an initial, and probably utterly crude, dataset. Genuine data pre-processing has only just started. The task is to assess the quality of the data. One of the topics for discussion in this chapter is the methods by which one finds out the potential drawbacks of the dataset. 

The two main ways of data pre-processing are mean-centering and scaling. Mean-centering is a procedure by which one computes the means for each column (variable), and then subtracts them from each element of the column. One can do the same with the rows (i.e., for each object). ScaUng is a a slightly more sophisticated procedure. Let us consider unit-variance scaling. First we calculate the standard deviation of each column, and then we divide each element of the column by the deviation. 

Mean-centering, as is shown by experience, can be successfully employed in combination with another data pre-processing technique, namely scaling, which is discussed later. 

Another popular form of data pre-processing with near-infrared data is the application of the Multiplicative Scatter Correction (MSC, [28]). It is well known that particle size distribution of non-homogeneous powders has an overall effect on the spectrum, raising all intensities as the average particle size increases. Individual spectra x, are approximated by a general offset plus a multiple of a reference spectrum, z. 

The development of a calibration model is a time consuming process. Not only have the samples to be prepared and measured, but the modelling itself, including data pre-processing, outlier detection, estimation and validation, is not an automated procedure. Once the model is there, changes may occur in the instrumentation or other conditions (temperature, humidity) that require recalibration. Another situation is where a model has been set up for one instrument in a central location and one would like to distribute this model to other instruments within the organization without having to repeat the entire calibration process for all these individual instruments. One wonders whether it is possible to translate the model from one instrument (old or parent or master. A) to the others (new or children or slaves, B). 

Fig. 2.7 Sensor production technologies needs a reliable and cost effective integration of technologies for the sensor element, the electronic data (pre) processing assembly.

The scope of this chapter is devoted to solving Eq. (2) and the implications that thereby arise. Accordingly, a description of the physical system is presented (Section 4.2), experimental design is discussed at length (Section 4.3), data pre-processing is addressed (Section 4.4) the current status of pure component spectral recovery is reviewed along with the salient mathematical issues (Section 4.5), and future directions are indicated (Section 4.6). 

Savolainen et al. investigated the role of Raman spectroscopy for monitoring amorphous content and compared the performance with that of NIR spectroscopy [41], Partial least squares (PLS) models in combination with several data pre-processing methods were employed. The prediction error for an independent test set was in the range of 2-3% for both NIR and Raman spectroscopy for amorphous and crystalline a-lactose monohydrate. The authors concluded that both techniques are useful for quantifying amorphous content however, the performance depends on process unit operation. Rantanen et al. performed a similar study of anhydrate/hydrate powder mixtures of nitrofurantoin, theophyllin, caffeine and carbamazepine [42], They found that both NIR and Raman performed well and that multivariate evaluation not always improves the evaluation in the case of Raman data. Santesson et al. demonstrated in situ Raman monitoring of crystallisation in acoustically levitated nanolitre drops [43]. Indomethazine and benzamide were used as model... 

This improvement strategy involves the reduction of irrelevant information in the X-data, thus reducing the burden on the modeling method to define the correlation with the Y-data. Various types of X-data pre-processing, discussed earlier (Section 8.2.5), can be used to reduce such irrelevant information. Improvement can also be obtained through elimination of X-variables that are determined to be irrelevant. Specific techniques for the selection of relevant X-variables are discussed in a later section. 

This chapter describes the algorithms of the various data analysis methods currently used for developing toxicological QSAR models. Data collection, data pre-processing, computation and selection of molecular descriptors, and model validation have been extensively reviewed elsewhere [2-11], so they are not described here. Freely available online software and commercial software available for constructing QSAR models of various toxicological properties prediction are also discussed. 

I) Application of Filter cuts rejecting accidental noise triggers or events detected with non-optimal atmospheric condition. The trigger rate after filter is (0.5 0.1)Hz. Then the data pre-processing and image reconstruction are performed [10]. 

Bijlsma, S. et al., Large-scale human metabolomics studies A strategy for data (pre-) processing and validation, Anal. Chem., 78(2), 567, 2006. 

A special application of FA in QS AR work is its use as a data pre-processing step in multiple regression [1, 50, 51] and other analyses [212], The ta matrix to be analysed then contains the biological potency to be considered as well as all molecular parameters to be checked as descriptors. After varimax rotation to obtain a simple stmcture the following can be deduced from the factor pattern ... 

---