Data pretreatment

Data pretreatment or data preprocessing is the mathematical manipulation of 

An effective preprocessing method is the use of standard normal variates (SNV). This type of standardization boils down to considering each spectmm x, as a set of q observations and calculating their z-scores  

It has the effect of removing an overall offset by subtracting the mean spectral reading x and it corrects for differences affecting the overall variation. In various settings it has been found to be an effective preprocessing method. 

Another popular form of data pre-processing with near-infrared data is the application of the Multiplicative Scatter Correction (MSC, [28]). It is well known that particle size distribution of non-homogeneous powders has an overall effect on the spectrum, raising all intensities as the average particle size increases. Individual spectra x, are approximated by a general offset plus a multiple of a reference spectrum, z. 

The offset a, and the multiplication constant bj are estimated by simple linear regression of the ith individual spectrum on the reference spectrum z. For the latter one may take the average of all spectra. The deviation e, from this fit carries the unique information. This deviation, after division by the multiplication constant, is used in the subsequent multivariate calibration. For the above correction it is not mandatory to use the entire spectral region. In fact, it is better to compute the offset and the slope from those parts of the wavelength range that contain no relevant chemical information. However, this requires spectroscopic knowledge that is not always available. 

A special type of data pre-treatment is the transformation of data into a smaller number of new variables. Principal components analysis is a natural example and we have treated it in Section 36.2.3 as PCR. Another way to summarize a spectrum in a few terms is through Fourier analysis. McClure [29] has shown how a NIR 

---

Improved x-data pretreatment Newly discovered effects in the on-line analyzer data could be more effectively filtered out using a different pretreatment method. More effective pretreatment would then reduce the burden on the modeling method to generate an effective model. 

An increase from 5 to 10 in the number of factors representing the original data results in a substantial reduction in the error. Because of the data pretreatment used, the spectral error cannot be directly compared to the experimental error determined from the data set. When five factors were used, two different lignite samples were flagged as possible outliers based on their spectral variances relative to the rest of the data set. With ten factors, one of the lignites was accommodated within the factor model (although ten factors may not have been required to accommodate it). With thirteen factors, both lignites were accommodated. 

To maintain consistency of statistical analyses, an identical microtiter plate setup was used by all participants, and all samples were analyzed in an identical manner. Both raw data and pretreated data from analyzed samples were submitted to OpdenKamp Registration and Notifieation for statistical evaluation. Data pretreatment consisted of all necessary calculations to convert the luminosity readings as submitted by the participating laboratories to effective dioxin-receptor activity (pM 2,3,7,8-TCDD TEQ). In addition to the analysis results of the defined samples (phase 1), the cleaned sediment extracts (phase 2), and the complete sediments (phase 3), all participants also submitted the results of the complete 2,3,7,8-TCDD calibration curves for statistical evaluation. 

The summary statistics for accuracy described in the previous section are just average statistics for the whole set of samples. They are important, because they allow monitoring of changes when the calibration model is optimised i.e. a dilferent data pretreatment or optimal number of factors is used). However, they do not provide an indication of the uncertainty for individual predicted concentrations. Uncertainty is defined as aparameter, associated with the result of a measurement, which characterises the dispersion of the values that could reasonably be attributed to the measurancT [60]. Therefore, uncertainty gives an idea of the quality of the result since it provides the range of values in which the analyst believes that the true concentration of the analyte is situated. Its estimation is a requirement for analytical laboratories [61] and is especially important when analytical results have to be compared with an established legal threshold. 

Martin et al. [102] reported a study in which LIBS was applied for the first time to wood-based materials where preservatives containing metals had to be determined. They applied PLS-1 block and PLS-2 block (because of the interdependence of the analytes) to multiplicative scattered-corrected data (a data pretreatment option of most use when diffuse radiation is employed to obtain spectra). They authors studied the loadings of a PCA decomposition to identify the main chemical features that grouped samples. Unfortunately, they did not extend the study to the PLS factors. However, they analysed the regression coefficients to determine the most important variables for some predictive models. 

Explained Variances by PCA for Three Data Pretreatment Methods(%)... 

Figure 11.19 shows the results obtained after applying MCR-ALS with nonnegativity constraints to the same SVOC data set shown in Figure 11.16. Using five components, the explained data variance was 84.1%, very close to the value obtained by PCA (84.4%) for mean-centered data. MCR-ALS was directly applied to raw data, without any further data pretreatment apart from imputation of missing data (PLS Toolbox missing.m function) and setting values below the detection limit to...

G. W. Small et al., Evaluation of Data Pretreatment and Model Building Methods for the Determination of Glucose from Near-Infrared Single-Beam Spectra, Appl. Spectrosc., 53(4), 402 (1999). 

---