Data pre-treatment

One of the most frequently overlooked aspects of data analysis is consideration of the data that is going to be analysed. How accurate is it How complete is it How representative is it These are some of the questions that should be asked about any set of data, preferably before starting to try and understand it, along with the general question what do the numbers, or symbols, or categories mean  

The next few sections discuss some of the more important aspects of the nature and properties of data. It is often the data itself that dictates which particular analytical method may be used to examine it and how successful the outcome of that examination will be. 

It is most often the case that raw data from analytical instruments needs to be treated by one or more operations before optimal results can be obtained from chemometric modeling methods. Although such pre-treatments often result in improved model performance, it is critically important to understand the inherent assumptions of these pretreatment methods in order to use them optimally. 

Mathematically, this operation involves the subtraction of each variable s response from the mean response of that variable over all of the samples in the data set. In the case where one wants to mean-center all of the independent variables in a data set, mean-centering is represented by the following equation  

The mean-centering operation effectively removes the absolute intensity information from each of the variables, thus enabling one to focus on the response variations. This can effectively reduce the burden on chemometric modeling techniques by allowing them to focus on explaining variability in the data. For those who are still interested in absolute intensity information for model interpretation, this information is stored in the mean vector (x), and can be retrieved after modeling is done. 

This procedure is an extension of mean-centering. It is also a variable-wise pre-treatment that consists of mean-centering followed by division of the resulting intensities by the variable s standard deviation  

Autoscaled data have the unique characteristic that each of the variables has a zero mean and a standard deviation of one. Like mean-centering, autoscaling removes absolute intensity information. However, unlike mean-centering, it also removes total variance information in each of the variables. It effectively puts each of the variables on equal footing before modeling is done. 

---

A special type of data pre-treatment is the transformation of data into a smaller number of new variables. Principal components analysis is a natural example and we have treated it in Section 36.2.3 as PCR. Another way to summarize a spectrum in a few terms is through Fourier analysis. McClure [29] has shown how a NIR... 

Figure 8.3 A set of uncorrected NIR reflectance spectra to be used to demonstrate different data pre-treatment methods.

For example, NIR spectra are temperature-sensitive, so control to 2°C is required. Ultrasonics using two probes at 5 MHz offers the possibility of a robust and less expensive method, but it is still under development and there are few commercially available systems. Chemometrics (Section 3.5) offers the best method of data analysis, and, by using PLS with three different data pre-treatments, it was possible to obtain real-time values of the composition of a fire-retardant to better than 0.1% accuracy (Fischer et al, 2006). 

Janne K, Pettersen J, Lindberg N-O, LundstedtT. Hierarchical Principal Component Analysis (PCA) and Projection to Latent Structure (PLS) technique on spectroscopic data as a data pre-treatment for calibration. J Chemom 2001 15 203-213. 

---