Noise and Outliers

It is not unreasonable to expect data drawn from real world experiments to contain high levels of noise and/or outliers. Spectral dimensionality reduction methods are highly susceptible to noise, as shown in Fig. 3.4. As the noise level increases the measured performance of spectral dimensionality reduction decreases. This is not surprising as high noise levels will make it difficult for the underlying manifold to be adequately and accurately modelled. Therefore, there is need for methods to be used that enable spectral dimensionality reduction methods to be employed in the presence of noisy data. 

---

Figure 6.16. The robust filtering technique performed on the Bumps benchmark signal (grey solid) with noise and outliers. The dashed signal is the signal estimate. Reprinted from [59]. Copyright 2001 with permission from Elsevier.

A powerful and efficient solution to these problems is to use more robust measures of the margin distribution. As opposed to the maximal bound, such measures provide a more feasible bound in the case of noise and outliers (see [41] and Chapter 4 in [42]). This bound is associated with non-negative variables, 0, also known as slack variables. 

Keywords Neighbourhood graphs Manifold approximations Noise and outliers Data topologies... 

This method is extremely useful to detect the points that have a large noise component (outliers) and therefore are exceedingly far from the subspace of the significant principal components. 

In today s process analytical instruments, where response noise and reproducibility have been greatly improved, it is quite possible to encounter outliers that are not easily visible by plotting the raw data. These outliers could involve single variables or samples that have relatively small deviations from the rest of the data, or they could involve sets of variables or sets of samples that have a unique multivariate pattern. In either case, these outliers, if they represent unwanted or erroneous phenomena, can have a negative impact on the calibration model. 

The column was operated four times at various operating conditions. The first three data sets corresponding to a total of 12.8 hr of operation were used to train the 0-NLPCA network, and the fourth one was used for model validation. However, prior to building a calibration model, both the training and the testing data were processed through the robust tandem filter to remove noise and suppress possible outliers. 

Fig. 15 (a) A bumps signal contaminated with white noise of variance 0.5 and outlier patches of length 3. (b) robust OLMS filtering, median filter length = 9, Haar wavelet, scale depth = 2 (MSE = 0.8366). 

Vulnerability to Noisy Data and Outliers In the conventional clustering algorithms, cluster proximity is measured by distance metrics. Outliers and high level of noise, often present in the biological data, can substantially influence the calculation of cluster centroids. 

Raman data collected into spectra contains a lot of information. However, this information is not usually directly available and the data must be processed to get qualitative and quantitative chemical informations. The main fact is that Raman line intensities are proportional to the concentration of chemical components and the Raman line positions are characteristics of the chemical bonding. Nevertheless, some phenomena like undesirable fluorescence, band overlapping, noise and spikes (outlier points) complicate the experimenter s task. 

Frequently, the measurement error distributions arising in a practical data set deviate from the assumed Gaussian model, and they are often characterized by heavier tails (due to the presence of outliers). A typical heavy-tailed noise record is given in Fig. 7, while Fig. 8 shows the QQ-plots of this record, based on the hypothesized standard normal distribution. 

One advantage of the cross-validation residuals is that they are more sensitive to outliers. Because the left out samples do not influence the construaion of the PCA models, unusual samples will have inflated residuals. The cross-validation PCA models are also less prone to modeling noise in the data and therefore the resulting residuals better reflect the inherent noise in the data set. The identification and removal of outliers and better estimation of noise can provide a more realistic estimate of the inherent dimensionaliw of a data set. 

---