Outlier detection

Kelly, P. G. Outlier Detection in Gollaborative Studies, Anal. Chem. 1990, 73, 58-64. 

The 2 and 3 sigma cut-off limits of the conventional outlier-detection models are plotted. 

B. Mertens, M. Thompson and T. Fearn, Principal component outlier detection and SIMCA a synthesis. Analyst 119(1994) 2777-2784. 

The development of a calibration model is a time consuming process. Not only have the samples to be prepared and measured, but the modelling itself, including data pre-processing, outlier detection, estimation and validation, is not an automated procedure. Once the model is there, changes may occur in the instrumentation or other conditions (temperature, humidity) that require recalibration. Another situation is where a model has been set up for one instrument in a central location and one would like to distribute this model to other instruments within the organization without having to repeat the entire calibration process for all these individual instruments. One wonders whether it is possible to translate the model from one instrument (old or parent or master. A) to the others (new or children or slaves, B). 

B. Walczak and D.L. Massart, Robust PCR as outliers detection tool. Chemom. Intell. Lab. Syst., 27 (1995) 41-54. 

Rousseeuw PJ, Leroy AM (1987) Robust regression and outlier detection. Wiley, New York... 

Penrose R (1955) A generalized inverse for matrices. Proc Cambridge Phil Soc 51 406 Rousseeuw PJ, Leroy AM (1987) Robust regression and outlier detection. Wiley, New York Sachs L (1992) Angewandte Statistik. Springer, Berlin Heidelberg New York Sharaf MA, Illman DL, Kowalski BR (1986) Chemometrics. Wiley, New York... 

Outlier detection limits, n - the limiting value for application of an outlier detection method to a spectrum, beyond which the spectrum represents an extrapolation of the calibration model. 

Outlier detection methods, n - statistical tests which are conducted to determine if the analysis of a spectrum using a multivariate model represents an interpolation of the model. 

Outlier detection, in chemometrics, 6 56-57 Outokumpu flash smelting, 16 146 Outokumpu lead smelting process, 14 745 Outokumpu Oy process, selenium recovery via, 22 83... 

Chen, J., Bandoni, A., and Romagnoli, J. A. (1998). Outlier detection in process plant data. Comput. Chem. Eng. 22,641-646. 

Rousseeuw, P. J., Leroy, A. M. Robust Regression and Outlier Detection. Wiley, New York, 1987. 

For identifying outliers, it is crucial how center and covariance are estimated from the data. Since the classical estimators arithmetic mean vector x and sample covariance matrix C are very sensitive to outliers, they are not useful for the purpose of outlier detection by taking Equation 2.19 for the Mahalanobis distances. Instead, robust estimators have to be taken for the Mahalanobis distance, like the center and... 

The Mahalanobis distance used for multivariate outlier detection relies on the estimation of a covariance matrix (see Section 2.3.2), in this case preferably a robust covariance matrix. However, robust covariance estimators like the MCD estimator need more objects than variables, and thus for many applications with m>n this approach is not possible. For this situation, other multivariate outlier detection techniques can be used like a method based on robustified principal components (Filzmoser et al. 2008). The R code to apply this method on a data set X is as follows ... 

Leardi, R. J. Chemom. 8, 1994, 65-79. Application of a genetic algorithm for feature selection under full validation conditions and to outlier detection. 

Outliers demand special attention in chemometrics for several different reasons. During model development, their extremeness often gives them an unduly high influence in the calculation of the calibration model. Therefore, if they represent erroneous readings, then they will add disproportionately more error to the calibration model. Furthermore, even if they represent informative information, it might be determined that this specific information is irrelevant to the problem. Outliers are also very important during model deployment, because they can be informative indicators of specific failures or abnormalities in the process being sampled, or in the measurement system itself. This use of outlier detection is discussed in the Model Deployment section (12.10), later in this chapter. 

The fact that not all outliers are erroneous leads to the following suggested practice of handling outliers during calibration development (1) detect, (2) assess and (3) remove if appropriate. In practice, however, there could be hundreds or thousands of calibration samples and x variables, thus rendering individual detection and assessment of all outliers a rather time-consuming process. However, time-consuming as it may be, outlier detection is one of the most important processes in model development. The tools described below enable one to accomplish this process in the most efficient and effective manner possible. 

One other important consideration in constructing a library is checking that all spectra are correct. This can be done by visual inspection of the spectra and outlier detection tools. For various reasons (e.g. an inadequately filled cuvette, large temperature fluctuations), spectra may differ from others due to factors other than natural variability. Any such spectra should be excluded from the library. Spectral libraries are... 

Major steps In this type of analysis Include Initial data scaling and transformation, outlier detection, determination of the underlying factors, and evaluation of the effect that experimental procedures may have on the variance of the results. Most of the calculations were performed with the ARTHUR software package (O. 

The classical KNN approach does not have outlier detection capabilities. That is, adassification is always made, whether or not the unknown is a member of any of the classes in the training set. In Section 4.3-1, the method presented indudes outlier diagnostics which are generally not present in commercial satistical software. 

Raw Measurement Plot In multivariate calibration, it is normally not necessary to plot the prediction data if the outlier detection technique has not flagged the sample as an outlier. However, with MLR, the outlier detection methods are not as robust as with the full-spectrum techniques (e.g., CLS, PLS, PCR) because few variables are considered. Figure 5.75 shows all of the prediction data with the variables used in the modeling noted by vertical lines. One sample appears to be unusual, with an extra peak centered at variable 140. The prediction of this sample might be acceptable because the peak is not located on the variables used for the models. However, it is still suspect because the new peak is not expected and can be an indication of other problems. 

Method Inner space Outer space Affected by scaling Provision for modelling and outlier detection... 

The calculations for the Youden matched pairs procedure are less complicated than those for the lUPAC protocol and do not involve outlier detection and removal. For the example, the results are shown in Table 27. 

For this reason, it is of interest to learn the diverse types of calibration, together with their mathematical/statistical assumptions, the methods for validating these models and the possibilities of outlier detection. The objective is to select the calibration method that will be most suited for the type of analysis one is carrying out. 

Equation (4.20) was proposed by Hoskuldsson [65] many years ago and has been adopted by the American Society for Testing and Materials (ASTM) [59]. It generalises the univariate expression to the multivariate context and concisely describes the error propagated from three uncertainty sources to the standard error of the predicted concentration calibration concentration errors, errors in calibration instrumental signals and errors in test sample signals. Equations (4.19) and (4.20) assume that calibrations standards are representative of the test or future samples. However, if the test or future (real) sample presents uncalibrated components or spectral artefacts, the residuals will be abnormally large. In this case, the sample should be classified as an outlier and the analyte concentration cannot be predicted by the current model. This constitutes the basis of the excellent outlier detection capabilities of first-order multivariate methodologies. 

Outliers are also very important when one is applying a model because they can be used to indicate whether the model is being applied to an inappropriate sample. Details regarding such on-line outlier detection are provided in a later section (Section 8.4.3). 

For such outliers, detection and assessment can actually be accomplished using some of the modeling tools themselves.1,3 In this work, the use of PCA and PLS for outlier detection is discussed. Since the PCA method only operates on the X-data, it can be used to detect X-sample and X-variable outliers. The three entities in the PCA model that are most commonly used to detect such outliers are the estimated PCA scores (T), the estimated PCA loadings (P), and the estimated PCA residuals (E), which are calculated from the estimated PCA scores and loadings ... 

Of the three types of outliers listed earlier, there is one type that cannot be detected using the PCA method the Y-sample outlier. This is simply because the PCA method does not use any Y-variable information to build the model. In this case, outlier detection can be done using the PLS regression method. Once a PLS model is built, the Y-residuals (f) can be estimated from the PLS model parameters Tpls and q ... 

As a result, it is very important to evaluate process samples in real time for their appropriateness of use with the empirical model. Historically, this task has been often overlooked. This is very unfortunate not only because it is relatively easy to do, but also because it can effectively prevent the misuse of quantitative results obtained from a multivariate model. I would go so far as to say that it is irresponsible to implement a chemometric model without prediction outlier detection. 

---