Detection of outliers

It is advisable always to use principal components analysis to check the consistency when large data tables are anafysed. Two or three components are sufficient to detect clear outliers. Outh ers are almost always found, and in most cases they are artifacts arising from blunder-type errors committed in the compilation of the data or errors committed in the preparation of the data, e.g. typing errors when data are transferred, aberrant observations, or errors committed in analysis or experimental procedures. Erroneous data of this kind are almost impossible to detect by visual inspection of the data table, but are easily detected firom the score plots. 

Outliers due to real differences in data will also be detected by plotting the first two score vectors against each other. It depends on the specific problem whether or not they should be included in the final analysis. Nevertheless, they can be detected at an early stage of the invesitgation. 

---

The detection of outliers, particularly when working with a small number of samples, is discussed in the following papers. Efstathiou, G. Stochastic Galculation of Gritical Q-Test Values for the Detection of Outliers in Measurements, /. Chem. Educ. 1992, 69, 773-736. 

For example, a single estimate for total PCB s has been historically collected in the NHATS program. Current advances in chemical analysis protocols now allow for the determination of isomer specific resolution of PCB s. Given the 209 PCB s that are now possible to detect, an adequate evaluation of the data without the use of pattern recognition techniques seems impossible. From a QA/QC perspective, these methods can facilitate the detection of outliers and aid in the interpretation of human chemical residue data. The application of statistical analysis must keep abreast with these advances made in chemisty. To handle the complexity and quantity of such data, the use of more sophisticated statistical analyses is needed. 

Once a chemometric model is built, and it is used to produce concentration or property values in real time from on-line analyzer profiles, the detection of outliers is a particularly critical task. This is the case for two reasons ... 

An advantage of PLS is that the latent variable structure can be studied. This can lead to the detection of outliers and data classes. There are also clues about which spectroscopic variables (wavelengths) influence the solution the most. A detailed view of PLS is given by the equations (6) - (8), also visualized in figures 12.15 e and f. 

Exploratory data analysis (EDA). This analysis, also called pretreatment of data , is essential to avoid wrong or obvious conclusions. The EDA objective is to obtain the maximum useful information from each piece of chemico-physical data because the perception and experience of a researcher cannot be sufficient to single out all the significant information. This step comprises descriptive univariate statistical algorithms (e.g. mean, normality assumption, skewness, kurtosis, variance, coefficient of variation), detection of outliers, cleansing of data matrix, measures of the analytical method quality (e.g. precision, sensibility, robustness, uncertainty, traceability) (Eurachem, 1998) and the use of basic algorithms such as box-and-whisker, stem-and-leaf, etc. 

The deviation from mean (DFM) in well n is the relative deviation of the volume in one well from the mean volume over the plate. This value is obtained from precision OD or fluorescence measurements and is used for the detection of outliers and systematic errors. The DFM is defined as... 

FURTHER DEVELOPMENTS IN DATA ANALYSIS 9.1. Regression analysis/detection of outliers... 

It is often claimed that partial residual plots are useful omnibus plots that allow detection of outliers, influential or leverage observations, nonlinearity, and other informative nonrandom patterns. The detection of nonlinearity, however, is the central motivation for partial residual plots. 

Figure 7.31. Detection and diagnosis of process upsets, (a) Detection of outliers based on residuals, (b) Detection based on test of scores, (c) Diagnosis statistics considering each possible disturbance, (d) Index of chosen disturbance for each observation. Reprinted from [243]. Copyright 1997 with permission from Elsevier.

Interpretation of the Tucker3 core is very important and sometimes core rotations are needed. Some authors rotate in order to simplify the core, while others see the need for a varimax rotation of the loadings. These rotations are very subjective and require extensive background knowledge. The loadings are used in line or scatter plots and allow the detection of outliers, groupings and trends. Joint plots are used frequently. 

These error values may be depicted graphically with the number n of the corresponding training spectrum on the abscissa (cf. Fig. 22.10). This graph permits an easy detection of outlier spectra. 

As has been shown, SPC serves as a useful tool for the detection of outliers and harmonics patterns in electrical networks. However, its use depends on the data of the studied sample being distributed normally. 

Control graphs and the concept of rational subgroups can be used successfully in the search and elimination of outliers in harmonics present in electrical systems, provided that the data set follows a normal distribution. When data do not follow a normal distribution. Functional Data Analysis can be utilized effectively in the detection of outliers, also contributing major advantages in the detection of specific variability compared to traditional techniques such as Statistical Control Process. The functional approach greatly enhances the capacity for analysis facilitating massive and systematic analysis of the data. 

Provided the limitations of QSARs in environmental sciences are considered and the respective application criteria are complied with, structure-activity relationships can be a powerful tool for elucidating modes of interaction, screening the validity and the plausibility of experimental data, detecting of outliers and predicting activity parameters for product development, identification of priority pollutants and range finding for further testing. 

---