Outlier detection and remediation

The fact that not all outliers are erroneous leads to the following suggested practice of handling outliers during calibration development (1) detect, (2) assess and (3) remove if appropriate. In practice, however, there could be hundreds or thousands of calibration samples and x variables, thus rendering individual detection and assessment of all outliers a rather time-consuming process. However, time-consuming as it may be, outlier detection is one of the most important processes in model development. The tools described below enable one to accomplish this process in the most efficient and effective manner possible. 

It shonld be noted that Equations 12.21 and 12.22 refer to the and Q statistics for samples in the x data, not for variables. However, the and Q valnes for the variables in the x data are calculated in a very similar manner. The T values for x variables are calculated using the PCA loadings, rather than the scores  

Jnst as PCA can be effective for detecting subtle x-variable and x-sample ontliers, PLS or PCR can be effective for detecting y-sample ontliers. This is mainly done throngh the use of the y residuals (f) (see Equation 12.44). The squares of the individual N elements of the y-residual vector (f) can then be observed to detect the presence of y-sample ontliers. In a manner similar to that described above for Q and values, the squares of each of the elements in f can be compared to the 95% confidence level of these values in order to assess whether one or more samples might be ontliers dne to their y data. 

---