Robust estimates of location and spread

A simple robust estimate of the standard deviation is provided by the interquartile range (IQR, see Section 6.2). For a normal error distribution, the IQR is ca. 1.35 r. (The IQR divided by 1.35 is sometimes called the standardized IQR, abbreviated SIQR.) This relationship supplies a standard deviation estimate that is not affected by any value taken by the largest or smallest measurements. Unfortunately, the IQR is not a very meaningful concept for very small data sets. Moreover, and somewhat surprisingly, there are several different conventions for its calculation. For large samples the convention chosen makes little difference, but for small samples the differences in the calculated IQR values are large, so the IQR has little application in analytical chemistry. 

In experimental science we normally have no advance knowledge of whether or not our data might come from a heavy-tailed distribution or might contain outliers. So ideally we would like to use an estimate of location (the mean and the median are location estimates) that behaves like the mean when the underlying distribution is truly normal, but has the robust properties of the median when outliers or heavy tails do occur. Analogous arguments apply to measures of spread. Over 30 years ago Huber and others showed that these desirable properties are available. 

Suppose we have a series of n results x. .. x , and we wish to estimate n, the mean of the reliable results. Usually our estimate of jx, given here the symbol jx, is obtained by minimizing the sum of squares (SS) X ( i - (This sum of squared terms is the source of the sensitivity of the mean to large errors.) The expression (Xj - jx) is referred to as a distance function, since it measures the distance of a point from jx. A more useful distance function in the present context is Xj-/r. A widely used method to test measurements for downweighting (winsorization) is to compare Xj-,u with ccT, where c is usually taken to be 1.5 and cr is a robust estimate of the standard deviation. We consider first the estimation of a, and then discuss the downweighting procedure. 

The robust variance estimate can be derived from a statistic related to the unfortunately abbreviated median absolute deviation (MAD ), which is calculated from 

The MAD is an extremely useful statistic one rough method for evaluating outliers (xo) is to reject them if [ xq-median (Xj) ]/MAD 5. It can be shown that MAD/0.6745 is a useful robust estimate of a (often called the standard deviation based on MAD, the standardized MAD, or SMAD, and given the symbol a) which can be used unchanged during the iterative estimates of jx. 

---

FUZZY CALIBRATION OF ANALYTICAL METHODS AND FUZZY ROBUST ESTIMATION OF LOCATION AND SPREAD... 

A solution in this case is to use robust models and robust statistics (such as median as measure of location and median of absolute deviation around the median as measure of spread) in estimating PCA parameters. The aim is to construct models and estimates clearly describing the majority of the data. Moreover, construction of robust models allows a proper identification of outlying observations. A review illustrating the basis of robust techniques in data analysis and chemometrics can be found in reference [91]. 

Figure 3 shows the true and false positive rates obtained by evaluating the STCA system on 500 bootstrap replications for parameters on the front. While there is considerable spread about each location on the front, these scatter diagrams provide an estimate of the robustness of the parameter set to the data and indicate the range of true and false positive rates that may be expected at a particular operating point. Plots and statistics such as these permit the decision maker to accurately assess the probability of the true... 

---