Data distribution outliers

The AO AC (cited in Ref. [11]) described the precision acceptance criteria at different concentrations within or between days, the details of which are provided in Table 2. Other parameters that should be tested in the precision study are the David-, Dixon- or Grubbs-, and Neumann-tests. The David-test is performed when determining whether the precision data are normally distributed. Outlier testing of the data is performed by the Dixon-test (if n < 6-8) or by the Grubbs-test (if n > 6-8), while trend testing of the data is performed by Neumann-test. Detailed methods have been described in the book written by Kromidas [29]. 

Xr from the robust approach, as expected, still gives the correct answer however, the conventional approach fails to provide a good estimate of the process variables. Although the main part of the data distribution is Gaussian, the conventional approach fails in the task because of the presence of just one outlier. In a strict sense, the presence of this outlier results in the invalidation of the statistical basis of data reconciliation,... 

Section 1.6.2 discussed some theoretical distributions which are defined by more or less complicated mathematical formulae they aim at modeling real empirical data distributions or are used in statistical tests. There are some reasons to believe that phenomena observed in nature indeed follow such distributions. The normal distribution is the most widely used distribution in statistics, and it is fully determined by the mean value p. and the standard deviation a. For practical data these two parameters have to be estimated using the data at hand. This section discusses some possibilities to estimate the mean or central value, and the next section mentions different estimators for the standard deviation or spread the described criteria are fisted in Table 1.2. The choice of the estimator depends mainly on the data quality. Do the data really follow the underlying hypothetical distribution Or are there outliers or extreme values that could influence classical estimators and call for robust counterparts ... 

A robust measure for the central value—much less influenced by outliers than the mean—is the median xm (x median). The median divides the data distribution into two equal halves the number of data higher than the median is equal to the number of data lower than the median. In the case that n is an even number, there exist two central values and the arithmetic mean of them is taken as the median. Because the median is solely based on the ordering of the data values, it is not affected by extreme values. 

The data fall on a straight line and are concluded to be normally distributed. Outliers in the data are seen as points to the far right and far left of the line. 

Exploratory data analysis is a collection of techniques that search for structure in a data set before calculating any statistic model [Krzanowski, 1988]. Its purpose is to obtain information about the data distribution, about the presence of outliers and clusters, and to disclose relationships and correlations between objects and/or variables. Principal component analysis and cluster analysis are the most well-known techniques for data exploration [Jolliffe, 1986 Jackson, 1991 Basilevsky, 1994]. 

Methods for identifying and handling of possible outlier data should be specified in the protocol. Medical or pharmacokinetic explanations for such observations should be sought and discussed. As outliers may be indicative of product failure, post hoc deletion of outlier values is generally discouraged. An approach to dealing with data containing outliers is to apply distribution-free (non-parametric), statistical methods (72). 

Occasionally, naive samples are encountered with preexisting or high levels of ADAs. If a confirmatory test shows that the ADA response is specific to the drug, such samples should be excluded from the cut point calculations. In addition, samples that are identified as outliers using appropriate statistical criteria [34] should also be excluded or down weighted in the analyses. If a substantial number of naive samples are positive from the relevant disease population, it is acceptable to include samples from a healthy or non-diseased population for determining the cut point. For example, if the distribution of the data of outlier-removed samples is not statistically different between healthy and disease-matched subjects (i.e., means and variances are not significantly different), then the cut point evaluation can be made from a collection of samples where half the samples are from healthy subjects and the other half are from the relevant disease population. 

Prior to a formal analysis, a database should be examined for any unusual characteristics of the data distribution. A database may have some number of outliers, an inherent nonnormal or skewed distribution, or a bimodal character due to the presence of two separate underlying distributions. Most tests for normality are intended for fairly large sample sizes of the order of 15 or more. Smaller databases may be reviewed for unusual characteristics by way of the usual statistical algorithms available with spreadsheets. Tests for normality are listed in Part 2. 

When control charts are developed for abnormal data distributions, as in the studied case, the results are as saw in Figure 3. In this figure, the control graph for the THDV of the LI phase of Sample A shows a very high number of outliers, 37 out of a set of 39 elements, which is totally contrary to the definition of outlier in section 2 of this work. This behavior persists even when submitting the data set to a Box-Cox transformation in an attempt to correct the distribution bias, the differences of the variances on the time axis or the possible nonlinearity of the data. 

Figure 5 contains a graph that shows the results of the functional outliers obtained for the THDV of Sample A, which could not be analyzed using traditional SCP control graphs as it contained an abnormal data distribution. The 39 functions recorded based on the 144 data/day appear in each of... 

Another problem which is frequently encountered in the distribution of data is the presence of outliers. Consider the data shown in Table 3.1 where calculated values of electrophilic superdelocalizability (ESDLIO) are given for a set of analogues of antimycin Al, compounds which kill human parasitic worms, Dipetalonema vitae. The mean and standard deviation of this variable give no clues as to how well it is distributed and the skewness and kurtosis values of —3.15 and 10.65 respectively might not suggest that it deviates too seriously from normal. A frequency distribution for this variable, however, reveals the presence of a single extreme value (compound 14) as shown in Fig. 3.5. This data was analysed by multiple linear regression (discussed further in Chapter 6), which is a parametric method based on the normal distribution. The presence of this outlier had quite profound effects on the analysis, which could have been avoided if the data distribution had been checked at the outset (particularly by the present author). Outliers can be very informative and should not simply be discarded as so frequently happens. If an outlier is found in one of the descriptor variables (physicochemical data), then it may show... 

A final point on the matter of data distribution concerns the non-parametric methods. Although these techniques are not based on distributional assumptions, they may still suffer from the effects of strange distributions in the data. The presence of outliers or the effective conversion of interval to ordinal data, as in the electrostatic potential example, may lead to misleading results. 

A different praetieal problem occurs when one or more observations apparently do not follow the same model as the other data. Such outliers or leverage points strongly influence the regression model. Real data are often subjeet to these problems that sometimes make the use of elassieal statisties, based on normal distribution, suboptimal. This issue needs more eareful diseussions and will be addressed in Section 2.4.2. A possible alternative to the LS criterion in such instances is to use proeedures that are robust to outliers or deviations from normalitywhieh are out of the seope of this chapter. 

One of the basic concepts in robust statistics that was introduced by Hampel [1] is the so-called breakdown point of an estimator. As the name suggests, this is the fraction of outliers that makes an estimator break, that is an estimator provides estimates that are arbitrarily small or large. Let us consider two important classic estimators (i) a sample mean describing the location of data distribution and (ii) a sample standard deviation characterizing the distribution scale. Both classic estimators are non-robust with a breakdown point of 0%. 

With only 9 data points, the data distribute nicely about zero, and the magnitude does not (visually) appear to be a function of x. It can be concluded that the fit is adequate. Note that a single point with a very large residual is a candidate "outlier" and might be omitted if this can be justified (poor experimental procedure, other extenuating circumstances, etc.). Elowever, the arbitrary exclusion of outliers must be avoided. 

This experiment uses the change in the mass of a U.S. penny to create data sets with outliers. Students are given a sample of ten pennies, nine of which are from one population. The Q-test is used to verify that the outlier can be rejected. Glass data from each of the two populations of pennies are pooled and compared with results predicted for a normal distribution. 

With small data sets or if there is reason to suspect deviations from the Gaussian distribution, a robust outlier test should be used. 

The development of a calibration model is a time consuming process. Not only have the samples to be prepared and measured, but the modelling itself, including data pre-processing, outlier detection, estimation and validation, is not an automated procedure. Once the model is there, changes may occur in the instrumentation or other conditions (temperature, humidity) that require recalibration. Another situation is where a model has been set up for one instrument in a central location and one would like to distribute this model to other instruments within the organization without having to repeat the entire calibration process for all these individual instruments. One wonders whether it is possible to translate the model from one instrument (old or parent or master. A) to the others (new or children or slaves, B). 

We briefly discuss in this section the results of the valence MaxEnt calculation on the noisy data set for L-alanine at 23 K we will denote this calculation with the letter A. The distribution of residuals at the end of the calculation is shown in Figure 5. It is apparent that no gross outliers are present, the calculated structure factor amplitudes being within 5 esd s from the observed values at all resolution ranges. 

The most important methods of explorative data analysis concern the study of the distribution of the data and the recognition of outliers by boxplots (Fig. 8.18), histograms (Fig. 8.19), scatterplot matrices (Fig. 8.20), and various schematic plots. 

---