Data distribution skewness

If the data distribution is extremely skewed it is advisable to transform the data to approach more symmetry. The visual impression of skewed data is dominated by extreme values which often make it impossible to inspect the main part of the data. Also the estimation of statistical parameters like mean or standard deviation can become unreliable for extremely skewed data. Depending on the form of skewness (left skewed or right skewed), a log-transformation or power transformation (square root, square, etc.) can be helpful in symmetrizing the distribution. 

The standard deviation is very sensitive to outliers if the data are skewed, not only the mean will be biased, but also s will be even more biased because squared deviations are used. In the case of normal or approximately normal distributions,, v is the best measure of spread because it is the most precise estimator for standard deviation is often uncritically used instead of robust measures for the spread. 

As already noted in Section 1.6.1, many statistical estimators rely on symmetry of the data distribution. For example, the standard deviation can be severely increased if the data distribution is much skewed. It is thus often highly recommended to first transform the data to approach a better symmetry. Unfortunately, this has to be done for each variable separately, because it is not sure if one and the same transformation will be useful for symmetrizing different variables. For right-skewed data, the log transformation is often useful (that means taking the logarithm of the data values). More flexible is the power transformation which uses a power p to transform values x into xp. The value of p has to be optimized for each variable any real number is reasonable for p, except p 0 where a log-transformation has to be taken. A slightly modified version of the power transformation is the Box Cox transformation, defined as... 

In Sections 1.6.3 and 1.6.4, different possibilities were mentioned for estimating the central value and the spread, respectively, of the underlying data distribution. Also in the context of covariance and correlation, we assume an underlying distribution, but now this distribution is no longer univariate but multivariate, for instance a multivariate normal distribution. The covariance matrix X mentioned above expresses the covariance structure of the underlying—unknown—distribution. Now, we can measure n observations (objects) on all m variables, and we assume that these are random samples from the underlying population. The observations are represented as rows in the data matrix X(n x m) with n objects and m variables. The task is then to estimate the covariance matrix from the observed data X. Naturally, there exist several possibilities for estimating X (Table 2.2). The choice should depend on the distribution and quality of the data at hand. If the data follow a multivariate normal distribution, the classical covariance measure (which is the basis for the Pearson correlation) is the best choice. If the data distribution is skewed, one could either transform them to more symmetry and apply the classical methods, or alternatively... 

The goal of dimension reduction can be best met with PCA if the data distribution is elliptically symmetric around the center. It will not work well as a dimension reduction tool for highly skewed data. Figure 3.9 (left) shows skewed autoscaled... 

Apply a transformation to the data to make the transformed data normal. If the distribution is skewed to the right, one might try a log, inverse, square root, or cube root transformation of the data to make the data normal. If the data are skewed to the left, an exponential, squared, or cubed transformation might be applied. A histogram can be applied before and after the transformation to assess the ability of the transformation to make the data normal. It is important to remember that the transformation must be applied to the USL and LSL, in addition to the data, before computing the capability index of interest. 

Use the index as is as a relative measure and note in report to this effect. Even if the data are nonnormal but symmetrical, the calculations are generally reasonable. However, if the data distribution is heavily skewed, one should be hesitant to use this particular option. 

The results in that table are consistent with the picture of chain metalation since the data demonstrate the formation of multiple polystyrene blocks. The grafted polystyrene recovered has a rather broad molecular-weight distribution skewed toward the low-molecular-weight range. The high value of catalyst efficiency as determined in polymers 1 and 3 could result from either limitations in the analysis of molecular-weight distribution or from some form of chain-transfer mechanism. 

Prior to a formal analysis, a database should be examined for any unusual characteristics of the data distribution. A database may have some number of outliers, an inherent nonnormal or skewed distribution, or a bimodal character due to the presence of two separate underlying distributions. Most tests for normality are intended for fairly large sample sizes of the order of 15 or more. Smaller databases may be reviewed for unusual characteristics by way of the usual statistical algorithms available with spreadsheets. Tests for normality are listed in Part 2. 

Questions that can be answered by the histogram are What is the shape of the distribution What is the mean of the distribution How much dispersion of the data is there Is the distribution symmetrical Is the distribution skewed Is there only one peak Is the distribution cliff-like Does the distribution look like a cogwheel What is the relationship of the distribution with the customer s specifications ... 

Another problem which is frequently encountered in the distribution of data is the presence of outliers. Consider the data shown in Table 3.1 where calculated values of electrophilic superdelocalizability (ESDLIO) are given for a set of analogues of antimycin Al, compounds which kill human parasitic worms, Dipetalonema vitae. The mean and standard deviation of this variable give no clues as to how well it is distributed and the skewness and kurtosis values of —3.15 and 10.65 respectively might not suggest that it deviates too seriously from normal. A frequency distribution for this variable, however, reveals the presence of a single extreme value (compound 14) as shown in Fig. 3.5. This data was analysed by multiple linear regression (discussed further in Chapter 6), which is a parametric method based on the normal distribution. The presence of this outlier had quite profound effects on the analysis, which could have been avoided if the data distribution had been checked at the outset (particularly by the present author). Outliers can be very informative and should not simply be discarded as so frequently happens. If an outlier is found in one of the descriptor variables (physicochemical data), then it may show... 

The graphics window immediately displays Fig. 3.4. It shows the result of the relative error data distribution normality test. The discrete points are very close to the skew straight lines in Fig. 3.4, which means the graphic is linear, it can be concluded that the relative error approximately obeys the rule of normal distribution of data. 

The skew, denoted by y, measures the amount of asymmetry in the distribution. Skewness is determined by examining the relationship in the clustering of extreme values, that is, the tails. If more of the data set is clustered towards the smaller extreme values, then it is said that the system has positive or right skewness. On the other hand, if the data set is clustered towards the larger extreme values, then it is said that the system has negative or left skewness. The skew of a data set can be computed as... 

Currie LA (2001) Some case studies of skewed (and other ab-normal) data distributions arising in low-level environmental research. Fresenius J Anal Chem 370 705-718... 

Many distributions occurring in business situations are not symmetric but skewed, and the normal distribution cui ve is not a good fit. However, when data are based on estimates of future trends, the accuracy of the normal approximation is usually acceptable. This is particularly the case as the number of component variables Xi, Xo, etc., in Eq. (9-74) increases. Although distributions of the individual variables (xi, Xo, etc.) may be skewed, the distribution of the property or variable c tends to approach the normal distribution. 

Data that is not evenly distributed is better represented by a skewed distribution such as the Lognormal or Weibull distribution. The empirically based Weibull distribution is frequently used to model engineering distributions because it is flexible (Rice, 1997). For example, the Weibull distribution can be used to replace the Normal distribution. Like the Lognormal, the 2-parameter Weibull distribution also has a zero threshold. But with increasing numbers of parameters, statistical models are more flexible as to the distributions that they may represent, and so the 3-parameter Weibull, which includes a minimum expected value, is very adaptable in modelling many types of data. A 3-parameter Lognormal is also available as discussed in Bury (1999). 

The variability or spread of the data does not always take the form of the true Normal distribution of course. There can be skewness in the shape of the distribution curve, this means the distribution is not symmetrical, leading to the distribution appearing lopsided . However, the approach is adequate for distributions which are fairly symmetrical about the tolerance limits. But what about when the distribution mean is not symmetrical about the tolerance limits A second index, Cp, is used to accommodate this shift or drift in the process. It has been estimated that over a very large number of lots produced, the mean could expect to drift about 1.5cr (standard deviations) from the target value or the centre of the tolerance limits and is caused by some problem in the process, for example tooling settings have been altered or a new supplier for the material being processed. 

The average nonuniform permeability is spatially dependent. For a homogeneous but nonuniform medium, the average permeability is the correct mean (first moment) of the permeability distribution function. Permeability for a nonuniform medium is usually skewed. Most data for nonuniform permeability show permeability to be distributed log-normally. The correct average for a homogeneous, nonuniform permeability, assuming it is distributed log-normally, is the geometric mean, defined as ... 

The shape of a frequency distribution curve will depend on how the size increments were chosen. With the common methods for specifying increments, the curve will usually take the general form of a skewed probability curve with a single peak. However, it may also have multiple peaks, as in Fig 2, There are various analytical relationships for representing size distribution. One or the other may give a better fit of data in a particular instance. There are times, however, when analytical convenience may justify one. The log-probability relationship is particularly useful in this respect... 

If, as is usual, standard deviations are inserted for e cr has a similar interpretation. Examples are provided in Refs. 23, 75, 89, 93, 142, 169-171 and in Section 4.17. In complex data evaluation schemes, even if all inputs have Gaussian distribution functions, the output can be skewed,however. 

No version of micellar entry theory has been proposed, which is able to explain the experimentally observed leveling off of the particle number at high and low surfactant concentrations where micelles do not even exist. There is a number of additional experimental data that refute micellar entry such as the positively skewed early time particle size distribution (22.), and the formation of Liesegang rings (30). Therefore it is inappropriate to include micellar entry as a particle formation mechanism in EPM until there is sufficient evidence to do so. 

The conclusion that highly vibrationally excited H2 correlated with low-7 CO represents a new mechanistic pathway, and the elucidation of that pathway, is greatly facilitated by comparison with quasiclassical trajectory calculations of Bowman and co-workers [8, 53] performed on a PES fit to high level electronic structure calculations [54]. The correlated H2 / CO state distributions from these trajectories, shown as the dashed lines in Fig. 8, show reasonably good agreement with the data. Analysis of the trajectories confirms that the H2(v = 0—4) population represents dissociation over the skewed transition state, as expected. 

Secondly, knowledge of the estimation variance E [P(2c)-P (2c)] falls short of providing the confidence Interval attached to the estimate p (3c). Assuming a normal distribution of error In the presence of an Initially heavily skewed distribution of data with strong spatial correlation Is not a viable answer. In the absence of a distribution of error, the estimation or "krlglng variance o (3c) provides but a relative assessment of error the error at location x Is likely to be greater than that at location 2 " if o (2c)>o (2c ). Iso-varlance maps such as that of Figure 1 tend to only mimic data-posltlon maps with bull s-eyes around data locations. 

Consider, for example, a site characterized by a highly skewed distribution of pollutant concentrations, as apparent In the histogram of data values of Figure 2a. These values present a coefficient of... 

In this paper, we discuss studies based on comparison with background measurements that may have a skew distribution. We discuss below the design of such a study. The design is intended to insure that the model for the comparison is valid and that the amount of skewness is minimized. Subsequently, we present a statistical method for the comparison of the background measurements with the largest of the measurements from the suspected region. This method, which is based on the use of power transformations to achieve normality, is original in that it takes into account estimation of the transformation from the data. 

---