Normal data distributions

The classification methods discussed in the previous section are all based on statistical tests wliich require normal data distribution. If this condition is not fulfilled the so-called non-probabihstic , non-parametric or heuristic classification techniques must be used. These techniques are also frequently referred to as pattern recognition methods. They are based on geometrical and not on statistical considerations, starting from a representation of the compounds... 

Currie LA (2001) Some case studies of skewed (and other ab-normal) data distributions arising in low-level environmental research. Fresenius J Anal Chem 370 705-718... 

Distributed Control System (DCS) A system that divides process control functions into specific areas interconnected by communications (normally data highways) to form a single entity. It is characterized by digital controllers, typically administered by central operation interfaces and intermittent scanning of the data highway. 

Step 1. From a histogram of the data, partition the data into N components, each roughly corresponding to a mode of the data distribution. This defines the Cj. Set the parameters for prior distributions on the 6 parameters that are conjugate to the likelihoods. For the normal distribution the priors are defined in Eq. (15), so the full prior for the n components is... 

Step 2. Draw a value for each 6 = jj. , aj from the normal posterior distribution for Aj data points with average yi. 

In the above ealeulations of the mean, varianee and standard deviation, we make no prior assumption about the shape of the population distribution. Many of the data distributions eneountered in engineering have a bell-shaped form similar to that showed in Figure 1. In sueh eases, the Normal or Gaussian eontinuous distribution ean be used to model the data using the mean and standard deviation properties. 

Larsen (18-21) has developed averaging time models for use in analysis and interpretation of air quality data. For urban areas where concentrations for a given averaging time tend to be lognormally distributed, that is, where a plot of the log of concentration versus the cumulative frequency of occurrence on a normal frequency distribution scale is nearly linear,... 

The cause of the weaker G dependence must be ascribed to some particular feature of geometry, although exactly what it is has not been found. All of the three bundles involved had their rods supported and correctly positioned by wires wrapped helically around certain of the rods, and it is possible that the wires caused an unfavorable distribution of steam and water. However, it is doubtful that the wire wraps were themselves responsible, since several of the bundles conforming to Eq. (28) were also wire wrapped. (Other devices used for rod supports are suitably spaced grids and ferrules.) The explanation most probably lies in a combination of the effects of the wire wraps with the effects of given rod diameters and rod spacings. For ease of identification, the data that conform with Eq. (28) are hereafter called normal data. 

Means and standard deviations for these distributions were normalized to daily breathing rates (m3/day), and an acceptable range was defined. It was assumed that the "day" represents the duration of time within a working day that chlorpyrifos may be handled by an individual (0.25 to 6.0 hr). It was also assumed that exposures would be negligible for the remainder of the working day following application or other contact. Both the dermal and inhalation exposures were assumed to follow lognormal distributions, which is consistent with common practice for exposure data distributions (for example, in the Pesticide Handlers Exposure Database, PHED). 

A further consideration is that the value of the calculated nonlinearity will depend not only on the function that fits the data, we suspect that it will also depend on the distribution of the data along the X-axis. Therefore, for pedagogical purposes, here we will consider the situation for two common data distributions the uniform distribution and the Normal (Gaussian) distribution. 

As was shown, the conventional method for data reconciliation is that of weighted least squares, in which the adjustments to the data are weighted by the inverse of the measurement noise covariance matrix so that the model constraints are satisfied. The main assumption of the conventional approach is that the errors follow a normal Gaussian distribution. When this assumption is satisfied, conventional approaches provide unbiased estimates of the plant states. The presence of gross errors violates the assumptions in the conventional approach and makes the results invalid. 

Most water-atomized metal particles (powders) have been observed to follow the log-normal size distribution pattern. Relatively narrow size distributions of both fine and coarse particles may be generated by water atomization. A review of published data for droplet size distributions generated by gas and water atomization of a variety of liquid metals and alloys has been made by Lawley,[4] along with presentations of micrographs of surface morphology and internal microstructure of solidified particles. 

The actual noise distribution of the data is often not known. The most common response is to ignore this fact and assume a normal, white distribution of the noise. Even if the assumption of white noise is incorrect, it is still useful to perform the least-squares fit. There is no real alternative and the results are generally not too wrong. 

Section 1.6.2 discussed some theoretical distributions which are defined by more or less complicated mathematical formulae they aim at modeling real empirical data distributions or are used in statistical tests. There are some reasons to believe that phenomena observed in nature indeed follow such distributions. The normal distribution is the most widely used distribution in statistics, and it is fully determined by the mean value p. and the standard deviation a. For practical data these two parameters have to be estimated using the data at hand. This section discusses some possibilities to estimate the mean or central value, and the next section mentions different estimators for the standard deviation or spread the described criteria are fisted in Table 1.2. The choice of the estimator depends mainly on the data quality. Do the data really follow the underlying hypothetical distribution Or are there outliers or extreme values that could influence classical estimators and call for robust counterparts ... 

Like other statistical methods, the user has to be careful with the requirements of a statistical test. For many statistical tests the data have to follow a normal distribution. If this data requirement is not fulfilled, the outcome of the test can be biased and misleading. A possible solution to this problem are nonparametric tests that are much less restrictive with respect to the data distribution. There is a rich literature on... 

Bartlett test. H0 the variances of the data distributions are equal. Requirements normal distribution of all data sets, independent samples. 

In Sections 1.6.3 and 1.6.4, different possibilities were mentioned for estimating the central value and the spread, respectively, of the underlying data distribution. Also in the context of covariance and correlation, we assume an underlying distribution, but now this distribution is no longer univariate but multivariate, for instance a multivariate normal distribution. The covariance matrix X mentioned above expresses the covariance structure of the underlying—unknown—distribution. Now, we can measure n observations (objects) on all m variables, and we assume that these are random samples from the underlying population. The observations are represented as rows in the data matrix X(n x m) with n objects and m variables. The task is then to estimate the covariance matrix from the observed data X. Naturally, there exist several possibilities for estimating X (Table 2.2). The choice should depend on the distribution and quality of the data at hand. If the data follow a multivariate normal distribution, the classical covariance measure (which is the basis for the Pearson correlation) is the best choice. If the data distribution is skewed, one could either transform them to more symmetry and apply the classical methods, or alternatively... 

For the Bayesian discriminant rule, an underlying data distribution fj for each group j =l,..., k is required, which is usually assumed to be a multivariate normal... 

The formidable problems that are associated with the interpretation of LP kinetic data for nonstatistical IM reactions can be entirely avoided if the reactions can be studied in the HPL of kinetic behavior. In the HPL, the energy content of the initially formed species, X and Y, in reaction (2) would be very rapidly changed by collisions with the buffer gas so that the altered species, X and Y, would have normal Boltzmann distributions of energy. Furthermore, those Boltzmann energy distributions would be continuously refreshed as the most energetic X and Y within the distributions move forwards or backwards along the reaction coordinate. The interpretation of rate constants measured in the HPL is expected to be relatively straightforward because conventional transition-state theory can then be applied. 

In SIMCA the distribution of the object in the inner model space is not considered, so the probability density in the inner space is constant and the overall PD appears as shown in Figs. 29, 30 for the enlarged and reduced SIMCA models. In CLASSY, Kernel estimation is used to compute the PD in the inner model space, whereas the errors in the outer space are considered, as in SIMCA, uncorrelated and with normal multivariate distribution, so that the overall distribution, in the inner and outer space of a one-dimensional model, looks like that reported in Fig. 31. Figures 32, 33 show the PD of the bivariate normal distribution and Kernel distribution (ALLOC) for the same data matrix as used for Fig. 31. Although in the data set of French wines no really important differences have been detected between SIMCA (enlarged model), ALLOC and CLASSY, it seems that CLASSY should be chosen when the number of objects is large and the distribution on the components of the inner model space is very different from a rectangular distribution. 

Current methods for supervised pattern recognition are numerous. Typical linear methods are linear discriminant analysis (LDA) based on distance calculation, soft independent modeling of class analogy (SIMCA), which emphasizes similarities within a class, and PLS discriminant analysis (PLS-DA), which performs regression between spectra and class memberships. More advanced methods are based on nonlinear techniques, such as neural networks. Parametric versus nonparametric computations is a further distinction. In parametric techniques such as LDA, statistical parameters of normal sample distribution are used in the decision rules. Such restrictions do not influence nonparametric methods such as SIMCA, which perform more efficiently on NIR data collections. 

Normal probability paper is obtained by adjusting the vertical in such a way that the plot of P versus X is a straight line. Thus, data that follow a normal probability distribution will produce a straight line when plotted on normal probability paper, as shown in Figure 6. 

---