Multivariate data covariance

Scaling is a very important operation in multivariate data analysis and we will treat the issues of scaling and normalisation in much more detail in Chapter 31. It should be noted that scaling has no impact (except when the log transform is used) on the correlation coefficient and that the Mahalanobis distance is also scale-invariant because the C matrix contains covariance (related to correlation) and variances (related to standard deviation). 

In multivariate data analysis frequently the covariance matrix S is used... 

Statistical properties of a data set can be preserved only if the statistical distribution of the data is assumed. PCA assumes the multivariate data are described by a Gaussian distribution, and then PCA is calculated considering only the second moment of the probability distribution of the data (covariance matrix). Indeed, for normally distributed data the covariance matrix (XTX) completely describes the data, once they are zero-centered. From a geometric point of view, any covariance matrix, since it is a symmetric matrix, is associated with a hyper-ellipsoid in N dimensional space. PCA corresponds to a coordinate rotation from the natural sensor space axis to a novel axis basis formed by the principal... 

Principal component analysis (PCA) is aimed at explaining the covariance structure of multivariate data through a reduction of the whole data set to a smaller number of independent variables. We assume that an m-point sample is represented by the nxm matrix X which collects i=l,...,m observations (measurements) xt of a column-vector x with j=, ...,n elements (e.g., the measurements of n=10 oxide weight percents in m = 50 rocks). Let x be the mean vector and Sx the nxn covariance matrix of this sample... 

In Chapter 2, we approach multivariate data analysis. This chapter will be helpful for getting familiar with the matrix notation used throughout the book. The art of statistical data analysis starts with an appropriate data preprocessing, and Section 2.2 mentions some basic transformation methods. The multivariate data information is contained in the covariance and distance matrix, respectively. Therefore, Sections... 

FIGURE 2.9 Basic statistics of multivariate data and covariance matrix. xT, transposed mean vector vT, transposed variance vector vXOtal. total variance (sum of variances vb. .., vm). C is the sample covariance matrix calculated from mean-centered X. 

If it can be assumed that the multivariate data follow a multivariate normal distribution with a certain mean and covariance matrix, then it can be shown that the squared Mahalanobis distance approximately follows a chi-square distribution... 

The distance between object points is considered as an inverse similarity of the objects. This similarity depends on the variables used and on the distance measure applied. The distances between the objects can be collected in a distance matrk. Most used is the euclidean distance, which is the commonly used distance, extended to more than two or three dimensions. Other distance measures (city block distance, correlation coefficient) can be applied of special importance is the mahalanobis distance which considers the spatial distribution of the object points (the correlation between the variables). Based on the Mahalanobis distance, multivariate outliers can be identified. The Mahalanobis distance is based on the covariance matrix of X this matrix plays a central role in multivariate data analysis and should be estimated by appropriate methods—mostly robust methods are adequate. 

On the other hand, factor analysis involves other manipulations of the eigen vectors and aims to gain insight into the structure of a multidimensional data set. The use of this technique was first proposed in biological structure-activity relationship (i. e., SAR) and illustrated with an analysis of the activities of 21 di-phenylaminopropanol derivatives in 11 biological tests [116-119, 289]. This method has been more commonly used to determine the intrinsic dimensionality of certain experimentally determined chemical properties which are the number of fundamental factors required to account for the variance. One of the best FA techniques is the Q-mode, which is based on grouping a multivariate data set based on the data structure defined by the similarity between samples [1, 313-316]. It is devoted exclusively to the interpretation of the inter-object relationships in a data set, rather than to the inter-variable (or covariance) relationships explored with R-mode factor analysis. The measure of similarity used is the cosine theta matrix, i. e., the matrix whose elements are the cosine of the angles between all sample pairs [1,313-316]. 

PCA is a statistical technique that has been used ubiquitously in multivariate data analysis." Given a set of input vectors described by partially cross-correlated variables, the PCA will transform them into a set that is described by a smaller number of orthogonal variables, the principle components, without a significant loss in the variance of the data. The principle components correspond to the eigenvectors of the covariance matrix, m, a symmetric matrix that contains the variances of the variables in its diagonal elements and the covariances in its off-diagonal elements (15) ... 

Each measure of an analysed variable, or variate, may be considered independent. By summing elements of each column vector the mean and standard deviation for each variate can be calculated (Table 7). Although these operations reduce the size of the data set to a smaller set of descriptive statistics, much relevant information can be lost. When performing any multivariate data analysis it is important that the variates are not considered in isolation but are combined to provide as complete a description of the total system as possible. Interaction between variables can be as important as the individual mean values and the distributions of the individual variates. Variables which exhibit no interaction are said to be statistically independent, as a change in the value in one variable cannot be predicted by a change in another measured variable. In many cases in analytical science the variates are not statistically independent, and some measure of their interaction is required in order to interpret the data and characterize the samples. The degree or extent of this interaction between variables can be estimated by calculating their covariances, the subject of the next section. 

Just as variance describes the spread of data about its mean value for a single variable, so the distribution of multivariate data can be assessed from the covariance. The procedure employed for the calculation of variance can be extended to multivariate analysis by computing the extent of the mutual variability of the variates about some common mean. The measure of this interaction is the covariance. 

Nystrom, A., Andersson, P.M. and Lundstedt, T. (2000) Multivariate data analysis of topographically modified a-melanotropin analogues using auto and cross auto covariances (ACC). Quant. Struct. -Act. Relat., 19, 264—269. 

The root of these methods is the decomposition of multivariate data into a series of orthogonal factors, eigenvectors, also called abstract factors. These factors are linear combinations of a set of orthogonal basis vectors that are the eigenvectors of the variance-covariance matrix (X X) of the original data matrix. The eigenvalues of this variance-covariance matrix are the solutions Kz,. . ., of the determinantal equation... 

All variances and covariances of a multivariate data set can be arranged in a symmetric matrix, with the variances located in the main diagonal. This covariance matrix C plays an important role in multivariate statistics it describes the dispersion of multivariate data. Highly correlating (collinear) features - a frequent situation in chemical data - make C singular and then the inversion of C (which is necessary in some methods)... 

As mentioned above, there are several well-known methods of multivariate data analysis (Juts et al, 2(XX)), but peihaps the most popular is PCA (JoUife, 1986). This feature extraction method consists in projecting the A-dimensional data set (in this case N is the number of sensors) onto a new base of the same dimension N, but now defined by the eigenvectors of the covariance or the correlation matrix of the data set. The components (projections) of the original data vectors onto this new base are the so-called Principal Components,... 

These various covariance models are Inferred directly from the corresponding indicator data i(3 z ), i-l,...,N. The indicator kriging approach is said to be "non-parametric, in the sense that it draws solely from the data, not from any multivariate distribution hypothesis, as was the case for the multi- -normal approach. 

In Sections 1.6.3 and 1.6.4, different possibilities were mentioned for estimating the central value and the spread, respectively, of the underlying data distribution. Also in the context of covariance and correlation, we assume an underlying distribution, but now this distribution is no longer univariate but multivariate, for instance a multivariate normal distribution. The covariance matrix X mentioned above expresses the covariance structure of the underlying—unknown—distribution. Now, we can measure n observations (objects) on all m variables, and we assume that these are random samples from the underlying population. The observations are represented as rows in the data matrix X(n x m) with n objects and m variables. The task is then to estimate the covariance matrix from the observed data X. Naturally, there exist several possibilities for estimating X (Table 2.2). The choice should depend on the distribution and quality of the data at hand. If the data follow a multivariate normal distribution, the classical covariance measure (which is the basis for the Pearson correlation) is the best choice. If the data distribution is skewed, one could either transform them to more symmetry and apply the classical methods, or alternatively... 

The Mahalanobis distance used for multivariate outlier detection relies on the estimation of a covariance matrix (see Section 2.3.2), in this case preferably a robust covariance matrix. However, robust covariance estimators like the MCD estimator need more objects than variables, and thus for many applications with m>n this approach is not possible. For this situation, other multivariate outlier detection techniques can be used like a method based on robustified principal components (Filzmoser et al. 2008). The R code to apply this method on a data set X is as follows ... 

If the assumptions (multivariate normal distributions with equal group covariance matrices) are fulfilled, the Fisher rule gives the same result as the Bayesian rule. However, there is an interesting aspect for the Fisher rule in the context of visualization, because this formulation allows for dimension reduction. By projecting the data... 

Principal Component Analysis (PCA) is the most popular technique of multivariate analysis used in environmental chemistry and toxicology [313-316]. Both PCA and factor analysis (FA) aim to reduce the dimensionality of a set of data but the approaches to do so are different for the two techniques. Each provides a different insight into the data structure, with PCA concentrating on explaining the diagonal elements of the covariance matrix, while FA the off-diagonal elements [313, 316-319]. Theoretically, PCA corresponds to a mathematical decomposition of the descriptor matrix,X, into means (xk), scores (fia), loadings (pak), and residuals (eik), which can be expressed as... 

How can multivariate methods be used to avoid the problems associated with the OVAT approach In general, multivariate methods use the information contained in the relation between the variables (correlations or covariances) and therefore data like those in Figure 6.3 present no problem. The risk of type I errors is kept under control in multivariate analysis by considering all variables simultaneously. To consider all variables simultaneously involves a... 

The task remains to determine the mean annual enthalpy from plant physiognomy. An analysis is presented relating foliar physiognomic characters to mean annual values of enthalpy, temperature, specific humidity, and relative humidity that exploits the method and data in the Climate-Leaf Analysis Multivariate Program (Wolfe 1993). From present-day plant data collected from North America, Puerto Rico, and Japan, the leaf parameters are searched for linear combinations of the foliar characteristics that covary with the local climates. By doing so, the foliar characteristics can be determined that covary with one another and which best correlate with climate parameters. 

---