Multivariate least squares regression

One might suspect that fitting all T-variables simultaneously, i.e. in one overall multivariate regression, might make a difference for the regression model. This is not the case, however. To see this, let us state the multivariate (i.e. two or more dependent variables) regression model as  

The total residual sum of squares, taken over all elements of E, achieves its minimum when each column Cj separately has minimum sum of squares. The latter occurs if each (univariate) column of Y is fitted by X in the least-squares way. Consequently, the least-squares minimization of E is obtained if each separate dependent variable is fitted by multiple regression on X. In other words the multivariate regression analysis is essentially identical to a set of univariate regressions. Thus, from a methodological point of view nothing new is added and we may refer to Chapter 10 for a more thorough discussion of theory and application of multiple regression. 

In eqs. (35.13) and (35.14) X may include a column of ones, when an intercept has to be fitted for each response, giving (p+l x m) B. Otherwise, X and Y are supposed to be mean centered, and (pxm) B does not contain a column of intercepts. 

The geometric meaning of Eq. (35.14) is that the best fit is obtained by projecting all responses orthogonally onto the space defined by the columns of X, using the orthogonal projection matrix X(X X) X (see Section 29.8). 

All other methods discussed in this chapter provide such a dimension reduction. They search for the most interesting directions in T-space and/or interesting directions in X-space that are linearly related. They differ in the optimizing criterion that is used to discover those interesting directions. 

---

Reduced rank regression (RRR), also known as redundancy analysis (or PCA on Instrumental Variables), is the combination of multivariate least squares regression and dimension reduction [7]. The idea is that more often than not the dependent K-variables will be correlated. A principal component analysis of Y might indicate that A (A m) PCs may explain Y adequately. Thus, a full set of m... 

Step 2. Multivariate least squares regression of Y on the major A principal components, using either the unit-norm singular vectors or the principal components = XVj j = U[a,S[aj ... 

OLS is synonymous with the following terms least squares regression, linear least squares regression, multiple least squares regression, multivariate least squares regression. OLS provides the best linear unbiased estimator (BLUE) that has the smallest variance among all linear and unbiased estimators. 

Sections 9A.2-9A.6 introduce different multivariate data analysis methods, including Multiple Linear Regression (MLR), Principal Component Analysis (PCA), Principal Component Regression (PCR) and Partial Least Squares regression (PLS). 

A difficulty with Hansch analysis is to decide which parameters and functions of parameters to include in the regression equation. This problem of selection of predictor variables has been discussed in Section 10.3.3. Another problem is due to the high correlations between groups of physicochemical parameters. This is the multicollinearity problem which leads to large variances in the coefficients of the regression equations and, hence, to unreliable predictions (see Section 10.5). It can be remedied by means of multivariate techniques such as principal components regression and partial least squares regression, applications of which are discussed below. 

Multivariate calibration has the aim to develop mathematical models (latent variables) for an optimal prediction of a property y from the variables xi,..., jcm. Most used method in chemometrics is partial least squares regression, PLS (Section 4.7). An important application is for instance the development of quantitative structure—property/activity relationships (QSPR/QSAR). 

Traditionally, the determination of a difference in costs between groups has been made using the Student s r-test or analysis of variance (ANOVA) (univariate analysis) and ordinary least-squares regression (multivariable analysis). The recent proposal of the generalized linear model promises to improve the predictive power of multivariable analyses. 

Human perception of flavor occurs from the combined sensory responses elicited by the proteins, lipids, carbohydrates, and Maillard reaction products in the food. Proteins Chapters 6, 10, 11, 12) and their constituents and sugars Chapter 12) are the primary effects of taste, whereas the lipids Chapters 5, 9) and Maillard products Chapter 4) effect primarily the sense of smell (olfaction). Therefore, when studying a particular food or when designing a new food, it is important to understand the structure-activity relationship of all the variables in the food. To this end, several powerful multivariate statistical techniques have been developed such as factor analysis Chapter 6) and partial least squares regression analysis Chapter 7), to relate a set of independent or "causative" variables to a set of dependent or "effect" variables. Statistical results obtained via these methods are valuable, since they will permit the food... 

Partial least squares regression analysis (PLS) has been used to predict intensity of sweet odour in volatile phenols. This is a relatively new multivariate technique, which has been of particular use in the study of quantitative structure-activity relationships. In recent pharmacological and toxicological studies, PLS has been used to predict activity of molecular structures from a set of physico-chemical molecular descriptors. These techniques will aid understanding of natural flavours and the development of synthetic ones. 

It may be possible to use an array of electrodes containing various enzymes in combination with multivariate statistical analyses (principal component analysis, discriminant analysis, partial least-squares regression) to determine which pesticide(s) the SPCE has been exposed to and possibly even how much, provided sufficient training sets of standards have been measured. The construction methods for such arrays would be the same as described in this protocol, with variations in the amounts of enzyme depending on the inhibition constants of other cholinesterases for the various pesticides of interest. 

Calibration is the process by which a mathematical model relating the response of the analytical instrument (a spectrophotometer in this case) to specific quantities of the samples is constructed. This can be done by using algorithms (usually based on least squares regression) capable of establishing an appropriate mathematical relation such as single absorbance vs. concentration (univariate calibration) or spectra vs. concentration (multivariate calibration). 

Dependencies are mainly investigated by multiple or multivariate regression (one dependent and several independent variables), by multidimensional multivariate regression or partial least squares regression (several dependent and several independent variables), and by the method of simultaneous equations (explicitly allowing for corre-... 

Univariate and multivariate spectroscopy was applied to the analysis of spironolactone in presence of chlorthalidone [17]. Satisfactory results were obtained by partial least squares regression, with the calibration curve being linear over the range 2.92 -14.6 pg/mL. A kinetic-spectrophotometric method was described for the determination of spironolactone and canrenone in urine that also used a partial least-squares regression method [18]. After the compounds were extracted from urine, the spectra were recorded at 400 - 520 nm for 10 minutes at 30 second intervals. The relative error was less than 5%. 

Partial Least Squares Regression (PLS) is a multivariate calibration technique, based on the principles of Latent Variable Regression. Originated in a slightly different form in the field of econometrics, PLS has entered the spectroscopic scene.46,47,48 It is mostly employed for quantitative analysis of mixtures with overlapping bands (e.g. mixture of glucose, fructose and sucrose).49,50... 

Partial Least Squares Regression is a valuable tool in FTIR-spectroscopy, not only for (routine) quantitative analysis of mixtures, but also as a research application. Due to its ability to expose correlations in complex, multivariate data sets, PLS is gaining importance rapidly in spectroscopy-assisted-research. 

Chemometrics is the discipline concerned with the application of statistical and mathematical methods to chemical data [2.18], Multiple linear regression, partial least squares regression and the analysis of the main components are the methods that can be used to design or select optimal measurement procedures and experiments, or to provide maximum relevant chemical information from chemical data analysis. Common areas addressed by chemometrics include multivariate calibration, visualisation of data and pattern recognition. Biometrics is concerned with the application of statistical and mathematical methods to biological or biochemical data. 

Due to the large number of descriptors (commonly 15,000 - 20,000 for each field), the multivariate regression analysis is usually performed by partial least squares regression (PLS), with or without - variable selection. Moreover, a similarity matrix can be calculated from distance functions based on interaction fields between pairs of molecules. 

---