Regression linear multiple

For multiple-descriptor data sets, one could use the methods in Section 17.3 to derive a correlation between y and xl, between y and x2, between y and x3, etc., to find the k data set giving the best correlation with y. It is very likely, however, that one of the other X variabled can describe part of the y variation, which is not described by k, and a third x variable describing some of the remaining variation, etc. Since the x vectors may be internally correlated, however, the second most important x vector found in a one-to-one correlation is not necessarily the most important once the k vector has been included. 

In order to use all the information in the x variables, a Multiple Linear Regression (MLR) of the type indicated in eq. (17.13) can be attempted. 

Note that each data set (y and x ) is a vector containing N data points, and the constant corresponding to b in eq. (17.5) is eliminated if the data are centred with a mean value of zero. Since the expansion coefficients are multiplied directly onto the x variables, MLR is independent of a possible scaling of the x data (a scaling just affects the magnitude of the a coefficients but does not change the correlation). 

The number of fitting parameters is M, and N must therefore be larger than or equal to M, and in practice one should not attempt fitting unless N 5M, as overfitting is otherwise a strong possibility. The fitting coefficients contained in the a vector can be obtained from the generalized inverse (Section 16.2) of the X matrix. 

This procedure works fine as long as there are relatively few x variables that are not internally correlated. In reality, however, it is very likely that some of the x vectors describe almost the same variation, and in such cases there is a large risk of overfitting the data. This can also be seen from the solution vector in eq. (17.14), the (X X) matrix has dimension M x M, and will be poorly conditioned (Section 16.2) if the X vectors are (almost) linearly dependent. Note that the presence of (experimental) noise in the x data can often mask the linear dependence, and MLR methods are therefore sensitive to noisy x data. 

Robust Regression Robust regression is based at an iterative Robustness means 

Computation is started by conventional regression analysis. Subsequently, the residuals are determined and the weights for a value of 1 are calculated for each observation. Regression is repeated as long as the parameters change only by a predefined small amount. The smaller the value for k, the more the residuals are weighted down. 

In the case of multivariate modeling, several independent as well as several dependent variables may operate. Out of the many regression methods, we will learn about the conventional method of ordinary least squares (OLS) as well as methods that are based on biased parameter estimations reducing simultaneously the dimensionality of the regression problem, that is, principal component regression (PCR) and the partial least squares (PLS) method. 

As an example of the application of these methods, spectro-metric multicomponent analysis will be considered, leading to an introduction to regression diagnostics in multiple linear regression. 

The general least squares problem that relates a matrix of dependent variables y to a matrix of independent variables X can be stated as follows  

Multiple linear regression is an extension of simple linear regression by the inclusion of extra independent variables 

What about an assessment of the significance of the fit of a multiple regression equation (or simple regression) to a set of data A guide to the overall significance of a regression model can be obtained by calculation of a quantity called the F statistic. This is simply the ratio of the explained mean square (MSB) to the residual mean square (MSR) 

An F statistic is used by looking up a standard value for F from a table of F statistics and comparing the calculated value with the tabulated value. If the calculated value is greater than the tabulated value, the equation is significant at that particular confidence level. F tables normally have 

The squared multiple correlation coefficient gives a measure of how well a regression model fits the data and the F statistic gives a measure of the overall significance of the fit. What about the significance of individual terms This can be assessed by calculation of the standard error of the regression coefficients, a measure of how much of the dependent variable 

Another useful statistic that can be calculated to characterize the fit of a regression model to a set of data is the standard error of prediction. This gives a measure of how well one might expect to be able to make 

In MLR, the model is a straightforward linear combination of descriptors or functions of descriptors. Because MLR is based on a linear equation, it is called a linear model (as opposed to the non-linear models to be discussed later in this 

This inherent simplicity is valuable in two ways. First, the number of parameters to be fit is minimal, avoiding the common problem of trying to solve too many variables with too little data. This problem is often referred to as the Curse of Dimensionality and results in a degradation of a model s generalizabiUty. Second, MLR models are easy to interpret. Each descriptor used is accompanied by a coefficient and a sign, and this information provides the relative weight and direction of each descriptor s contribution to the property of interest. 

MLR is most suited to the modeling of simple physical properties for which the contributions of the descriptors are more likely to be independent than cooperative. It is also the preferred technique when the amount of available data is extremely limited. MLR is less suited for modeling complex physical or biological processes, as these tend to be non-Unear in nature. 

An extension of linear regression, multiple linear regression (MLR) involves the use of more than one independent variable. Such a technique can be very effective if it is suspected that the information contained in a single dependent variable (x) is insufficient to explain the variation in the independent variable (y). In PAT, such a situation often occurs because of the inability to find a single analyzer response variable that is affected solely by the property of interest, without interference from other properties or effects. In such cases, it is necessary to use more than one response variable from the analyzer to build an effective calibration model, so that the effects of such interferences can be compensated. 

The multiple linear regression model is simply an extension of the linear regression model (Equation 12.7), and is given below  

The difference here is that X is a matrix that contains responses from M ( 1) different x variables, and b contains M regression coefficients for each of the x variables. As for linear regression, the coefficients for MLR (b) are determined using the least-squares method  

At this point, it is important to note two limitations of the MLR method  

In real applications, where there is noise in the data, it is rare to have two x variables exactly correlated to one another. However, a high degree of correlation between any two x variables leads to an unstable matrix inversion, which resnlts in a large amonnt of noise being introduced to the regression coefficients. Therefore, one mnst be very wary of intercorrelation between x variables when nsing the MLR method. 

Note that this formula involves the inversion of XjX, the covariance matrix of X. If this inverse does not exist, then A cannot be calculated by means of MLR. Singularity of XjX corresponds to the linear dependence of a subset of the X variables. So, if X contains such relationships, MLR cannot be applied. In practice, it is rare to have exact linear dependence, since the measurement errors will tend to preclude this. However, near linearity will tend to make the inverse numerically unstable and subject to large errors, so much the same effect occurs. Another formulation of the 

MLR equation can be obtained by expressing it in terms of the matrix of eigenvectors, P, and the diagonal matrix of eigenvalues, diag(A/), as follows  

If X contains (almost) collinear variables, then A/ will be (close to) zero for some U giving an eigenvalue matrix containing large values which are susceptible to small alterations in the calibration data. In our application, collinearity of the X variables is to be expected, since harmonic responses at similar frequencies and amplitudes are likely to be similar. 

Forward selection. Here the model is built up by adding variables in X, one at a time, to the model. The variables are added by choosing those which improve the model most at each step according to some statistical test. Variables which are collinear with a variable already added will contribute little to the quality of a subsequent model. Likewise irrelevant variables contribute little. Such variables will not, therefore, be added. Variables are added until a stopping criterion is reached, for example when the improvement afforded by addition of the next variable drops below a threshold value. 

Backward elimination. Here, all the variables are used to form an initial model. Variables are then selected for deletion, once again by using a statistical test and threshold to determine when to stop. In the case of BE, exactly collinear variables must be eliminated before the initial model is formed. 

The analysis presented for fitting a straight line to experimental data is easily extended to a curve or to a system with more than one independent variable on the basis of the principle of least squares. 

Consider first a system where the observed quantity y depends on two independent variables, and Xj. The resulting relationship describes the equation of a plane in three dimensions and may be written as 

Three parameters, namely, a, b, and c are required to specify the relationship. The equation describing the principle of least squares is 

The sum A is now minimized with respect to each of the adjustable parameters. This leads to the following three equations  

After simplification, and using the notation introduced earlier, the resulting normal equations are 

---

To gain insight into chemometric methods such as correlation analysis, Multiple Linear Regression Analysis, Principal Component Analysis, Principal Component Regression, and Partial Least Squares regression/Projection to Latent Structures... 

Kohonen network Conceptual clustering Principal Component Analysis (PCA) Decision trees Partial Least Squares (PLS) Multiple Linear Regression (MLR) Counter-propagation networks Back-propagation networks Genetic algorithms (GA)... 

While simple linear regression uses only one independent variable for modeling, multiple linear regression uses more variables. 

Multiple linear regression (MLR) models a linear relationship between a dependent variable and one or more independent variables. 

Besides these LFER-based models, approaches have been developed using whole-molecule descriptors and learning algorithms other then multiple linear regression (see Section 10.1.2). 

Step S Building a Multiple Linear Regression Analysis (MLRA) Model... 

Multiple linear regression analysis is a widely used method, in this case assuming that a linear relationship exists between solubility and the 18 input variables. The multilinear regression analy.si.s was performed by the SPSS program [30]. The training set was used to build a model, and the test set was used for the prediction of solubility. The MLRA model provided, for the training set, a correlation coefficient r = 0.92 and a standard deviation of, s = 0,78, and for the test set, r = 0.94 and s = 0.68. 

Alternatives to Multiple Linear Regression Discriminant Analysis, Neural Networks and Classification Methods... 

Multiple linear regression is strictly a parametric supervised learning technique. A parametric technique is one which assumes that the variables conform to some distribution (often the Gaussian distribution) the properties of the distribution are assumed in the underlying statistical method. A non-parametric technique does not rely upon the assumption of any particular distribution. A supervised learning method is one which uses information about the dependent variable to derive the model. An unsupervised learning method does not. Thus cluster analysis, principal components analysis and factor analysis are all examples of unsupervised learning techniques. 

Using a multiple linear regression computer program, a set of substituent parameters was calculated for a number of the most commonly occurring groups. The calculated substituent effects allow a prediction of the chemical shifts of the exterior and central carbon atoms of the allene with standard deviations of l.Sand 2.3 ppm, respectively Although most compounds were measured as neat liquids, for a number of compounds duplicatel measurements were obtained in various solvents. 

Most of the 2D QSAR methods are based on graph theoretic indices, which have been extensively studied by Randic [29] and Kier and Hall [30,31]. Although these structural indices represent different aspects of molecular structures, their physicochemical meaning is unclear. Successful applications of these topological indices combined with multiple linear regression (MLR) analysis are summarized in Ref. 31. On the other hand, parameters derived from various experiments through chemometric methods have also been used in the study of peptide QSAR, where partial least square (PLS) [32] analysis has been employed [33]. 

It may be necessary and possible to achieve a good Brf nsted relationship by adding another term to the equation, as Toney and Kirsch did in correlating the effects of various amines on the catalytic activity of a mutant enzyme. A simple Brf nsted plot failed, but a multiple linear regression on the variables pKa and molecular volume (of the amines) was successful. 

Numerous authors have devised multiple linear regression approaches to the eorrelation of solvent effects, the intent being to widen the applieability of the eorrelation and to develop insight into the moleeular factors controlling the eorrelated proeess. For example, Zilian treated polarity as a eombination of effeets measured by molar refraction, AN, and DN. Koppel and Palm write... 

In the above paragraphs we saw that multiple linear regression analysis on equations of the form... 

We now consider a type of analysis in which the data (which may consist of solvent properties or of solvent effects on rates, equilibria, and spectra) again are expressed as a linear combination of products as in Eq. (8-81), but now the statistical treatment yields estimates of both a, and jc,. This method is called principal component analysis or factor analysis. A key difference between multiple linear regression analysis and principal component analysis (in the chemical setting) is that regression analysis adopts chemical models a priori, whereas in factor analysis the chemical significance of the factors emerges (if desired) as a result of the analysis. We will not explore the statistical procedure, but will cite some results. We have already encountered examples in Section 8.2 on the classification of solvents and in the present section in the form of the Swain et al. treatment leading to Eq. (8-74). 

In contrast to points (l)-(3) of discussion, the effect of ion association on the conductivity of concentrated solutions is proven only with difficulty. Previously published reviews refer mainly to the permittivity of the solvent or quote some theoretical expressions for association constants which only take permittivity and distance parameters into account. Ue and Mori [212] in a recent publication tried a multiple linear regression based Eq. (62)... 

We will explore the two major families of chemometric quantitative calibration techniques that are most commonly employed the Multiple Linear Regression (MLR) techniques, and the Factor-Based Techniques. Within each family, we will review the various methods commonly employed, learn how to develop and test calibrations, and how to use the calibrations to estimate, or predict, the properties of unknown samples. We will consider the advantages and limitations of each method as well as some of the tricks and pitfalls associated with their use. While our emphasis will be on quantitative analysis, we will also touch on how these techniques are used for qualitative analysis, classification, and discriminative analysis. 

Classical least-squares (CLS), sometimes known as K-matrix calibration, is so called because, originally, it involved the application of multiple linear regression (MLR) to the classical expression of the Beer-Lambert Law of spectroscopy ... 

Multiple Linear Regression (MLR), Classical Least-Squares (CLS, K-matrix), Inverse Least-Squares (ILS, P-matrix)... 

Gonzalez, A. G., TWo Level Factorial Experimental Designs Based on Multiple Linear Regression Models A Tutorial Digest Illustrated by Case Studies, Analytica Chimica Acta 360, 1998, 227-241. 

Experimental polymer rheology data obtained in a capillary rheometer at different temperatures is used to determine the unknown coefficients in Equations 11 - 12. Multiple linear regression is used for parameter estimation. The values of these coefficients for three different polymers is shown in Table I. The polymer rheology is shown in Figures 2 - 4. 

Sathe PM, Venitz J. Comparison of neural networks and multiple linear regression as dissolution predictors. Drug Dev Ind Pharm 2003 29 349-55. 

---