Linear regression, 5.24

Linear regression [1,22,23] is typically used to build a linear model that relates a single independent variable 

Note Each set consists of an analyzer measurement and its matching known analyte concentration obtained from an off-line wet-chemistry method used for illustrating linear regression. 

As the name suggests, the residuals (f) contain the variation in y that cannot be explained by the model. 

Given the matching sets of measured data, x and y, it is now possible to estimate the model regression coefficients b. Assuming that the model errors (values in f) are Gaussian-distributed, it can be proven that the value of b that minimizes the sum of squares of the model errors is determined using the least squares method  

There are several properties of linear regression that should be noted. First, it is assumed that the model errors are normally distributed. Second, the relationship between the x and y variables is assumed to be linear. In analytical chemistry, the first assumption is generally a reasonable one. However, the second assumption might not be sufficiently accurate in many situations, especially if a strong nonlinear relationship is suspected between x and y. There are some nonlinear remedies to deal with such situations, and these will be discussed later. 

The nonlinear regression techniques discussed in Section 19.3.2 are extensions of the linear regression formalism described below. A more detailed description is provided by Press et al.  

In this case, multiple linear regression may be used to determine the significance of the demographic variables, which are often called covariates. The model may then be formulated as 

As in simple linear regression, the same assumptions are made s is normally distributed, uncorrelated with each other and have mean zero with variance u2. In addition, the covariates are measured without error. In matrix notation then, the general linear model can be written as 

Rarely in a single experiment is one dependent variable and one independent variable collected. More often, many dependent variables and many independent variables are collected. Then, a scientist may wish to use the independent variables to explain a particular dependent 

In this case, x is a n x (k + 1) matrix of independent variables where the first column of the matrix is a column of ones, which is necessary for inclusion of the intercept in the model, and k is the number of independent variables. An estimate of MSE is obtained by 

Similarly, a (1 a) 100% confidence band for the response function at any x can be developed using 

Matlab recognises the importance of linear regression calculations and introduced a very elegant and useful notation the / forward- and backslash operators, see p.l 17-118. 

Note that the term Linear Regression is somewhat misleading. It is not restricted to the task of just fitting a straight line to some data. While this task is an example of linear regression, the expression covers much more. However, to start with, we return to the task of the straight line fit. 

In this chapter, we start by describing linear regression, which is a method for determining parameters in a model. The accuracy of the parameters can be estimated by confidence intervals and regions, which will be discussed in Section 7.5. Correlation between parameters is often a major problem for large mathematical models, and the determination of so-called correlation matrices will be described. In more complex chemical engineering models, non-hnear regression is required, and this is also described in this chapter. 

Regression analysis is a statistical method for determining parameters in models. The simplest form is a first-order straight-hne model, which can be described by 

In this chapter, the vectors and matrices will be written in bold itahc style. The straight-hne model can be divided into two parts die deterministic part, E y), and 

A more general equation for the linear regression is the so-called multiple linear regression that is described by 

Another set of models, which are also linear with respect to the parameters, are the so-called sinusoidal models  

From these two equations, it is obvious that the derivatives depend on the parameters, and so the resulting system is nonlinear. 

In linear regression, it is assumed that the function of interest can be written as  

Another critical building block for chemometrics is the technique of linear regression.1,20,21 In chemometrics, this technique is typically used to build a linear model that relates an independent variable (X) to a dependent variable (Y). For example, in PAC, one 

Another assumption, which becomes apparent when one carefully examines the model (Equation 8.7), is that all of the model error (f) is in the dependent variable (y). There is no provision in the model for errors in the independent variable (x). In PAC, this is equivalent to saying that there is error only in the reference method, and no error in the on-line analyzer responses. Although this is obviously not true, practical experience over the years has shown that linear regression can be very effective in analytical chemistry applications. 

There are several figures of merit that can be used to describe the quality of a linear regression model. One very common figure of merit is the correlation coefficient, r, which is defined as  

Situations arise very often where data need to be fitted to linear equations. Linear regression is one of the classical procedures in general regression analysis, and before the advent of accessible non-linear fitting methods it was the only one that could be readily used. For n data pairs in the form (x,y) where y is a function of x, the linear equation of the form y = a + bx that minimises the sum of errors squared (SSD) is given by  

Coefficients Pq, P/ can easily be obtained by using the method of least squares. Nevertheless, the interest is to have the original coefficients of the transcendental regression. To do so, we apply an inverse operator transformation to Po and Pj. Here, we can note that Pq and P/ are the bypassed estimations for their correspondents Po and Pi. 

When the studied case concerns obtaining a relationship for the characterization of a process with multiple independent variables and only one dependent variable, we can use a multiple linear regression  

It is clear that Eq. (5.70) results from the general relation (5.3). In this case, when k= 2, we have a regression surface whereas, when k 2, a hypersurface is obtained. For surface or hypersurface constructions, we have to represent the corresponding values of the process parameters (factors and one dependent variable) for each axis of the phase s space. The theoretical starting statistical material for a multiple regression problem is given in Table 5.11. 

The starting data are frequently transformed into a dimensionless form by a normalization method in order to produce a rapid identification of the coefficients in the statistical model. The dimensionless values of the initial statistical data (y and x ) are computed using Eqs. (5.71) and (5.72), where Sy, s j are the square roots of the correspondent variances  

At this step of the data preparation, we can observe that each column of the transformed statistical data has a zero mean value and a dispersion equal to one. A proof of these properties has already been given in Section 5.2 concerning a case of normal random variable normalization. 

Whenever one property is measured as a function of another, the question arises of which model should be chosen to relate the two. By far the most common model function is the linear one that is, the dependent variable y is defined as a linear combination containing two adjustable coefficients and X, the independent variable, namely. 

A good example is the absorption of a dyestuff at a given wavelength X (lambda) for different concentrations, as expressed by the well-known Lambert-Beer s law  

If the measurements do not support the assumption of a linear relationship, one often tries transformations to linearize it. One does not have to go far for good reasons  

the linear model is undoubtedly the most important one in the treatment of two-dimensional data and will therefore be discussed in detail. 

Overdetermination of the system of equations is at the heart of regression analysis, that is one determines more than the absolute minimum of two coordinate pairs (xj/yi) and xzjyz) necessary to calculate a and b by classical algebra. The unknown coefficients are then estimated by invoking a further model. Just as with the univariate data treated in Chapter 1, the least-squares model is chosen, which yields an unbiased best-fit line subject to the restriction  

Curve fitting is applied when a dependent variable y is measured experimentally as a function of an independent variable x. The simplest relationship between these variables is a straight line. Under some circumstances, v and y do not come directly from experiment but are calculated from the experimental data on the basis of a theory which predicts that y should be linear in v. In the present context, the independent variable v is always that with a negligible level of error, whereas any significant experimental error is only associated with the dependent variable y. Under these circumstances the principle of least squares is applied to estimating the best straight line through the points with respect to the random error in y. 

The minimization of A is performed by optimizing the parameters a and b. Mathematically, this leads to two equations called the normal equations. In the present case, they are 

After simplification by placing the summation operator with each term, one obtains 

The difference is the variance of x multiplied by the number of points N Q y is the covariance of x and y also multiplied by N. Quantities like these turn up frequently in the solution of the normal equations associated with regression analysis. 

Another important quantity associated with the fit of the best straight line to the data is the estimate of the standard deviation in y. From equations (C.2.2), (C.2.6), and (C.2.7) one obtains 

With three measurements, it is possible to calculate the mean My, the standard deviation SDy, and the coefficient of variation cv(%) using the method of external and internal standards. If linear regression leads to a straight line with an intercept not significantly different from zero - which is normal in chromatography - then the cv(%) also holds for the relative deviation in the concentration axes. 

MU is the measurement imcertainty at and t is Student s t-factor, which depends on the number of calibration points. 

Measurement uncertainty MUl can be roughly estimated from the product of 1.5 X t X cv(%). If the governing rules allow a maximal MUj of 25%, 33% or 50%, the highest tolerated cv(%) can be readily obtained through division by 1.5 x t. 

For the internal standard method, the areas (heights) of the compound peaks divided by the areas (heights) of the related internal standard peaks in the same chromatogram should be used, instead of using the areas (heights) directly. It can immediately be seen if the internal standard decreases the cv(%), which is the purpose of the exercise. 

After tentative predefinition, the working range must be verified with several calibration concentrations in order to establish a linear relationship between measurand (y) and concentration (%). 

It frequently occurs in analytical spectrometry that some characteristic, y, of a sample is to be determined as a function of some other quantity, x, and it is necessary to determine the relationship or fimction between x and y, which may be expressed as y = /(x). An example would be the calibration of an atomic absorption spectrometer for a specific element prior to the determination of the concentration of that element in a series of samples. 

A series of /i absorbance measurements is made, y one for each of a suitable range of known concentration, x,-. The n pairs of measurements (x/, y/) can be plotted as a scatter diagram to provide a visual representation of the relationship between x and y. 

The data set comprises pairs of measurements of an independent variable x (concentration) and a dependent variable y (absorbance) and it is required to fit the data using a Unear model with the well known form. 

The total error between the model and observed data is the sum of these individual errors. Each error value is squared to make all values positive and prevent negative and positive errors from cancelling. Thus the total error, e, is given by, 

The total error is the sum of the squared deviations. For some model defined by coefficients a and b, this error will be a minimum and this minimum point can be determined using partial differential calculus. 

---

Figure A3.6.7. Viscosity dependence of reduced -decay rate constants of ers -stilbene in various solvents [90], The rate constants are divided by the slope of a linear regression to the measured rate constants in the respective solvent.

Figure B2.4.2. Eyring plot of log(rate/7) versus (1/7), where Jis absolute temperature, for the cis-trans isomerism of the aldehyde group in fiirfiiral. Rates were obtained from tln-ee different experiments measurements (squares), bandshapes (triangles) and selective inversions (circles). The line is a linear regression to the data. The slope of the line is A H IR, and the intercept at 1/J = 0 is A S IR, where R is the gas constant. A and A are the enthalpy and entropy of activation, according to equation (B2.4.1)...

To gain insight into chemometric methods such as correlation analysis, Multiple Linear Regression Analysis, Principal Component Analysis, Principal Component Regression, and Partial Least Squares regression/Projection to Latent Structures... 

Kohonen network Conceptual clustering Principal Component Analysis (PCA) Decision trees Partial Least Squares (PLS) Multiple Linear Regression (MLR) Counter-propagation networks Back-propagation networks Genetic algorithms (GA)... 

Figure 9-5. Linear regression The sum of the squares of the vertical distances of the points from the line is minimized.

Linear regression models a linear relationship between two variables or vectors, x and y Thus, in two dimensions this relationship can be described by a straight line given by tJic equation y = ax + b, where a is the slope of tJie line and b is the intercept of the line on the y-axis. 

The goal of linear regression is to adapt the values of the slope and of the intercept so that the line gives the best prediction of y from x. This is achieved by minimizing the sum of the squares of the vertical distances of the points from the line. An example of linear regression is given in Figure 9-S. 

While simple linear regression uses only one independent variable for modeling, multiple linear regression uses more variables. 

Multiple linear regression (MLR) models a linear relationship between a dependent variable and one or more independent variables. 

PLS is a linear regression extension of PCA which is used to connect the information in two blocks of variables X and Yto each other. It can be applied even if the features are highly correlated. 

Besides these LFER-based models, approaches have been developed using whole-molecule descriptors and learning algorithms other then multiple linear regression (see Section 10.1.2). 

Step S Building a Multiple Linear Regression Analysis (MLRA) Model... 

Multiple linear regression analysis is a widely used method, in this case assuming that a linear relationship exists between solubility and the 18 input variables. The multilinear regression analy.si.s was performed by the SPSS program [30]. The training set was used to build a model, and the test set was used for the prediction of solubility. The MLRA model provided, for the training set, a correlation coefficient r = 0.92 and a standard deviation of, s = 0,78, and for the test set, r = 0.94 and s = 0.68. 

Alternatives to Multiple Linear Regression Discriminant Analysis, Neural Networks and Classification Methods... 

Multiple linear regression is strictly a parametric supervised learning technique. A parametric technique is one which assumes that the variables conform to some distribution (often the Gaussian distribution) the properties of the distribution are assumed in the underlying statistical method. A non-parametric technique does not rely upon the assumption of any particular distribution. A supervised learning method is one which uses information about the dependent variable to derive the model. An unsupervised learning method does not. Thus cluster analysis, principal components analysis and factor analysis are all examples of unsupervised learning techniques. 

Montgomery D C and A A Peck 1992. Introduction to Linear Regression Analysis. New York, John Wiley Sons. 

Using a hand calculator, find the slope of the linear regression line that passes through the origin and best satisfies the points... 

Using a multiple linear regression computer program, a set of substituent parameters was calculated for a number of the most commonly occurring groups. The calculated substituent effects allow a prediction of the chemical shifts of the exterior and central carbon atoms of the allene with standard deviations of l.Sand 2.3 ppm, respectively Although most compounds were measured as neat liquids, for a number of compounds duplicatel measurements were obtained in various solvents. 

The value of d obtained by linear regression is 0.96 with a correlation coefficient of 0.9985. For 2 alkylpyridines 8 is 2.030 (256), which leads to the conclusion that 2-alkylpyridines are twice as sensitive to steric effects as their thiazole analogs. 

Linear regression, also known as the method of least squares, is covered in Section 5C. 

---