Partial least square regressions

The purpose of Partial Least Squares (PLS) regression is to find a small number A of relevant factors that (i) are predictive for Y and (u) utilize X efficiently. The method effectively achieves a canonical decomposition of X in a set of orthogonal factors which are used for fitting Y. In this respect PLS is comparable with CCA, RRR and PCR, the difference being that the factors are chosen according to yet another criterion. 

We have seen that PCR and RRR form two extremes, with CCA somewhere in between. RRR emphasizes the fit of Y (criterion ii). Thus, in RRR the X-components t, preferably should correlate highly with the original T-variables. Whether X itself can be reconstructed ( back-fitted ) from such components t, is of no concern in RRR. With standard PCR, i.e. top-down PCR, the emphasis is initially more on the X-side (criterion i) than on the T-side. CCA emphasizes the importance of correlation whether the canonical variates t and u account for much variance in each respective data set is immaterial. Ideally, of course, one would like to have the best of all three worlds, i.e. when the major principal components of X (as in PCR) and the major principal components of Y (as in RRR) happen to be very similar to the major canonical variables (as in CCA). Is there a way to combine these three desiderata — summary of X, summary of Y and a strong link between the two — into a single criterion and to use this as a basis for a compromise method The PLS method attempts to do just that. 

PLS has been introduced in the chemometrics literature as an algorithm with the claim that it finds simultaneously important and related components of X and of Y. Hence the alternative explanation of the acronym PLS Projection to Latent Structure. The PLS factors can loosely be seen as modified principal components. The deviation from the PCA factors is needed to improve the correlation at the cost of some decrease in the variance of the factors. The PLS algorithm effectively mixes two PCA computations, one for X and one for Y, using the NIPALS algorithm. It is assumed that X and Y have been column-centred as usual. The basic NIPALS algorithm can best be demonstrated as an easy way to calculate the singular vectors of a matrix, viz. via the simple iterative sequence (see Section 31.4.1)  

Instead of separately calculating the principal components for each data set, the two iterative sequences are interspersed in the PLS-NIPALS algorithm (see Fig. 

Let us take a closer look at this covariance criterion. A covariance involves three terms (see Section 8.3)  

Consider the NIPALS algorithm for PCA discussed earlier (Section 10.4.1). It was stated that the method uses the fact that the PCA loadings are regression coefficients for scores, and vice versa that is, for X, an r x c 

Equations (10.19) and (10.20) represent a system of three variables where each is linearly dependent on the other two. The NIPALS method was soon shown to be applicable to many such systems (e.g. Lyttkens, 1973). 

The method we shall refer to as PLS is one of incorporating the Y data into the latent variable modelling step by bringing an extra projection into the iterative sequence, as follows  

To put it in a visual manner, the location of i gets pulled first towards the X variables then towards the Y variables each time round the iteration. In the case of PCR, the only alteration to t is to pull it towards the X variables, so we end up converging to principal components. The pull of the Y variables tends to bring convergence such that the latent variables chosen are of relevance to both X and Y. Both sets of loadings (w and q) are required for prediction in the case of PLS. 

A simplified PLS model exists for cases when only a single Y variable is to be modelled. PLS 1 is non-iterative and can therefore be used to generate a result quickly in these cases (details are given in Martens and Naes, 1989). 

The calibration model referred to as partial least-squares regression (PLSR) is a technique developed and popularized in analytical science by Wold. The method differs from PCR by including the dependent (concentration) variable in the data compression and decomposition operations, i.e. both y and x data are actively used in the data analysis. This serves to minimize the potential effects of y variables having large variances but which are irrelevant to the calibration model. The simultaneous use of Y and x information makes the method more complex than PCR as two loading vectors are required to provide orthogonality of the factors. 

The first method illustrated here employs the orthogonalized PLSR algorithm developed by Wold and extensively discussed by Martens and Naes.  

As with PCR, the dependent and independent variables are mean centred to give data matrix Fq and vector Xq. Then for each factor, k—l. .. A, to be included in the regression model, the following steps are performed. 

The optimum number of factors to include in the model is found by observation and usual validation statistics. 

A worked example, using the tryptophan data, will illustrate application of the algorithm. The results are presented in Table 6.22. 

Prediction of concentrations of constituents in an unknown mixture generally follows the procedure of PCR, except that it is a more iterative procedure. The spectral loadings from the decomposition step are used to calculate the scores from the unknown absorbance spectrum. The scores are used with the loading vectors for the constituents to calculate the unknown concentrations. Since both A and C are decomposed, the concentrations of the constituents also have loading vectors. As with PCR, an ILS basis is used. 

The number of principal components that should be used for a PLS-1 or PLS-2 analysis is usually determined by first calculating the root-mean-square error of cross validation (RMSECV) using one principal component (PC.) The process is repeated using 2, 3, 4, and so on, PCs. The RMSECV, which is sometimes called 

The validity of the calibration should then be tested using an independent sample set (known as the prediction set) for which the concentrations of each component of interest have been measured by the primary analytical technique. The concentrations of each component are calculated using the calibration matrix determined above and the standard deviation of the difference between the measured and calculated concentration of a given component is called the root-mean-square error of prediction (RMSEP) or simply the standard error of prediction (SEP). The difference between the SEC and SEP is that the analytical data for the prediction set was not used for the calibration. Thus, the SEP is usually, but not always, a little larger than the SEC. If the SEP is much larger than the SEC, the calibration is generally invalid. A much more detailed discussion on the validity of data is beyond the scope of this book but is available in a number of excellent monographs on chemometrics [11-13]. 

It is often necessary to include at least 50 samples in the calibration and prediction sets. Sometimes, measurement of the primary analytical data of so many samples is excessively time consuming. The number of samples can be approximately halved, at the cost of computation time, by using only one calibration set and calculating the root-mean-square error of cross validation (RMSECV), as described in Section 9.9. In general, however, it is preferable to use an independent prediction set to investigate the validity of the calibration but the leave-one-out method significantly reduces the number of samples for which primary analytical data are required. 

---

After an alignment of a set of molecules known to bind to the same receptor a comparative molecular field analysis CoMFA) makes it possible to determine and visuahze molecular interaction regions involved in hgand-receptor binding [51]. Further on, statistical methods such as partial least squares regression PLS) are applied to search for a correlation between CoMFA descriptors and biological activity. The CoMFA descriptors have been one of the most widely used set of descriptors. However, their apex has been reached. 

To gain insight into chemometric methods such as correlation analysis, Multiple Linear Regression Analysis, Principal Component Analysis, Principal Component Regression, and Partial Least Squares regression/Projection to Latent Structures... 

Partial Least Squares Regression/Projection to Laterrt Structures (PLS)... 

Partial Least Squares Regression, also called Projection to Latent Structures, can be applied to estabfish a predictive model, even if the features are highly correlated. 

On the other hand, techniques like Principle Component Analysis (PCA) or Partial Least Squares Regression (PLS) (see Section 9.4.6) are used for transforming the descriptor set into smaller sets with higher information density. The disadvantage of such methods is that the transformed descriptors may not be directly related to single physical effects or structural features, and the derived models are thus less interpretable. 

Partial least-squares in latent variables (PLS) is sometimes called partial least-squares regression, or PLSR. As we are about to see, PLS is a logical, easy to understand, variation of PCR. 

Donahue, S.M., Brown, C.W., Scott, M.J., "Analysis of Deoxyribonucleotides with Principal Component and Partial Least-Squares Regression of UV Spectra after Fourier Processing", Appl. Spec. 1990 (44) 407-413. 

Geladi, P., Kowalski, B.R., "Partial Least-Squares Regression A Tutorial", Anal. Chim. Acta, 1986 (185) 1-17. 

Rosipal R, Trejo LJ. Kernel partial least squares regression in reproducing Kernel Hilbert space. J Machine Learning Res 2001 2 97-123. 

Bennett KP, Embrechts MJ. An optimization perspective on kernel partial least squares regression. In Suykens JAK, Horvath G, Basu S, Micchelli J, Vandewalle J, editors. Advances in learning theory methods, models and applications. Amsterdam lOS Press, 2003. p. 227-50. 

Several techniques from statistics, such as partial least-squares regression, and from artificial intelligence, such as artificial neural networks have been used to learn empirical input/ output relationships. Two of the most significant disadvantages of these approaches are the following ... 

Partial least squares regression (PLS). Partial least squares regression applies to the simultaneous analysis of two sets of variables on the same objects. It allows for the modeling of inter- and intra-block relationships from an X-block and Y-block of variables in terms of a lower-dimensional table of latent variables [4]. The main purpose of regression is to build a predictive model enabling the prediction of wanted characteristics (y) from measured spectra (X). In matrix notation we have the linear model with regression coefficients b ... 

Section 35.4), reduced rank regression (Section 35.5), principal components regression (Section 35.6), partial least squares regression (Section 35.7) and continuum regression methods (Section 35.8). 

S. de Jong, SIMPLS an alternative approach to partial least squares regression. Chemom. Intell. Lab. Syst., 18 (1993) 251-263. 

We will see that CLS and ILS calibration modelling have limited applicability, especially when dealing with complex situations, such as highly correlated predictors (spectra), presence of chemical or physical interferents (uncontrolled and undesired covariates that affect the measurements), less samples than variables, etc. More recently, methods such as principal components regression (PCR, Section 17.8) and partial least squares regression (PLS, Section 35.7) have been... 

A difficulty with Hansch analysis is to decide which parameters and functions of parameters to include in the regression equation. This problem of selection of predictor variables has been discussed in Section 10.3.3. Another problem is due to the high correlations between groups of physicochemical parameters. This is the multicollinearity problem which leads to large variances in the coefficients of the regression equations and, hence, to unreliable predictions (see Section 10.5). It can be remedied by means of multivariate techniques such as principal components regression and partial least squares regression, applications of which are discussed below. 

T. Visser, H.J. Luinge and J.H. van der Maas, Recognition of visual characteristics of infrared spectra by artificial neural networks and partial least squares regression. Anal. Chim. Acta, 296 (1994). 

Geladi, P., and Kowalski, B., Partial least-squares regression a tutorial, AnaL Chim. Acta 185, 1-17 (1986). 

A rapid characterization of the viscosity of waterborne automotive paint was reported by Ito et al. [24], FT-Raman spectroscopy in conjunction with partial least squares regression (PLS) was applied and led to a reasonable correlation. 

Dayal, B. S., MacGregor, J. F., Taylor, P. A., Kildaw, R., and Marcikio, S., Application of Feedforward Neural Networks and Partial Least Squares Regression for Modelling Kappa Number in a Continuous Kamyr Digester, Pulp Paper Can., 95(1) 26 (1994)... 

Calculating the Solution for Regression Techniques Part 3 - Partial Least Squares Regression Made Simple... 

Partial least-squares regression analysis, 16 753, 754, 755-756 Partially alkoxylated chlorotitanates,... 

Multivariate calibration has the aim to develop mathematical models (latent variables) for an optimal prediction of a property y from the variables xi,..., jcm. Most used method in chemometrics is partial least squares regression, PLS (Section 4.7). An important application is for instance the development of quantitative structure—property/activity relationships (QSPR/QSAR). 

Regression can be performed directly with the values of the variables (ordinary least-squares regression, OLS) but in the most powerful methods, such as principal component regression (PCR) and partial least-squares regression (PLS), it is done via a small set of intermediate linear latent variables (the components). This approach has important advantages ... 

Broadhurst, D., Goodacre, R., Jones, A., Rowland, J. J., Kell, D. B. Anal. Chim. Acta 348, 1997, 71-86. Genetic algorithms as a method for variable selection in multiple Unear regression and partial least squares regression, with applications to pyrolysis mass spectrometry. 

---