Least squares, partial

Partial least-squares in latent variables (PLS) is sometimes called partial least-squares regression, or PLSR. As we are about to see, PLS is a logical, easy to understand, variation of PCR. 

In addition to the set of new coordinate axes (basis space) for the spectral data (the x-block), we also find a set of new coordinate axes (basis space) for the concentration data (the y-block). 

In addition to expressing the spectral data as projections onto the spectral factors (basis vectors), we express the concentration data as projections onto the concentration factors (basis vectors). 

On a rank-by-rank (i.e. factor-by-factor) basis, we rotate, or perturb, each pair of factors, (1 spectral factor and its corresponding concentration factor) towards each other to maximize the fit of the linear regression between the projections of the spectra onto the spectral factor with the projections of the concentrations onto the concentration factor. 

Partial Least Squares (PLS) attempts to find so-called latent variables that capture the variance in the data and at the same time achieves maximum correlation between predicted variables Y and predictor variables X. 

Originally, PLS was a technique that would produce a static linear model, although also non-linear and dynamic versions have been published in the literature. As principal components in principal component analysis, the use of latent variables in PLS can reduce the dimensionality of the problem considerably. 

The method of partial least squares (PLS) is also a regression technique which makes use of quantities like PCs derived from the set of independent variables. The PCs in PLS regression models are called latent variables (LV), as shown in the PLS equation, eqn (7.9). 

As in PCA, PLS will generate as many latent variables (q) as the smaller of N (dimensions) or P (samples). Thus far, PLS appears to generate identical models to PCR so what is the difference (other than terminology) The answer is that the PLS procedure calculates the latent variables and the regression coefficients in eqn (7.9) all at the same time. The algorithm is actually an iterative procedure (Wold 1978) but the effect is to combine the 

PCA step of PCR with the regression step. Latent variables, like PCs, are calculated to explain most of the variance in the x set while remaining orthogonal to one another. Thus, the first latent variable (LVi) will explain most of the variance in the independent set, LV2 the next largest amount of variance and so on. The important difference between PLS and PCR is that the latent variables are constructed so as to maximize their correlation with the dependent variable. Unlike PCR equations where the PCs do not enter in any particular order (see eqns 7.6 to 7.8) the latent variables will enter PLS equations in the order one, two, three, etc. The properties of latent variables are  

Note that this sum of squares looks similar to the residual sum of squares (RSS) given by eqn (6.12) but is different in eqn (6.12) the j/i is predicted from an equation that includes that data point here the y, is not in the model hence the term predictive residual sum of squares. The difference in predictive ability of two PLS models can be evaluated by comparison of their PRESS values. 

The E statistic compares a PLS model of i components with the model containing one component less and, in order to evaluate PRESS for the one component model, PRESS for the model containing no components is calculated by comparing predicted values with the mean. A critical value of 0.4 has been suggested for E (Wold 1978) and when this is exceeded, the PLS equation with i components is doing no better (or worse) in prediction than the model with i—1 latent variables. 

Another solution of (10.5) called bidiagonalization is used in Partial Least Squares (PLS). Here the original input matrix X is linked to two orthogonal matrices O and W by a bidiagonal matrix L. 

In contrast with PCR, the PLS factors (vectors w) are not independent of each other. 

The selection of PLS factors is therefore simplified because they are a priori ranked according to their value. 

The validity of the parameterized techniques discussed thus far relies on the linear and stable relationship between concentration and response of individual sensors. With this caveat in mind, these data reduction and evaluation tools can also be used for optimization of arrays in terms of numbers and orthogonality. 

Another well-established method for building predictive models is partial least squares (PLS) regression. PLS is a modem relative of MLR, having been established in the 1960 s by Wold. The method reaches beyond linear regression by replacing the descriptors with a matrix of latent variables distilled from both the structural features of the training compounds and their experimental results. In PLS, the use of the term latent variables differs from its formal definition in other regression methods.  

The use of these latent variables provides PLS with a greater capacity than MLR to distill most of the information contained in a large number of descriptors and properties into a smaller number of factors and thereby avoid some of the ravages of the Curse of Dimensionality. It is especially suited to dealing with systems containing many highly correlated descriptors. PLS is related to the methods of principal components analysis (PCA) and maximum redundancy analysis (MRA), in that all three methods augment the raw desaiptors with matrices derived from variance found in the descriptors themselves (PCA), the properties to be modeled (MRA), or a combination of both (PLS).  

The performance of PLS is comparable to that of methods such as ridge regression and neural networks, particularly when it is used to model properties that have a principally linear relationship to combinations of the descriptors. An excellent example of the use of PLS in the interpretation of the spectra of chemical mixtures is provided by Tobias.  

In Section 33.2.2 we showed how LDA classification can be described as a regression problem with class variables. As a regression model, LDA is subject to the problems described in Chapter 10. For instance, the number of variables should not exceed the number of objects. One solution is to apply feature selection or 

PLS is often presented as the major regression technique for multivariate data. In fact its use is not always justified by the data, and the originators of the method were well aware of this, but, that being said, in some applications PLS has been spectacularly successful. In some areas such as QSAR, or even biometrics and psychometrics, 

An important feature of PLS is that it takes into account errors in both the concentration estimates and the spectra. A method such as PCR assumes that the concentration estimates are error free. Much traditional statistics rests on this assumption, that all errors are of the variables (spectra). If in medicine it is decided to determine the concentration of a compound in the urine of patients as a function of age, it is assumed that age can be estimated exactly, the statistical variation being in the concentration of a compound and the nature of the urine sample. Yet in chemistry there are often significant errors in sample preparation, for example accuracy of weighings and dilutions, and so the independent variable in itself also contains errors. Classical and inverse calibration force the user to choose which variable contains the error, whereas PLS assumes that it is equally distributed in both the x and c blocks. 

The most widespread approach is often called PLS1. Although there are several algorithms, the main ones due to Wold and Martens, the overall principles are fairly straightforward. Instead of modelling exclusively the x variables, two sets of models are obtained as follows  

Additionally, tire analogy to ga or tire eigenvalue of a PC involves multiplying tire sum of squares of both ta and pa together, so we define tire magnitude of a PLS component as 

This will have tire property that tire sum of values of ga for all nonzero components add up to tire sum of squares of tire original (preprocessed) data. Note that in contrast to PCA, tire size of successive values of ga does not necessarily decrease as each component is calculated. This is because PLS does not only model the x data, and is a compromise between x and c block regression. 

---

Some methods that paitly cope with the above mentioned problem have been proposed in the literature. The subject has been treated in areas like Cheraometrics, Econometrics etc, giving rise for example to the methods Partial Least Squares, PLS, Ridge Regression, RR, and Principal Component Regression, PCR [2]. In this work we have chosen to illustrate the multivariable approach using PCR as our regression tool, mainly because it has a relatively easy interpretation. The basic idea of PCR is described below. 

Another problem is to determine the optimal number of descriptors for the objects (patterns), such as for the structure of the molecule. A widespread observation is that one has to keep the number of descriptors as low as 20 % of the number of the objects in the dataset. However, this is correct only in case of ordinary Multilinear Regression Analysis. Some more advanced methods, such as Projection of Latent Structures (or. Partial Least Squares, PLS), use so-called latent variables to achieve both modeling and predictions. 

After an alignment of a set of molecules known to bind to the same receptor a comparative molecular field analysis CoMFA) makes it possible to determine and visuahze molecular interaction regions involved in hgand-receptor binding [51]. Further on, statistical methods such as partial least squares regression PLS) are applied to search for a correlation between CoMFA descriptors and biological activity. The CoMFA descriptors have been one of the most widely used set of descriptors. However, their apex has been reached. 

To gain insight into chemometric methods such as correlation analysis, Multiple Linear Regression Analysis, Principal Component Analysis, Principal Component Regression, and Partial Least Squares regression/Projection to Latent Structures... 

Kohonen network Conceptual clustering Principal Component Analysis (PCA) Decision trees Partial Least Squares (PLS) Multiple Linear Regression (MLR) Counter-propagation networks Back-propagation networks Genetic algorithms (GA)... 

One widely used algorithm for performing a PCA is the NIPALS (Nonlineai Iterative Partial Least Squares) algorithm, which is described in Ref [5],... 

Partial Least Squares Regression/Projection to Laterrt Structures (PLS)... 

Partial Least Squares Regression, also called Projection to Latent Structures, can be applied to estabfish a predictive model, even if the features are highly correlated. 

On the other hand, techniques like Principle Component Analysis (PCA) or Partial Least Squares Regression (PLS) (see Section 9.4.6) are used for transforming the descriptor set into smaller sets with higher information density. The disadvantage of such methods is that the transformed descriptors may not be directly related to single physical effects or structural features, and the derived models are thus less interpretable. 

The previously mentioned data set with a total of 115 compounds has already been studied by other statistical methods such as Principal Component Analysis (PCA), Linear Discriminant Analysis, and the Partial Least Squares (PLS) method [39]. Thus, the choice and selection of descriptors has already been accomplished. 

Partial Least Squares and Molecular Field Analysis... 

The ability of partial least squares to cope with data sets containing very many x values is considered by its proponents to make it particularly suited to modern-day problems, where it is very easy to compute an extremely large number of descriptors for each compound (as in CoMFA). This contrasts with the traditional situation in QSAR, where it could be time-consuming to measure the required properties or where the analysis was restricted to traditional substituent constants. 

E Johansson and M Cocchi 1993. PLS - Partial Least-squares Projections to Latent Structures. In binyi H (Editor) 3D QSAR in Drug Design. Leiden, ESCOM, pp. 523-550. 

The field points must then be fitted to predict the activity. There are generally far more field points than known compound activities to be fitted. The least-squares algorithms used in QSAR studies do not function for such an underdetermined system. A partial least squares (PLS) algorithm is used for this type of fitting. This method starts with matrices of field data and activity data. These matrices are then used to derive two new matrices containing a description of the system and the residual noise in the data. Earlier studies used a similar technique, called principal component analysis (PCA). PLS is generally considered to be superior. 

PLS (partial least-squares) algorithm used for 3D QSAR calculations PM3 (parameterization method three) a semiempirical method PMF (potential of mean force) a solvation method for molecular dynamics calculations... 

Partial least-squares path modeling with latent variables (PLS), a newer, general method of handling regression problems, is finding wide apphcation in chemometrics. This method allows the relations between many blocks of data ie, data matrices, to be characterized (32—36). Linear and multiple regression techniques can be considered special cases of the PLS method. 

Other chemometrics methods to improve caUbration have been advanced. The method of partial least squares has been usehil in multicomponent cahbration (48—51). In this approach the concentrations are related to latent variables in the block of observed instmment responses. Thus PLS regression can solve the colinearity problem and provide all of the advantages discussed earlier. Principal components analysis coupled with multiple regression, often called Principal Component Regression (PCR), is another cahbration approach that has been compared and contrasted to PLS (52—54). Cahbration problems can also be approached using the Kalman filter as discussed (43). 

Mancozeb is a dithiocarbamate pesticide with a very low solubility in organic and inorganic solvent. In this work we have developed a solvent free, accurate and fast photoacoustic FTIR-based methodology for Mancozeb determination in commercial fungicides. The proposed procedure was based on the direct measurement of the solid samples in the middle infrared region using a photoacoustic detector. A multivariate calibration approach based on the use of partial least squares (PLS) was employed to determine the pesticide content in commercially available formulations. 

DETERMINATION OF NUTRITIONAL PARAMETERS OF YOGHURT SAMPLES THROUGH PARTIAL-LEAST-SQUARES ATTENUATED TOTAL REFLECTANCE FOURIER TRANSFORM INFRARED SPECTROMETRY... 

The aim of this work is the determination of several nutritional parameters, such as Energetic Value, Protein, Fat, and Carbohydrates content, in commercially available yoghurt samples by using Attenuated Total Reflectance Fourier Transform Infrared (ATR-FT-IR) spectrometry and a partial least square approach. 

The aim of this researeh eonsists on the development of an analytieal methodology to quantify Ca(OH), eontent in neutralized anliydrite samples, using FT-IR speetroseopy and partial least squares quantitative analysis teehnique. 

Most of the 2D QSAR methods are based on graph theoretic indices, which have been extensively studied by Randic [29] and Kier and Hall [30,31]. Although these structural indices represent different aspects of molecular structures, their physicochemical meaning is unclear. Successful applications of these topological indices combined with multiple linear regression (MLR) analysis are summarized in Ref. 31. On the other hand, parameters derived from various experiments through chemometric methods have also been used in the study of peptide QSAR, where partial least square (PLS) [32] analysis has been employed [33]. 

S Wold, A Ruhe, H Wold, WJ Dunn III. The collmearity problem m linear regression. The partial least squares (PLS) approach to generalized inverses. SIAM I Sci Stat Comput 5 735-743, 1984. 

---