Multi-way regression

In the following, several approaches to constructing a regression model will be tested on some simple example data sets. In these examples, traditional second-order calibration is not possible, as there is no causal or direct relationship between specific measured variables and the responses. The purpose of this example is mainly to illustrate the relative merits of different calibration approaches on different types of data, whereas the model building (residual and influence analysis) itself is not treated in detail. 

Multi-way Analysis With Applications in the Chemical Sciences 

The emission spectra of 268 sugar samples dissolved in phosphate buffered water were measured at seven excitation wavelengths (excitation 230, 240, 255, 290, 305, 325 and 340 nm, emission 365-558 nm, 4.5 nm intervals). Hence the data can be arranged in a 268 x 44 x 7 three-way array or a 268 x 308 matricized two-way matrix. The color was determined by a standard wet-chemical procedure as described by Nprgaard [1995], 

Models (PARAFAC, Tucker3, A-PLS) of the fluorescence data are built. For PARAFAC and Tucker3, the scores of the model of the fluorescence data are used as independent variables which are related to the color by multiple linear regression [Bro 1997, Geladi et al. 1998], In A-PLS, the regression model is determined simultaneous with finding the scores. 

The quality of the models was determined by test set validation with the first 134 samples (corresponding to the first 1.5 months of the sampling period) in the calibration and the last 134 in the test set. This systematic approach was adopted in order to test the predictions on future samples. It is known that the chemistry of the process may change over time, so this test provides a more realistic measure of the quality of the models than if a random sampling had been chosen. 

---

Some problems in chemistry can be cast as three-way regression problems. Consider a batch process in which a chemical product is made batchwise. For each batch run, J process variables are measured at K points in time. At the end of each batch run, the quality of the product (y) is measured. If I batch runs are available, then a three-way array X (/ x J x K) has the first mode in common with the quality measurements collected in y (lx 1). It might be worthwhile to find a model predicting y from X. This is a multi-way regression problem. 

Cross-validation of multi-way regression models does not essentially differ from crossvalidating two-way regression models. Suppose that a three-way X (7 x J x K) is available and a univariate y (7 x 1). Then cross-validation can be performed in the following way ... 

In step (ii) any multi-way regression model may be used and tested. Usually, different model types (e.g. Af-PLS and Tucker3-based regression on scores model), or models with a different number of components (e.g. a two-component Af-PLS model and a three-component W-PLS model) are tested. To have complete independence of and y, the matrices involved in building the model have to be preprocessed based on interim calibration data each time step (ii) is entered. 

In multi-way regression models there are the residuals Ey and Ex(see Chapters 4). These residuals can be used as described in Chapter 7 for calculating a number of different sums of squares. These can also be visualized as described here for component models. 

Process chemometrics is a field where multi-way methods were introduced more recently. A batch process gives rise to an array of batches followed over time and measured by multiple process sensors such as temperature and pressure sensors or perhaps spectroscopy, as in Figure 10.3. Multi-way component models are used for the construction of control charts and multi-way regression is used for constructing predictive calibration models. 

N-PLS is the multi-way extension of the two-way PLS method. In the same way that it is possible to calculate a regression model between a two-dimensional X matrix and a reference value (Y) by means of PLS it... 

In order to concisely describe multi-way models, the usual matrix product is not sufficient. Three other types of matrix products are introduced the Kronecker ( ), Hadamard ( ) and Khatri-Rao (O) product [McDonald 1980, Rao Mitra 1971, Schott 1997], The Kronecker product allows a very efficient way to write Tucker models (see Chapter 4). Likewise, the Khatri-Rao product provides means for an efficient way to write a PARAFAC model (see Chapter 4). The Hadamard product can, for instance, be used to formalize weighted regression (see Chapter 6). 

Predicting product quality with multi-way covariates regression... 

Cross-validation of X shows that a (3,2,3) Tucker3 model is a reasonable model for X [Louwerse et al. 1999], Hence, a multi-way covariates regression model relating y to X, and assuming a (3,2,3) Tucker3 structure for X was calculated. The optimal a was found by cross-validation to be 0.9. This indicates that stabilizing the predictions by... 

Multi-way covariates regression can be extended to multi-way arrays X and Y of arbitrary numbers of modes. For each array a specific multi-way structure can be assumed. For example, if both X and Y are three-way arrays and a Tucker3 structure is assumed for both arrays, then the multi-way covariates regression model is... 

The similarity between a Tucker 1 model of a three-way array X and a PCA model of a two-way array X has already been mentioned in Section 4.1. This similarity is the basis for the most popular three-way regression model, the Tucker 1-PLS model. In the literature this regression method has also been called three-way partial least squares regression [Wold et al. 1987]. This name should be avoided for the Tucker 1 version specifically, because it gives rise to much confusion as to what is multi-way methodology. Even though the Tuckerl version of three-way partial least squares regression is indeed one way to implement a three-way method, it is actually the model that assumes the least three-way structure in the data. 

The most significant difference between Tuckerl-PLS and /V-PLS on the one hand and multi-way covariates regression models on the other hand is that Tuckerl-PLS and /V-PLS models are calculated sequential and the multi-way covariates regression model is derived in a simultaneous fashion. This has several consequences ... 

The solution of a Tuckerl-PLS, /V-PLS and a multi-way covariates regression model for a given data set will be different. 

The solutions of Tuckerl-PLS models with an increasing number of components are nested this also holds for /V-PLS models but not for multi-way covariates regression models. 

The algorithms of Tucker- 1-PLS, /V-PLS and multi-way covariates regression are different and this will result in differences in speed and properties of the estimated parameters. 

Important multi-way component and regression models have been described in this chapter. PARAFAC and Tucker3 are the best-known methods which can both be viewed as extensions of ordinary two-way PCA. PARAFAC is an extension in the sense that it provides the bestfitting rank R component model of a three-way data set, and Tucker3 is an extension of PCA... 

The algorithm for multi-way covariates regression is explained using the example of regressing y (/ x 1) on X (/ x J x K), where a Tucker3 structure for X is assumed, with P, Q and R components in the I, J and K direction, respectively. Generalizations to multivariate (and multi-way) Y and other three-way structures for X can be inferred from this example. 

When data from new samples are to be evaluated with an existing multi-way covariates regression model, the principle follows almost immediately from the procedures used in other decomposition models. The scores of size P x 1 of a new sample can be found by from the preprocessed data, Xnew (J x K) and the weights, W (JK x R)... 

In multi-way analysis, a distinction is made between cross-validation of regression models and cross-validation of component models. In regression models, the purpose is to predict a y and cross-validation can then be used to find the best regression model to achieve this. In component models cross-validation can be used to find the most parsimonious but adequate... 

Simplified equations result because A A equals the identity matrix I, B B equals I, and C C equals I. The same equations are valid for PARAFAC models, but the middle cross-product is not identity. To summarize the properties of the squared Mahalanobis distances, the following can be said. Leverage can be defined for multiple linear regression as an influence measure. It is related to a specific Mahalanobis distance. The term leverage is sometimes also used for similar Mahalanobis distances in low-rank regression methods such as PCR and PLS. Then it becomes dependent on the rank of the model. The squared Mahalanobis distances can also be defined for PCA and multi-way models and can be calculated for both variables and objects. 

In the same way as for the % x 2 design, we can select the data for these 12 experiments from the experimental results of table 3.8 (2 effervescent tablet factor study) and estimate the coefficients by multi-linear regression. (The linear combinations method is not applicable here.) The results, given in table 3.36, are almost identical to those found with the full design and reported in table 3.9. 

---