Variables stepwise selection

To benchmark our learning methodology with alternative conventional approaches, we used the same 500 (x, y) data records and followed the usual regression analysis steps (including stepwise variable selection, examination of residuals, and variable transformations) to find an approximate empirical model, / (x), with a coefficient of determination = 0.79. This model is given by... 

A stepwise variable selection method adds or drops one variable at a time. Basically, there are three possible procedures (Miller 2002) ... 

An often-used version of stepwise variable selection (stepwise regression) works as follows Select the variable with highest absolute correlation coefficient with the y-variable the number of selected variables is mo= 1. Add each of the remaining x-variables separately to the selected variable the number of variables in each subset is nii = 2. Calculate F as given in Equation 4.44,... 

Stepwise Perform a stepwise variable selection in both directions start once from the empty model, and once from the full model the AIC is used for measuring the performance. 

Lin (1993) suggested using stepwise variable selection and Wu (1993) suggested forward selection or all (estimable) subsets selection. Lin (1993) gave an illustrative analysis by stepwise selection of the data in Table 6. He found that this identified factors 15,12,19,4, and 10 as the active factors, when their main effects are entered into the model in this order. Wang (1995) analyzed the other half of the Williams experiment and identified only one of the five factors that Lin had identified as being nonnegligible, namely, factor 4. 

In order to avoid some drawbacks of the stepwise approaches, the i-fold stepwise variable selection method was recently proposed [Lucic et ai, 1999b]. This technique is based on descriptor orthogonalization and, at each subsequent step, adds the set of the best i descriptors. 

Chemometric analysis of the HPLC data was also used to predict cheese ripening time by multiple linear regression analysis with stepwise variable selection two variables, ttsi-casein and asi-I peptide, and a constant were used. The estimation error was 2.5 days. 

The following three performance measures are commonly used for variable selection by stepwise regression or by best-subset regression. An example in Section 4.5.8 describes use and comparison of these measures. 

Criteria for the different strategies were mentioned in Section 4.2.4. For example, if the AIC measure is used for stepwise model selection, one would add or drop that variable which allows the biggest reduction of the AIC. The process is stopped if the AIC cannot be further reduced. This strategy has been applied in the example shown in Section 4.9.1.6. 

An exhaustive search for an optimal variable subset is impossible for this data set because the number of variables is too high. Even an algorithm like leaps-and-bound cannot be applied (Section 4.5.4). Instead, variable selection can be based on a stepwise procedure (Section 4.5.3). Since it is impossible to start with the full model, we start with the empty model (regress the y-variable on a constant), with the scope... 

With these argnments as a backdrop, I will review some empirical variable selection methods in addition to the prior knowledge-based, stepwise and all possible-combinations methods discnssed earlier in the MLR section (Section 12.3.2). 

The above paragraph describes the forward option of the interval methods, where one starts with no variables selected, and sequentially adds intervals of variables until the stop criterion is reached. Alternatively, one could operate the interval methods in reverse mode, where one starts using all available x variables, and sequentially removes intervals of variables until the stop criterion is reached. Being stepwise selection methods, the interval methods have the potential to select local rather than global optima, and they require careful selection of the interval size (number of variables per interval) based on prior knowledge of the spectroscopy, to balance computation time and performance improvement. However, these methods are rather straightforward, relatively simple to implement, and efficient. 

A general alternative to stepwise-type searching methods for variable selection would be methods that attempt to explore as much of the possible solution space as possible. An exhaustive search of all possible combinations of variables is possible only for problems that involve relatively few x variables. However, it... 

As discussed in the introduction, the solution of the inverse model equation for the regression vector involves the inversion of R R (see Equation 5 23). In many anal al chemistry experiments, a large number of variables are measured and R R cannot be inverted (i.e., it is singular). One approach to solving this problem is called stepwise MLR where a subset of variables is selected such that R R is not singular. There must be at least as many variables selected as there are chemical components in the system and these variables must represent different sources of variation. Additional variables are required if there are other soairces of variation (chemical or physical) that need to be modeled. It may also be the case that a sufficiently small number of variables are measured so that MIR can be used without variable selection. 

The error in differential vaporization calculations caused by variable Kj can be minimized by making the calculations illustrated by Examples 12-8 and 12-9 in stepwise manner. Select relatively small changes in pressure, obtain values of Kj at the average pressure, then calculate the resulting nLf and xjf by trial and error. These values of nLf and xjf then are used as njj and Xj for the next calculation over another small change in pressure. 

Some of the earliest applications of chemometrics in PAC involved the use of an empirical variable selection technique commonly known as stepwise multiple linear regression (SMLR).8,26,27 As the name suggests, this is a technique in which the relevant variables are selected sequentially. This method works as follows ... 

One particular challenge in the effective use of MLR is the selection of appropriate X-variables to use in the model. The stepwise and APC methods are some of the most common empirical methods for variable selection. Prior knowledge of process chemistry and dynamics, as well as the process analytical measurement technology itself, can be used to enable a priori selection of variables or to provide some degree of added confidence in variables that are selected empirically. If a priori selection is done, one must be careful to select variables that are not highly correlated with one other, or else the matrix inversion that is done to calculate the MLR regression coefficients (Equation 8.24) can become unstable, and introduce noise into the model. 

Kelly et al. [13], when describing their work on the geographical origin of rice, also addressed the issue of over-fitting of data, and proposed a stepwise approach, selecting the minimum number of variables (from the 52 measured) in order to maximize the separation whilst ensuring that statistical over-fitting was minimized. 

According to the results of the study carried out by Derksen and Keselman [13], and concerning several automatic variable selection methods, in a typical case 20 to 74 percent of the selected variables are noise variables. The number of the noise variables selected varies with the number of the candidate predictors and with the degree of collinearity among the true predictors (due to the well-known problem of variance inflation when variables are correlated any model containing correlated variables are unstable). Sereening out noise variables while retaining true predietors seems to be a possible solution to the chance correlation problem in stepwise MLR. 

When compounds are strongly grouped, CV may not work well. Recent examples have shown that CV is misleading when it is applied after variable selection in stepwise MLR. Thus although cross-validation is considered as the state-of-the-art statistical validation technique, its results are only relevant when correctly applied. 

---