Variable subset selection Variance

As a result of the principal component calculation, the U matrix has a number of columns equal to tlie minimum of the number of samples or variables. Knowing tliat only some of the columns in U contain the relevant information, a subset is selected. Choosing the relevant number of PCs to include in the model is one of the most important steps in the PCR process because it is the key to the stabilization of the inverse. Ordinarily the columns in U are chosen sequentially, from highest to lowest percent variance described. 

As in the previous section, we are interested in linear combinations of variables, with the goal of determining that combination which best summarizes the n-dimensional distribution of data. We are seeking the Unear combination with the largest variance, with normalized coefficients applied to the variables used in the linear combinations. This axis is the so-called rst principal axis or first principal component. Once this is determined, then the search proceeds to find a second normalized linear combination that has most of the remaining variance and is uncorrelated with the first principal component. The procedure is continued, usually imtil all the principial components have been calculated. In this case, p = n and a selected subset of the principal components is then used for further analysis and for interpretation. 

Jolliffe techniques ofvariable reduction [Jolliffe, 1972 Jolliffe, 1973] exploitthe association of the original variables with the eigenvectors (PCs), usually obtained from the correlation matrix. The criterion of these techniques is to keep as much variance of the data in the subset of selected variables. The Jolliffe technique B2 associates one variable with each of the last p - M eigenvectors and deletes those p - M variables vdth the largest coefficients in the p - M... 

D-optimal design Given a limited number of experiments, or samples to collect, the algorithm for D-optimal experimental design are used to select a subset of exper-iments/samples representing the overall variability of the candidate samples/experi-ments as accurately as possible. Usually the algorithms used are based on maximisation of the determinant of the variance-covariance matrix of the subset, hence the term D-optimal. 

Variable selection in regression arises when the set of variables to include into the model is not predetermined. The problem to be addressed is from the list of potential candidates to include in the model which ones should be included and in what form. The objectives here are to include as many predictors that can influence the prediction while in addition including as few as possible because the variance of the prediction increases as the number of predictors increase. Hence the goal of variable selection is to find an appropriate subset regression model. 

PCA is by far the most important method in multivariate data analysis and has two main applications (a) visualization of multivariate data by scatter plots as described above (b) data reduction and transformation, especially if features are highly correlating or noise has to be removed. For this purpose instead of the original p variables X a subset of uncorrelated principal component scores U can be used. The number of principal components considered is often determined by applying a threshold for the score variance. For instance, only principal components with a variance greater than 1% of the total variance may be selected, while the others are considered as noise. The number of principal components with a non-negligible variance is a measure for the intrinsic dimensionality of the data. As an example consider a data set with three features. If all object points are situated exactly on a plane, then the intrinsic dimensionality is two. The third principal component in this example has a variance of zero. Therefore two variables (the scores of PCI and PC2) are sufficient for a complete description of the data structure. 

---