Why We Eliminate Variables

Variables are eliminated for two reasons. First, they are eliminated if they have a small variance, below some threshold value. It is not uncommon to find at least one variable that is constant or is nearly constant, with all but one entry being different from the others, even for moderately sized datasets. This situation can occur when a researcher includes variables (dummy variables) to record the presence or absence of particular properties of an object without checking how well the particular categories are represented. The problem is that when the data are split into training/validation groupings, it can happen that such variables take one value in one group and the other value in the other group. 

The second reason for variable removal is if they are redundant. Redundant variables develop in a dataset for two reasons (1) because more variables (p) exist than objects ( ) and (2) multicolinearity. 

---