Penalized least squares

Li and Lin (2002) used a form of penalized least squares with the smoothly clipped absolute deviation penalty proposed by Fan and Li (2001). This method estimates the parameters, /3, by minimizing not the usual residual sum of squares, but... 

The L B90 algorithm proceeds in two alternating steps, a penalized nonlinear least-squares (PNLS) step and a linear mixed effects (LME) step. 

The method to estimate the coefficients of these equations by least squares regression, is essentially the same for all these models. The idea of minimizing the sum (or weighted sum) of the e comes from Gauss and Legendre. Least squares minimization penalizes large deviations between measured and fitted values heavily. 

Here LSE is the usual least squares error, c is the number of basis functions, p is the number of total basis functions (which can exceed c), M is the number of compounds in the training set, and d is a smoothing function which is typically chosen equal to 1. This scaling of the LSE penalizes models that overfit the data due to using many features and/or basis functions. This is an example of a parsimonious fitting method. The population then undergoes selection, crossover, and mutation. The crossover operator simply exchanges subparts of the chromosomes of the two parents. The mutation operator adds or subtracts a basis funaion. There are additional operations when splines are used as basis functions. 

Here, X (the regularization parameter) is greater than or equal to zero and controls the amount of shrinkage towards zero. When the regularization parameter is zero, this approach is converted back to ordinary least square (OLS) estimation. The penalized formulation (2) has an equivalent formulation in terms of constrained optimization, which may be achieved using convex programming methods ... 

Huang, X. Pan, W. Park, S. Han, X. Miller L. W. HaU, J. (2004). Modeling the Relationship between LVAD Support Time and Gene Expression Changes in Human Heart by Penalized Partial Least Squares. Bioinformatics, Vol. 20, pp. 888-894. 

---