Numerical methods linear regression

As linear regression is a very fundamental operation, several methods have been developed in order to improve the numerical stability of the calculation. It is beyond the objective of this book to discuss these issues in any detail. We do feel, however, that the reader has to be aware of the potential problems and should be able to avoid them as much as possible. 

Current methods for supervised pattern recognition are numerous. Typical linear methods are linear discriminant analysis (LDA) based on distance calculation, soft independent modeling of class analogy (SIMCA), which emphasizes similarities within a class, and PLS discriminant analysis (PLS-DA), which performs regression between spectra and class memberships. More advanced methods are based on nonlinear techniques, such as neural networks. Parametric versus nonparametric computations is a further distinction. In parametric techniques such as LDA, statistical parameters of normal sample distribution are used in the decision rules. Such restrictions do not influence nonparametric methods such as SIMCA, which perform more efficiently on NIR data collections. 

In a strict sense parameter estimation is the procedure of computing the estimates by localizing the extremum point of an objective function. A further advantage of the least squares method is that this step is well supported by efficient numerical techniques. Its use is particularly simple if the response function (3.1) is linear in the parameters, since then the estimates are found by linear regression without the inherent iteration in nonlinear optimization problems. 

The two models in Sections VI. 1 and VI.2 have been solved by a numerical method based on a finite difference routine BAND (j).718,20 To solve a non-linear model, iteration with trail values is required. Furthermore, double iterations are needed in cases, for example, when it is required to optimize the thickness of the PBER, or to regress the key parameters from experimental data. These complex situations make the convergences of the solution difficult. 

Figure 6.5 Method for determining / and its derivatives. Slopes were calculated using linear regression over 5 points in a data set of 500 points. Double precision was required in the computer program in order to avoid noise in the second derivative. The fraction transformed versus temperature trace was numerically generated assuming a second order reaction.

Sadler, D.R. Numerical Methods for Non-linear Regression Queensland Press St. Lucia, Australia, 1975. 

A combination of the Runge Kutta method and methods of non-linear regression allows a parameter identification from the time-course data. This technique starts with a given set of parameters, performs the numeric integration of the rate equation... 

The numerical parameter estimation was based on kinetic experiments out of which the temperatures and concentrations were recorded for the compounds. The kinetic and adsorption parameters were estimated by using the Simplex-Levenberg-Marquardt method, which minimizes the residual sum of squares between the estimated and the experimental concentrations with non-linear regression. 

The kinetic parameters in the rate equations were determined with non-linear regression analysis. The rate equations were inserted into the mass balances, which were solved numerically with the backward difference method during the parameter estimation. In the parameter estimation, the following objective function was minimized ... 

Numerically, the LSA approach may be implemented by calcnlations that involve estimation by (general) linear regression, e.g., nse of cnbic spline functions." The intrinsic problems of nonlinear estimation common in more structured methods can thereby be avoided or significantly reduced. 

Eq. 6.2.6 was solved analytically to obtain the operation curve of the reactor (X vs t). Lumped kinetic parameters were determined by non-linear regression of experimental data using the numerical method of Newton-Raphson with first-order Taylor series expansion. Lumped parameters were smooth functions of temperature all parameters were adequately fitted to second order polynomials except for D that required a fourth order polynomial. The model can be used for reactor temperature optimization and can be extended to prolonged sequential batch operation provided that a sound model for enzyme inactivation is validated (Illanes et al. 2005b). 

---