The Normal Equations

Unfortunately, it is not possible for us to see the grey plane imbedded in an m-dimensional space, 3 dimensions have to suffice. 

Some people have a good 3 dimensional imagination. They immediately see that the closest point on the plane is just vertically underneath the tip of the vector y. Using more appropriate expressions the minimal residual vector r, which is the shortest difference between y and Fa, is orthogonal, or normal, to the plane defined by F. 

Now the expression Normal Equations starts to make sense. The residual vector r is normal to the grey plane and thus normal to both vectors f ,i and f , 2 As outlined earlier, in Chapter Orthogonal and Orthonormal Matrices (p.25), for orthogonal (normal) vectors the scalar product is zero. Thus, the scalar product between each column of F and vector r is zero. The system of equations corresponding to this statement is  

This set of equations can be further simplified and written as one matrix equation  

This last equation is very crucial and we will spend considerable time investigating it further. 

---

Solving the normal equations by Cramer s rule leads to the solution set in determinantal fomi... 

The form of the symmetric matrix of coefficients in Eq. 3-20 for the normal equations of the quadratic is very regular, suggesting a simple expansion to higher-degree equations. The coefficient matrix for a cubic fitting equation is a 4 x 4... 

We have already seen the normal equations in matrix form. In the multivariate case, there are as many slope parameters as there are independent variables and there is one intercept. The simplest multivariate problem is that in which there are only two independent variables and the intercept is zero... 

The left side of the normal equations can be seen to be a product including X, its transpose, and m. Matrix multiplication shows that... 

Obtain the normal equations [Eq. set (3-63)] from the minimization conditions... 

Any linearly independent set of simultaneous homogeneous equations we can construct has only the zero vector as its solution set. This is not acceptable, for it means that the wave function vanishes, which is contrai y to hypothesis (the electron has to be somewhere). We are driven to the conclusion that the normal equations (6-38) must be linearly dependent. 

The pressure drop for the exhaust opening, the jet supply opening, and for the hood opening if there are side walls, can be calculated using the normal equations for flow inside ducts and into ducts and openings. 

Because it is of particular interest in the present context, we now obtain the normal equations for linear regression with a single independent vanable. The model function is... 

Because there is only one independent variable, the subscript has been omitted. We now note that Zx/n = x and Zy/n = y, so we find Eqs. (2-75) as the normal equations for unweighted univariate least-squares regression. 

Carrying through the treatment as before yields Eqs. (2-78) as the normal equations for weighted linear univariate least-squares regression. 

It can be argued that the main advantage of least-squares analysis is not that it provides the best fit to the data, but rather that it provides estimates of the uncertainties of the parameters. Here we sketch the basis of the method by which variances of the parameters are obtained. This is an abbreviated treatment following Bennett and Franklin.We use the normal equations (2-73) as an example. Equation (2-73a) is solved for <2o-... 

Referring to the earlier treatment of linear least-squares regression, we saw that the key step in obtaining the normal equations was to take the partial derivatives of the objective function with respect to each parameter, setting these equal to zero. The general form of this operation is... 

Table 6-1 lists the experimental quantities, k, T, ct, the transformed variables x, y, and the weights w. (It is necessary, in least-squares calculations, to carry many more digits than are justified by the significant figures in the data at the conclusion, rounding may be carried out as appropriate.) The sums required for the solution of the normal equations are... 

The partial derivative of R with respect to each parameter is then minimized. The normal equations are... 

In order to improve the convergence characteristics and robustness of the Gauss-Newton method, Levenberg in 1944 and later Marquardt (1963) proposed to modify the normal equations by adding a small positive number, y2, to the diagonal elements of A. Namely, at each iteration the increment in the parameter vector is obtained by solving the following equation... 

As a result of this scaling, the normal equations AAk(i+,) = b become... 

With this transformation, the normal equations now become,... 

We strongly suggest the use of the reduced sensitivity whenever we are dealing with differential equation models. Even if the system of differential equations is non-stiff at the optimum (when k=k ), when the parameters are far from their optimal values, the equations may become stiff temporarily for a few iterations of the Gauss-Newton method. Furthermore, since this transformation also results in better conditioning of the normal equations, we propose its use at all times. This transformation has been implemented in the program for ODE systems provided with this book. 

Therefore, instead of modifying the normal equations, we propose a direct approach whereby the conditioning of matrix A can be significantly improved by using an appropriate section of the data so that most of the available sensitivity information is captured for the current parameter values. To be able to determine the proper section of the data where sensitivity infonnation is available, Kalo-gerakis and Lulls (1983b) introduced the Information Index for each parameter, defined as... 

Subsequent use of the stationary condition (3Sp/3k<-, 1))=0, yields the normal equations... 

The study of how fluids interact with porous solids is itself an important area of research [6], The introduction of wall forces and the competition between fluid-fluid and fluid-wall forces, leads to interesting surface-driven phase changes, and the departure of the physical behavior of a fluid from the normal equation of state is often profound [6-9]. Studies of gas-liquid phase equilibria in restricted geometries provide information on finite-size effects and surface forces, as well as the thermodynamic behavior of constrained fluids (i.e., shifts in phase coexistence curves). Furthermore, improved understanding of changes in phase transitions and associated critical points in confined systems allow for material science studies of pore structure variables, such as pore size, surface area/chemistry and connectivity [6, 23-25],... 

The normal equations will be written for the least squares regression of the nonlinear equation for N sets of data, (xj.y, ... 

---