Conditioning and Near Singularity

Ill-conditioned matrices can be due to either insufficient data to fit a model or a poor model. A model-driven situation where ill conditioning is a problem is when the model parameter estimates themselves are highly correlated (correlations greater than 0.95). An example of the latter situation using logarithms of thermometer resistance (Y) as a function of temperature (x) was reported by Simonoff and Tsai (1989) and Meyer and Roth (1972) (Table 3.3). 

Note Starting values for 0i,02, and 03 were — 5, 6000, and 344. Values are reported as estimate (standard error of estimate). Legend SSE, residual sum of squares. 

If the model parameters are highly correlated, one option is to reparameterize the model into a more stable form, which is often of a more simpler form. Simonoff and Tsai (1989), and later Niedzwiecki and Simonoff (1990), call these model transformations guided transformations and the idea is as follows. Since collinearity implies that at least one column of J is close to or nearly a linear combination of the other columns, i.e., using Column 1 as an example, 

Equation (3.85) is called a guiding equation. Once the appropriate guiding equation is found (there may be many for a model with many parameters), solve the guiding equations for one of the model parameters, substitute the result into the original equation, and simplify. The reformulated model is then fit to the data. 

In the Simonoff and Tsai example, the matrix of partial derivatives had a correlation coefficient of 0.99998 between 02 and 03. Hence 02 was highly collinear with 03 and would imply that 

---