Ill-Conditioning of Matrix A and Partial Remedies

If two or more of the unknown parameters are highly correlated, or one of the parameters does not have a measurable effect on the response variables, matrix A may become singular or near-singular. In such a case we have a so called ill-posed problem and matrix A is ill-conditioned. 

A measure of the degree of ill-conditioning of a nonsingular square matrix is through the condition number which is defined as 

The condition number is always greater than one and it represents the maximum amplification of the errors in the right hand side in the solution vector. The condition number is also equal to the square root of the ratio of the largest to the smallest singular value of A. In parameter estimation applications. A is a positive definite symmetric matrix and hence, the cond ) is also equal to the ratio of the largest to the smallest eigenvalue of A, i.e., 

Generally speaking, for condition numbers less than 10 the parameter estimation problem is well-posed. For condition numbers greater than 1010 the problem is relatively ill-conditioned whereas for condition numbers 10 ° or greater the problem is very ill-conditioned and we may encounter computer overflow problems. 

The condition number of a matrix A is intimately connected with the sensitivity of the solution of the linear system of equations A x = b. When solving this equation, the error in the solution can be magnified by an amount as large as cortd A) times the norm of the error in A and b due to the presence of the error in the data. 

---