Foundations of Least Squares

The Maximum Likelihood Criterion. Let it be assumed that for m different values of an independent variable x(x , i= 1. m) there are corresponding measured values of a dependent variable 7(7, i=. ., ni). Let us further assume that there is a theoretical model predicting the way in which jis expected to depend on vand that this model may be represented by an analytical function f  

The quantities UjiJ =, n)t are independently adjustable parameters that are initially 

There are virtually no additional complexities or difficulties of any consequence resulting from having more than one independent variable, but for the sake of simplicity of expression we will assume only one in the treatment presented below. 

Least-squares fitting deals with the problem of determining the best values of the n adjustable parameters a, a so as to maximize the agreement between the m observed 7 values and the values of 7 calculated with Eq. (1). In effect we are seeking the besf (though generally not exact) solutions to a set of m simultaneous equations (called obser vational equations or equations of condition) in n unknowns  

The number m of observations exceeds (usually by an order of magnitude or more) the number n of adjustable parameters. Thus the mathematical problem is overdetermined. (For a situation in which m= n, the corresponding set of simultaneous equations could in principle be solved exactly for the parameters aj. But this exact fit to the data would provide no test at all of the validity of the model.) In least squares as properly applied, the number of observations is made large compared to the number of parameters in order (1) to sample adequately a domain of respectable size for testing the validity of the model, (2) to increase the accuracy and precision of the parameter determinations, and (3) to obtain statistical information as to the quality of the parameter determination and the applicability of the model. 

---