Hat-matrix

Hat matrix and leverages in charge density refinements example of atomic net charges determination in a natural zeolite, the scolecite... 

Scolecite gave the opportunity to relate the electron density features of Si-O-Si and Si-O-AI bonds to the atomic environment and to the bonding geometry. After the multipolar density refinement against Ag Ka high resolution X-ray diffraction data, a kappa refinement was carried out to derive the atomic net charges in this compound. Several least-squares fit have been tested. The hat matrix method which is presented in this paper, has been particularly efficient in the estimation of reliable atomic net charges in scolecite. 

At each stage of the refinement of a new set of parameters, the hat matrix diagonal elements were calculated in order to detect the influential observations following the criterium of Velleman and Welsh [8,9]. The inspection of the residues of such reflections revealed those which are aberrant but progressively, these aberrations disappeared when the pseudo-atoms model was used (introduction of multipoler coefficients). This fact confirms that the determination of the phases in acentric structures is improved by sophisticated models like the multipole density model. 

Table 4. Diagonal elements (leverages H,) of the hat matrix and weighted residues ( F / - Fc /c(F )) of the pertinent data in the determination of the atomic net charges in scolecite. 

It can be calculated as usual for SEP, see Eq. (6.98) by use of test samples. It is also possible to estimate the PRESS-value on the basis of standard samples only applying cross validation by means of the so-called hat matrix H (Faber and Kowalski [1997a, b] Frank and Todeschini [1994]) ... 

The n x n hat matrix transforms the vector of the measured y-values to the vector of the estimated-values. An element h,j of the hat matrix is calculated... 

From the elements of the hat matrix some important relations can be derived, e.g. the rank of the X-matrix from the sum of the significant diagonal elements of the hat matrix... 

In the context of OLS, the so-called hat-matrix H plays an important role. H combines the observed and the predicted y-values by the equation... 

There is a further application of the diagonal elements of the hat-matrix in full CV. By using the values hit from the OLS model estimated with all n objects, one can avoid to estimate OLS models for all subsets of size n — 1, because there exists the relation... 

The leverage, / , of the z th calibration sample is the z th diagonal of the hat matrix, H. The leverage is a measure of how far the z th calibration sample lies from the other n - 1 calibration samples in X-space. The matrix H is called the hat matrix because it is a projection matrix that projects the vector y into the space spanned by the X matrix, thus producing y-hat. Notice the similarity between leverage and the Mahalanobis distance described in Chapter 4. 

Note that the most commonly used diagnostics to flag leverage points have traditionally been the diagonal elements / of the hat matrix H = X(X7X) X f These are equivalent to the Mahalanobis distances M/Xx,) because of the monotone relation... 

Hoaglin DC, Welsch RE, The hat matrix in regression and ANOVA, The American Statistician, 1978, 32, 17-22. 

Let h = x(xTx)-1xT be called the HAT matrix. Then least squares minimizes... 

The HAT matrix can be thought to map the observed values (Y) to the predicted values (Y). One important aspect of the least squares model is that a better fit is... 

Another way to look at the HAT matrix is as a distance measure—values with large HAT values are far from the mean of x. It can be shown that the HAT matrix has two useful properties 0 < h < 1 and h = p for i = 1 to n. The average size of h, is then p/n. It is desirable to have all independent variables to have equal influence, i.e., each data point has h p/n. As a rule of thumb, an independent variable has greater leverage than other observations when h is greater than 2p/n. Figure 2.4 presents an example of non-influential and influential x-values. 

More formally, leverage is defined as the partial derivative of the predicted value with respect to the corresponding dependent variable, i.e., h = 0Y /0Y , which reduces to the HAT matrix for linear models. 

Influential observations are ones that significantly affect the values of the parameter estimates, their standard errors, and the predicted values. One statistic used to detect influential observations has already been presented, the HAT matrix. An obvious way to detect these observations is to remove an observation one at a time and examine how the recalculated parameter estimates compare to their original values. This is the row deletion approach to influence diagnostics and on first glance it would appear that this process requires n-iter-ations—a numerically intensive procedure. Statisticians, however, have derived equations that directly reflect the influence of the ith observation without iteration. One useful diagnostic is DFFITS... 

Alternatively, a nonlinear analogy to the HAT matrix in a linear model can be developed using the Jacobian matrix, instead of x. In this case,... 

In a linear model, influence in the x direction is assessed using the HAT matrix... 

The formulation of Eq. (6) using the hat matrix makes the leave-out-one method of cross-validation quite a useful and relatively inexpensive procedure to employ. 

---