Partitioning the sums of squares

We have previously shown that through the operation called partitioning the sums of squares , the following equality holds [1] ... 

The calculation used is the calculation of the sum of squares of the differences [5], This calculation is normally applied to situations where random variations are affecting the data, and, indeed, is the basis for many of the statistical tests that are applied to random data. However, the formalism of partitioning the sums of squares, which we have previously discussed [6] (also in [7], p. 81 in the first edition or p. 83 in the second edition), can be applied to data where the variations are due to systematic effects rather than random effects. The difference is that the usual statistical tests (t, x2> F, etc.) do not apply to variations from systematic causes because they do not follow the required statistical distributions. Therefore it is legitimate to perform the calculation, as long as we are careful how we interpret the results. 

There are two (count them two) more very critical developments that come from this partitioning of sums of squares. First, the correlation coefficient is not just an arbitrarily chosen computation (or even concept), but as we have seen bears a close and fundamental relationship to the whole ANOVA concept, which is itself a very fundamental statistical operation that data is subject to. As we have seen here, all these quantities - standard deviation, correlation coefficient, and the whole process of decomposing a set of data into its component parts - are very closely related to each other, because they all represent various outcomes obtained from the fundamental process of partitioning the sums of squares. 

If (and only if) replicate experiments have been carried out on a system, it is possible to partition the sum of squares of residuals, SS, into two components (see Figure 6.10) one component is the already familiar sum of squares due to purely experimental uncertainty, 55. the other component is associated with variation attributed to the lack of fit of the model to the data and is called the sum of squares due to lack of fit, SS. ... 

Recall that the lack-of-fit test partitions the sum of squares error (SSe) into two components pure error, the actual random error component and lack of fit, a nonrandom component that detects discrepancies in the model. The lack-of-fit computation is a measure of the degree to which the model does not fit or represent the actual data. 

Therefore we come to the examination of ANOVA of data depending on more than one variable. The basic operation of any ANOVA is the partitioning of the sums of squares. 

In Section 6.4, it was shown for replicate experiments at one factor level that the sum of squares of residuals, SS can be partitioned into a sum of squares due to purely experimental uncertainty, SS, and a sum of squares due to lack of fit, SSi f. Each sum of squares divided by its associated degrees of freedom gives an estimated variance. Two of these variances, and were used to calculate a Fisher F-ratio from which the significance of the lack of fit could be estimated. 

The sum of squares corrected for the mean, SS, is equal to the sum of squares due to the factors, plus the sum of squares of residuals, SS,. This result can be obtained from the partitioning... 

Of the four different methods of cluster analysis applied, the method of Ward described in the Clustan User Manual (10), worked best when compared to the single-, complete-, or average-linkage methods. Using Ward s method, two clusters, Gn and Gm, are fused when by pooling the variance within two existing clusters the variance of the so formed clusters increases minimally. The variance or the sum of squares within the classes will be chosen as the index h of a partition. 

Analysis of Variance (ANOVA). Keeping in mind that the total variance is the sum of squares of deviations from the grand mean, this mathematical operation allows one to partition variance. ANOVA is therefore a statistical procedure that helps one to learn whether sample means of various factors vary significantly from one another and whether they interact significantly with each other. One-way analysis of variance is used to test the null hypothesis that multiple population means are aU equal. 

All of the statistical figures of merit used for judging the quality of least-squares fits are based upon the fundamental relationship shown in Equation 5.15, which describes how the total sum of squares is partitioned into two parts (1) the sum of squares explained by the regression and (2) the residual sum of squares, where y is the mean concentration value for the calibration samples. 

The Sum of Squares for Water Content can be partitioned exactly similarly and we get the results in Table 12.3. 

These results show clearly the importance of the optimization criterion to clustering. The computationally simple Ward s method performs better than the simulated annealing approach with a simplistic criterion. However, a criterion that more correctly accounts for the hierarchy, by minimizing the sum of squared error at each level, performs much better. As with partitional clustering the application of simulated annealing to hierarchical clustering requires careful selection of the internal clustering criterion. 

The total variation in the data can be partitioned between the variation amongst the sub-samples and the variation within the sub-samples. The computation proceeds by determining the sum of squares for each source of variation and then the variances. 

Point your Web browser to http //chemistry.brookscole.com/skoogfac/. From the Chapter Resources menu, choose Web Works, and locate the Chapter 7 section. Click on the link to the statistics on-line textbook. Click on the ANOVA/MANOVA button. Read about the partitioning of the sum of squares in ANOVA procedures. Click on the F-distribution link in this section. Look at the tail areas for an F-dis-tribution with both degrees of freedom equal to 10. Determine the value of F for a significance level of 0.10 with both degrees of freedom equal to 10. 

The sum of squares 2 dy" over all objects and all components is the residual sum of squares not accounted for by the model. This can be partitioned into components, SCj/, which show how much the "unexplained variance" for each descriptor, "fc", contributes to the total sum of squares, see Fig. 15.16. It is seen in Fig. 15.16 that... 

The total variance, expressed as the sum of squares of deviations from the grand mean, is partitioned into the variances within the different groups and between the groups. This means that the sum of squares corrected for the mean, is obtained from... 

Fork random partition of m. If Jt = m, the method is called leave-one-out cross-validation (LOOCV). A predicting function / is calculated for each i m, using the observations from m i = 0,..., - 1, i + 1,..., m - 1 exclusively for learning. The formula for the sum of squared residuals is simplified to... 

It can be noted that the sum of squares due to regression can be further partitioned if an orthogonal basis is used to define the regression parameters. This will be explored in greater detail in Chap. 4, including how to define an orthogonal basis for regression. 

Partitioning of the sums of squares. After Draper and Smith [1966]. 

The orthogonal contrasts have the property that they completely partition the total variation in the results. The sum of squared factor effects is equal to the total sum of squares (see, for example, Cochran and Cox, 1957). 

The t test on parameters, described in Sec. 7.3.2, is useful in establishing whether a model contains an insignificant parameter. This information can be used to make small adjustments to models and thus discriminate between models that vary from each other by one or two parameters. This test, however, does not give a criterion for testing the adequacy of this model. The residual sum of squares, calculated by Eq. (7.160), contains two components. One is due to the scatter in the experimental data and the other is due to the lack of fit of the model. In order to test the adequacy of the fit of a model, the sum of squares must be partitioned into its components. This procedure is called analysis of variance, which is summarized in Table 7.2. To maintain generality, we examine a set of nonlinear data and assume the availability of multiple values of the dependent variable y j at each point of the independent variable jc, (see Fig. 7.12). 

A multivariate ANOVA, however, has some properties different than the univariate ANOVA. In order to be multivariate, obviously there must be more than one variable involved. As we like to do, then, we consider the simplest possible case and the simplest case beyond univariate is obviously to have two variables. The ANOVA for the simplest multivariate case, that is, the partitioning of sums of squares of two variables (X and Y), proceeds as follows. From the definition of variance ... 

The first two terms on the RHS of equation 70-20 are the variances of X and Y. The third term, the numerator of which is known as the cross-product term, is called the covariance between X and Y. We also note (almost parenthetically) here that multiplying both sides of equation 70-20 by (re - 1) gives the corresponding sums of squares, hence equation 70-20 essentially demonstrates the partitioning of sums of squares for the multivariate case. 

There are several critical facts that come out of the partitioning of sums of squares and its consequences, as shown in equations 70-20 and 70-22. One is the fact that in the multivariate case, variances add only as long as the variables are uncorrelated, that is, the correlation coefficient (or the covariance) is zero. 

It is a characteristic of linear models and least squares parameter estimation that certain sums of squares are additive. One useful relationship is based on the partitioning... 

---