Hypothesis testing metrics

A data matrix produced by compositional analysis commonly contains 10 or more metric variables (elemental concentrations) determined for an even greater number of observations. The bridge between this multidimensional data matrix and the desired archaeological interpretation is multivariate analysis. The purposes of multivariate analysis are data exploration, hypothesis generation, hypothesis testing, and data reduction. Application of multivariate techniques to data for these purposes entails an assumption that some form of structure exists within the data matrix. The notion of structure is therefore fundamental to compositional investigations. 

Along with graphical assessment one may present metrics, actual numbers, that attempt to quantify the goodness of fit of the model. Two such metrics were presented in the previous section, SSE and MSE, and in this section other metrics will be presented. Formal hypothesis tests may be done on these metrics, such as comparing the metric from one model against another. However, many test statistics based on these metrics tend to be sensitive to the assumption of the underlying distribution, e.g., normally distributed, such that the results from these tests should be treated with skepticism (Cook and Weisberg, 1982). 

If any relations between the difference in metric values for a pair of program versions and the probability that the same pair of program versions fails simultaneously can be identified, one possible step forward would be to use this information to establish prior distributions for use in a Bayesian hypothesis testing approach (Kristiansen and Winther 2004 Kristiansen and Winther 2007). 

The rest of this paper is organised as follows. Section 2 gives a short introduction on how Bayesian hypothesis testing can be used to include dependency aspects in software reliability models. Section 3 presents earlier work related to the use of software metrics to assess failure dependencies between software components, as well as to predict software quahty. Section 4 presents the test case used in the experimental study, and Section 5 gives a short description of the internal and external software metrics used in the study. In Section 6, the analysis and results are summarised. Section 7 concludes with a short summary and directions for further work. 

In this paper, we investigate the possibility of using internal software metrics to assess software component dependencies, thus using metrics to establish priors in a Bayesian hypothesis testing approach. 

Because of the nature of the scientific method. Metrics is an indispensable tool of scientific research. It can provide rigorous indices of the internal consistency and the predictive power of "accepted knowledge" about study systems. Thus, it can aid with theory testing. It can also provide rigorous indices of the strength of correlations between the attributes of the study system and the external factors that might influence it. Thus, it can assist with statistical hypothesis formulation and testing. 

Having said all of this, it is important to remember, however (Popper, 1976 Appendix IX), ... that non-statistical theories have as a rule a form totally different from that of the h here described, that is, they are of the form of a universal proposition. The question thus becomes whether systematics, or phylogeny reconstruction, can be construed in terms of a statistical theory that satisfies the rejection criteria formulated by Popper (see footnote 1) and that, in case of favorable evidence, allows the comparison of degree of corroboration versus Fisher s likelihood function. As far as phylogenetic analysis is concerned, I found no indication in Popper s writing that history is subject to the same logic as the test of random samples of statistical data. As far as a metric for degree of corroboration relative to a nonstatistical hypothesis is concerned. Popper (1973 58-59 see also footnote 1) clarified. 

---