Safety audits criteria

A Mexico-domiciled motor carrier must meet the safety audit criteria listed in Part 385, Appendix A. 

Audit instruments can be evaluated on the basis of three different types of validity when linking the audit to safety performance. These are content-related, criterion-related, and construct-related procedures for accumulating evidence of validity (Anastasi 1988, 139). Content-related validity involves the examination of the audit content to determine whether it covers a representative sample of the behaviors to be measured. Criterion-related validity procedures indicate the effectiveness of an audit in predicting performance in specified activities. The performance on the audit is checked against a criterion that is considered a direct and independent measure of what the audit was intended to measure. The third type of validity, construct validity, is the extent to which the audit is said to be measuring a theoretical construct or trait. Construct validation requires accumulation of information from a variety of sources. 

There are very few studies on criterion-based validity of audits. Bailey and Petersen (1989, Polk 1987) describe an attempt to relate safety-program characteristics with accident statistics and monetary losses in a very large study of railroads. They surveyed 18 railroads and scored the them on the following areas of safety programs ... 

There is some evidence that supports the criterion-based validity of other audits. Uusitalo and Mattila (1989) found a fair correlation between audit scores and accident rates in 11 industrial companies. There are other studies described by Uusitalo that show some evidence of validity. Pringle and Brown (1990) report a 12 percent drop in the incident rates among the 2,395 North American companies that used the ISRS in the period 1978-1979. The apparent lack of a control group of similar companies that did not use the ISRS and the fact that the improvonent was relatively small leads us to some reservation about the conclusion that the ISRS was responsible for this change. It would not be surprising to find that a sample of similar companies that used methods for safety improvement that did not involve audits showed the same 12 percent reduction in accident rates. 

We found nothing in the literature regarding what underlying constructs might be measured by audits. Many authors assert that audits are a measure of safety performance. Audits may correlate with safety performance, but they do not represent performance in the sense of a criterion. A measure of safety performance should be related to loss. 

Validity is another question. We have said that audits typically have content-based validity. Testing for criterion-based validity may be beyond the scope of many situations for which safety professionals would like to apply audits. Certainly in a company with a small population and relatively low accident rates, there would probably not be enough data to find a relationship between audit scores and accident rates. An alternative would be to look for a relationship between audit scores and the scores on our safety survey (see Chapter 8). Since the scores of this survey are extensively validated as measures of system performance, they can serve as a criterion against which to test audits. 

---