Decision errors

Hall, R. E. et al., Post-Event Human Decision Errors Operator Action Tree/Time Reliability Correlation, BNL, November 1982. 

If an analytical test results in a lower value x, < x0, then the customer may reject the product as to be defective. Due to the variation in the results of analyses and their evaluation by means of statistical tests, however, a product of good quality may be rejected or a defective product may be approved according to the facts shown in Table 4.2 (see Sect. 4.3.1). Therefore, manufacturer and customer have to agree upon statistical limits (critical values) which minimize false-negative decisions (errors of the first kind which characterize the manufacturer risk) and false-positive decisions (errors of the second kind which represent the customer risk) as well as test expenditure. In principle, analytical precision and statistical security can be increased almost to an unlimited extent but this would be reflected by high costs for both manufacturers and customers. 

As would be expected, in order to be able to have at least 95% confidence that the true CV p does not exceed its target level, we must suffer the penalty of sometimes falsely accepting a "bad" method (i.e. one whose true CV p is unsatisfactory). Such decision errors, referred to as "type-1 errors", occur randomly but have a controlled long-term frequency of less than 5% of the cases. (The 5% probability of type-1 error is by definition the complement of the confidence level.) The upper confidence limit on CV p is below the target level when the method is judged acceptable... 

The sixth step of the DQO process addresses the fact that the true mean contaminant concentration p will never be known and that it can be only approximated by the sample mean concentration x calculated from the collected data. The mean sample concentration approximates the true mean concentration with a certain error. And therefore, a decision based on this mean concentration will also contain an inherent error. The latter is called decision error. 

Step 6 allows us to create a statistical approach for the evaluation of the collected data. Using a statistical test and the statistical parameters selected in Step 6, we will be able to control decision error and make decisions with a certain level of confidence. Decision error, like total error, can only be minimized, but never entirely eliminated. However, we can control this error by setting a tolerable level of risk of an incorrect decision. Conducting Step 6 enables the planning team to specify acceptable probabilities of making an error (the tolerable limits on decision errors). At this step of the DQO process, the project team will address the following issues ... 

Assign probability values to points above and below the action level that reflect tolerable probability for decision errors... 

The planning team will devise measures for control or minimization of decision error through controlling total error. To do this, the planning team may need the expertise of a statistician who will develop a data collection design with a tolerable probability for the occurrence of a decision error. 

If the sample mean concentration is close in value to the action level, the decision, however, becomes uncertain, and two types of decision errors may take place ... 

Based on the sample data, we may reject the null hypothesis when in fact it is true, and consequently accept the alternative hypothesis. By failing to recognize a true state and rejecting it in favor of a false state, we will make a decision error called a false rejection decision error. It is also called a false positive error, or in statistical terms, Type I decision error. The measure of the size of this error or the probability is named alpha (a). The probability of making a correct decision (accepting the null hypothesis when it is true) is then equal to 1—a. For environmental projects, a is usually selected in the range of 0.05-0.20. 

Based on the sample data, we may accept the null hypotheses when in fact it is false, and make a false acceptance decision error or a false negative error. In... 

Example 2.2 Baseline and alternative hypothesis and the two types of decision errors... 

Two types of decision errors may take place in decision making ... 

Suppose that the true mean concentration p is llOmg/kg, but the sample mean concentration is 90mg/kg. In this case, the null hypothesis is true (H0 llOmg/kg > lOOmg/kg). However, basing our decision on the sample data, we reject it in favor of the alternative hypothesis (Ha 90mg/kg< lOOmg/kg) and make a false rejection decision error. 

Now suppose that the true mean concentration p is 90 mg/kg, but the sample mean concentration is 110 mg/kg. Then the null hypothesis is false because the true mean concentration is not greater than the action level (90mg/kg < lOOmg/kg). But we are not able to recognize this, as the sample mean concentration supports the null hypothesis (H0 110mg/kg> lOOmg/kg). Based on the sample data we accept the null hypothesis and thus make a false acceptance decision error. 

Baseline condition is true (H0 P > Ca) Correct decision The probability of making a correct decision (1—a) False acceptance decision error False negative decision error Type II decision error Probability ji Risk, error rate 100 x ft... 

Table 2.1 summarizes possible conclusions of the decision-making process and common statistical terms used for describing decision errors in hypothesis testing. 

Example 2.2 describes lead-contaminated soil with the baseline condition stating that the true mean concentration of lead in soil exceeds the action level. If a false rejection decision error has been made, contaminated soil with concentrations of lead exceeding the action level will be used as backfill, and will therefore continue to pose a risk to human health and the environment. On the opposite, as a consequence of a false acceptance decision error, soil with lead concentrations below the action level will not be used as backfill and will require unnecessary disposal at an additional cost. 

In this example, the two decision errors have quite different consequences a false rejection decision error has consequences that will directly affect the environment, whereas a false acceptance decision error will cause unnecessary spending. Recognizing a decision error with more severe consequences is a pivotal point in the DQO process and a dominating factor in optimizing the data collection design. 

The two possible decision errors are as follows Deciding that the true mean TEQ concentration in soil exceeds the action level when it does not Deciding that the true mean TEQ concentration in soil is below the action level when it actually is not The two possible decision errors are as follows Deciding that the VOC concentrations in treated water exceed the discharge limitations when they do not Deciding that the VOC concentrations in treated water do not exceed the discharge limitations when they actually do... 

Clearly, collecting more data to obtain a more accurate estimate of the true mean concentration will reduce decision error risk but it also would increase the project costs. There is always a fine balance between the need to make a valid decision and the need to stay within an available or reasonable budget, and sometimes the two needs may be in conflict. As a compromise, after evaluating the severity of decision error... 

What are the potential consequences for each decision error ... 

The consequence of a false rejection decision error will be a more severe one because of the unmitigated threat to the environment. The consequence of a false acceptance decision error will be a more severe one because of the violation of the NPDES permit. 

The probability curve is the same as in Figure 2.3, but the gray region in this case is situated on the opposite side of the action level, and the false acceptance and false rejection decision errors have changed places on the probability curve. With low probabilities of decision error, soil with sample mean concentrations significantly below 100 mg/kg will be accepted as backfill, soil with sample mean concentrations clearly exceeding 100 mg/kg will be rejected. However, if the true mean concentration is slightly above the action level, for example, 110 mg/kg, and the sample mean concentration is less than 100 mg/kg (e.g. 90 mg/kg), then the project team will be likely to make a false acceptance decision error. A consequence of false acceptance... 

---