Performance metrics

A review of some of the topics we have discussed in this section as they relate to BP networks is given in Ref. 238. A discussion of training and testing for noisy data with feedforward networks is given in Refs. 239 and 240. 

A neural network application is of little use unless quantitative statements concerning its performance can be made. There are a large number of performance metrics (PMs), some of which are application specific, and we cannot be 

If you use percent correct as a PM, you should report the percent correct for each category and specify how many cases each category has. 

We can use the definitions above to introduce PMs that are often more enlightening in describing ANN performance. 

Sensitivity = TP/(TP + FN). This is also known as recall and as the true positive ratio. This is the number of cases correctly classified as belonging to A divided by the total number of cases that actually belong to A. Alternatively, it is the probability that a case will be correctly classified as belonging to A. It gives an indication of the relative number of false negatives and is an important PM when it is crucial that a case be correctly classified as belonging to A. For example, if a patient has cancer it is important that the patient be classified as having cancer. 

---

While the plan itself is not auditable by third parties, it may be auditable by second parties i.e. customers. The third party or registrar is entitled to examine the plan to ascertain that it is what it proclaims to be. The particulars are of no concern except those aspects relating to quality, such as the resources, quality objectives, customer satisfaction plans, and performance metrics. Whatever is stated on these aspects, the auditors will expect to see evidence that the business plan is not merely a wish list and that provisions have been made to enable implementation through the quality system. 

This is not to imply that a full-throttle acceleration from rest to that speed is a maneuver frequently executed by the typical driver. The time to 60 mph is rather an easily measured parameter that seiwes as a surrogate for other performance metrics. A car that is slow from 0 to 60 mph will likely have slow response from 40 to 60 mph for freeway merging, or prove lethargic when climbing hills. Reflecting the market preference of the typical new-car buyer, for the average new U.S. passenger car, the acceleration time from 0 to 60 mph has decreased from about 14 seconds in 1975 to fewer than 11 seconds iti 1995. 

Centre for Medicines Research, 2004 Global Clinical Performances Metrics Programme Report, Industry Report, July 2004. 

Given a set of existing (x, y) data records, where x is a vector of operating or decision variables, which are believed to influence the values taken on by y, y is a performance metric, usually assumed to be a quality characteristic of the product or process under analysis ... 

Prototypical application examples. To provide a more concrete notion of the type of systems where our approaches are expected to be particularly helpful and useful, we conclude this section with a sample of prototypical examples of what the performance metric, y, in the problem statement (2) may represent, together with a definition of the corresponding systems ... 

A solution space, a, consisting of hyperrectangles defined in the decision space, X, is a basic characteristic common to all the learning methodologies that will be described in subsequent sections. The same does not happen with the specific performance criteria tfi, mapping models /, and search procedures 5, which obviously depend on the particular nature of the systems under analysis, and the type of the corresponding performance metric, y. 

The nature of the performance metric, y, is determined by the characteristics of the specific process under analysis. Since we are particularly interested in analyzing situations where y is related to product or process quality, it is quite common to find systems where a categorical variable y is chosen to classify and evaluate their performance. This may happen due to the intrinsic nature of y (e.g., it can only be measured and assume qualitative values, such as good, high, and low ), or because y is derived from a quantization of the values of a surrogate continuous measure of performance (e.g., y = good if some characteristic z of the product has value within the range of its specifications, and y= bad, otherwise). 

In this section we will introduce the problem statements adopted for this type of performance metric, briefly describe the learning methodology employed to address it [for a more complete presentation, see Saraiva and Stephanopoulos (1992a)], and show a specific application case study. 

In Section IV we considered a categorical performance metric y. Although that represents a common practice, especially when y defines the quality of a product or process operation, there are many instances where system performance is measured by a continuous variable. Even when y is quality-related, it is becoming increasingly clear that explicit continuous quality cost models should be adopted and replace evaluations of performance based on categorical variables. 

This Section addresses cases with a continuous performance metric, y. We identify the corresponding problem statements and results, which are compared with conventional formulations and solutions. Then Taguchi loss functions are introduced as quality cost models that allow one to express a quality-related y on a continuous basis. Next we present the learning methodology used to solve the alternative problem statements and uncover a set of final solutions. The section ends with an application case study. 

To illustrate how different m(X ) and x may happen to be, let s consider as a specific example (others can be found in Saraiva and Stephanopoulos, 1992c) a Kraft pulp digester. The performance metric y, that one wishes to minimize, is determined by the kappa index of the pulp produced and the cooking yield. Two decision variables are considered H-factor (xj), and alkali charge (X2). Furthermore, we will assume as perfect an available deterministic empirical model (Saraiva and Stephanopoulos, 1992c), /, which expresses y as function of x, i.e., that y =/(xi, X2) is perfectly known. 

If, besides the quality-related measure, z, one also wishes to include operating costs, in the analysis, because quality loss functions express quality costs on a monetary basis, commensurate with operating costs, the final global performance metric, y, which reflects total manufacturing cost, is simply the sum of both quality and operating costs (Clausing, 1993),... 

In the previous paragraphs we defined the solution format f, performance criterion i/r, mapping procedure /, and performance metric y that characterize our learning methodology for systems with a quantitative metric y. Here we will assemble all these pieces together and briefly discuss the search procedure, S (further details can be found in Saraiva... 

Both situations with categorical and continuous, real-valued performance metrics will be considered and analyzed. Since Taguchi loss functions provide quality cost models that allow the different objectives to be expressed on a commensurate basis, for continuous performance variables only minor modifications in the problem definition of the approach presented in Section V are needed. On the other hand, if categorical variables are chosen to characterize the system s multiple performance metrics, important modifications and additional components have to be incorporated into the basic learning methodology described in Section IV. 

Since these loss functions express quality costs on a common and commensurate basis, extending the learning methodology of Section V to a situation with P objectives is straightforward. All one has to do is replace the original definition of the y performance metric [Eq. (23)] by the following more general version ... 

Similarly, for the case where a continuous performance metric, y, is employed, the following loss functions were defined (Saraiva and Stephanopoulos, 1992c) ... 

The solution found when the plasma etching was analyzed in terms of continuous performance metrics is also presented in Fig. 8, and is given by... 

Eissen, M. Metzger, J.O. (2002) Environmental Performance Metrics for Daily Use in Synthetic Chemistry, Chemistry A European Journal, 8, 3580-3585. 

MS equipment is evaluated on several performance metrics. Mass accuracy, mass resolution, and mass range are standard parameters frequently assessed to determine the suitability of an instrument. Mass accuracy is defined as the extent to which a mass analyzer reflects true m/z values and is measured in atomic mass units (amu), parts per million (ppm), or percent accuracy. 

Tab. 13.2 Performance metrics of display technologies, more biobs the better.

When The Home Depot was still a relatively new concept and the competition was less intense, growth came easily pretty much all we had to do was open new stores and customers would come. And, as long as we were growing fast and launching new stores without cannibalizing existing ones, we did not need to pay attention to most items below the sales line. So there was not much discussion of - or need for -performance metrics beyond sales. 

---