Validation metrics

Firms have generally been reluctant to publish the costs they attribute to validation, but some case studies have been published related to inspected systems, as can be found in the second part of this book. Based on this information and the author s own experience, the following validation metrics have emerged ... 

The ability to perform validation cost-effectively is dependent on an organization s understanding of requirements and its validation capability. This chapter applies the established Capability Maturity Model (CMM) to computer validation. Examples of validation metrics and measures are examined. The metrics cover prospective validation as well as operation and maintenance of computer systems. Lean Manufacturing and Six Sigma are promoted as tools that organizations can use to streamline and improve the performance of their validation processes. 

Figure 17.2 collates some project validation metrics from recent publications and conferences (data sources listed in Appendix 17B). These metrics have been collected to help practitioners understand validation and the allowance that should be made during project planning. Equally, the metrics will help challenge project planning where resource requirements seem excessive or too low to be credible. Ways of reducing validation costs are explored later in this chapter. 

The cost of validation can appear high if the total cost of ownership for a compnter system is not taken into acconnt. Validation can reportedly more than pay for itself throngh incnrred benefits over the operational life of a system. This book has looked at validation metrics and methods for identifying and implementing validation performance improvements. Opportnnities for redncing the cost of validation include the following ... 

Initially selected metrics may not prove to be valuable or appropriate. A useful strategy to validate metrics is to pilot specific metrics at a few, selected locations... 

Appropriateness means suitability or aptness that is the concept we are attempting to convey here. Model appropriateness is presented in such a way as to be compatible with terminology used in the statistical and other modeling literature, which are foundational for PM modeling. In subsequent sections, we present the steps in establishing model appropriateness, approaches to model evaluation and validation, metrics used in model validation, and an application example to illustrate the principles involved in estabhshing model appropriateness. 

Removing personal blame from incidents that set the stage for personal injury will enable more proactive reporting, evaluahng, and correcting. When your periodic environmental audits show less and less property damage, you can be assured you are preventing injuries. In fact, I am convinced this is actually a more reliable and valid metric for safety improvement than the standard injury and illness rates derived from employees self reports and visits to the plant infirmary. So a comprehensive safety measurement system... 

Pearlman RS, Smith KM. Metric validation and the receptor-relevant subspace concept. / Chem Inf Comput Sci 1999 39 28-35. 

An important aspect of all methods to be discussed concerns the choice of the model complexity, i.e., choosing the right number of factors. This is especially relevant if the relations are developed for predictive purposes. Building validated predictive models for quantitative relations based on multiple predictors is known as multivariate calibration. The latter subject is of such importance in chemo-metrics that it will be treated separately in the next chapter (Chapter 36). The techniques considered in this chapter comprise Procrustes analysis (Section 35.2), canonical correlation analysis (Section 35.3), multivariate linear regression... 

The pH-metric technique used to determine partition coefficients was first used in the 1950s in solvent extraction of metal complexes [280-282], but it is in pharmaceutical research that it is most widely used thanks to the recent development of a fully automated and computer-controlled apparatus [125,283]. The potentiometric approach has been validated in various solvent systems [284-287], and it has become a relevant and expanding experimental technique to obtain lipophilicity descriptors [257,287-289]. 

The pH-metric procedure has been validated against the standard shake-flask method [150,357], and many studies using it have been reported [56,149-151,153,161,162,224,225,229,246,250,268,269,275,276,280,281,324-363], Determinations of values of log P as low as —2 and as high as +8 have been documented [161,162,352]. The published literature clearly indicates that the Dyrssen technique is a reliable, versatile, dynamic, and accurate method for measuring logP. It may lack the speed of HPLC methods, and it cannot go as low in log P as the CV... 

The definition of these metrics appears somewhat arbitrary and is hard to understand in the framework of statistical reasoning. In particular, the meaning of maximum and minimum terms in the definition of the Chinchilli and the rho metrics cannot be easily verified. The fact that an arbitrarily defined index performs better for an arbitrarily selected set of experimental data cannot be accepted as a general proof of validation. 

An improved HPLC-photohydrolysis-colorimetry method was validated for twenty-eight reference nitrosamines. These were separated by HPLC and photolytically cleaved by UV radiation. The resulting nitric oxide was oxidized and hydrolyzed to nitrite ions, which were derivatized into an azo dye with Griess reagent and measured spectrophoto-metrically. The method was applied to separate and detect hitherto unknown nonvolatile nitrosamines in biological fluids and food extracts591. 

K.H. Esbensen and L.P. luhus. Representative sampling, data quality, validation - a necessary trinity in chemo-metrics, in Comprehensive Chemometrics, S. Brown, R. Taulor, and B. Walzak (eds), vol. 4, 1-20. Elsevier, Oxford, 2009. 

Kerrick and Darken, 1975 Helgeson et al., 1978) and is strictly valid only for isovolu-metric mixtures. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) Prediction error is a useful metric for selecting the optimum number of factors to include in the model. This is because the models are most often used to predict the concentrations in future unknown samples. There are two approaches for generating a validation set for estimating the prediction error internal validation (i.e., cross-validation with the calibration data), or external validation (i.e., perform prediction on a separate validation set). Samples are usually at a premium, and so we most often use a cross- validation approach. 

A similar study by Verrelli and coworkers, based on PCA of potentio-metric data, allowed the differentiation of Verdicchio wines according to the winemaker and the vintage. Furthermore, MLR and PLS models, validated by means of cross-validation, permitted the quantification of a number of oenological parameters, such as SO2, L-malic acid, and the total phenols content (Paolesse et al., 2008 Verrelli et ah, 2007). 

Once processes have been defined for the enterprise, sufficient system resolution should be determined. This is accomplished by evaluating the ability of the process owner to manage and execute the process requirements. Another factor in this determination is the data and metrics needed from the process by management and leadership. An example of a complex process that benefits the organization by being managed through distinct subprocesses is validation. 

---