Model validation/testing process

Model Predictions vs. Field Observations The Model Validation/Testing Process... 

Once the model parameters have been determined, it is necessary to validate the model. As before, three different components need to be considered (1) testing the residuals, (2) testing the adequacy of the model, and (3) taking corrective action. The general details of these components are the same as for regression analysis (see Sect. 3.3.5 Model Validation). However, some specific details are needed for model validation in process system identification. 

The goal of this paper Is to present the current status of model validation and field testing of chemical fate and transport models other papers in this symposium discuss the state-of-the-art of modeling specific processes, environments, and multimedia problems. The process of model validation, and its various components, is described considerations in field testing, where model results are compared to field observations, are discussedp an assessment of the current extent of field testing for various processes and media is presented and future field testing and data needs are enumerated. 

Figure 1 presents an overview of the model testing/valida-tion process as developed at the Pellston workshop. A distinction is drawn between validation of empirical versus theoretical models as discussed by Lassiter (4 ). In reality, many models are combinations of empiricism and theory, with empirical formulations providing process descriptions or interactions lacking a sound, well-developed theoretical basis. The importance of field data is shown in Figure 1 for each step in the model validation process considerations in comparing field data with model predictions will be discussed in a later section. 

The process of field validation and testing of models was presented at the Pellston conference as a systematic analysis of errors (6. In any model calibration, verification or validation effort, the model user is continually faced with the need to analyze and explain differences (i.e., errors, in this discussion) between observed data and model predictions. This requires assessments of the accuracy and validity of observed model input data, parameter values, system representation, and observed output data. Figure 2 schematically compares the model and the natural system with regard to inputs, outputs, and sources of error. Clearly there are possible errors associated with each of the categories noted above, i.e., input, parameters, system representation, output. Differences in each of these categories can have dramatic impacts on the conclusions of the model validation process. 

Model validation, v - the process of testing a calibration model to determine bias between the estimates from the model and the reference method, and to test the expected agreement between estimates made with the model and the reference method. 

It is important to note that theoretic argument and empiric study have shown that the LOO cross-validation approach is preferred to the use of an external test set for small to moderate sized chemical databases [39]. The problems with holding out an external test set include (1) structural features of the held out chemicals are not included in the modeling process, resulting in a loss of information, (2) predictions are made only on a subset of the available compounds, whereas LOO predicts the activity value for all compounds, and (3) personal bias can easily be introduced in selection of the external test set. The reader is referred to Hawkins et al. [39] and Kraker et al. [40] in addition to Section 31.6 for further discussion of proper model validation techniques. 

The full-scale industrial experiment demonstrated the feasibility of a convenient, nonintrusive aconstic chemometric facility for reliable ammonia concentration prediction. The training experimental design spanned the industrial concentration range of interest (0-8%). Two-segment cross-validation (test set switch) showed good accnracy (slope 0.96) combined with a satisfactory RMSEP. It is fully possible to further develop this pilot study calibration basis nntil a fnll industrial model has been achieved. There wonld appear to be several types of analogous chemical analytes in other process technological contexts, which may be similarly approached by acoustic chemometrics. 

The sensor output can be used to test the validity of processing models such as the Loos-Springer model [30]. Sensor measured values of t] can be compared with the Loos-Springer model predictions. Figure 4.14 is a comparison of the model s predictions and the measured values at the sixty-fourth ply. Agreement in the viscosity s time dependence and magnitude with the predictions of models is essential if the model is to be verified and used with confidence. 

Model Validation Validation of the calibration model is crucial before prospective application. Two types of validation schemes can be adopted internal and external. Internal validation, or cross-validation, is used when the number of calibration samples is limited. In cross-validation, a small subset of calibration data is withheld from the model building step. After the model is tested on these validation spectra, a different subset of calibration data is withheld and the b vector is recalculated. Various strategies can be employed for grouping spectra for calibration and validation. For example, a single sample is withheld in a leave-one-out scheme, and the calibration and validation process is repeated as many times as the number of samples in the calibration data set. In general, leave- -out cross-validation can be implemented with n random samples chosen from a pool of calibration data. 

These experiments can be modeled by utilizing processes (VI) (A), (B) and (C), and the associated rate constants. A computer simulation of an experiment such as that shown in Fig. 6 is a stringent test for the validity of a set of rate constants. 

In order to start the multiscale modeling, internal state variables were adopted to reflect void/crack nucleation, void growth, and void coalescence from the casting microstructural features (porosity and particles) under different temperatures, strain rates, and deformation paths [115, 116, 221, 283]. Furthermore, internal state variables were used to reflect the dislocation density evolution that affects the work hardening rate and, thus, stress state under different temperatures and strain rates [25, 283-285]. In order to determine the pertinent effects of the microstructural features to be admitted into the internal state variable theory, several different length scale analyses were performed. Once the pertinent microstructural features were determined and included in the macroscale internal state variable model, notch tests [216, 286] and control arm tests were performed to validate the model s precision. After the validation process, optimization studies were performed to reduce the weight of the control arm [287-289]. 

QSAR model validation mostly serves the purpose of demonstrating the overall prediction quality of the model. In practice, however, the way in which validation is performed largely depends on the model s intended use. If the model is to be applied to a known population of chemicals, regulatory acceptance of the model could depend entirely on the results of validation carried out that is specific to the particular chemical population. The model s validity can be demonstrated by comparing the predicted results with the experimental results on an external test set that is objectively selected from the application population Consequently the unbiased selection of an appropriate test set becomes an essential step in determining the validity of the model. The selected chemicals should represent the diversity of molecular structure and activity of the application population, and the selection process should provide statistically significant data to assess false positives and false negatives. A... 

---