Parameter errors, model validation testing

The process of field validation and testing of models was presented at the Pellston conference as a systematic analysis of errors (6. In any model calibration, verification or validation effort, the model user is continually faced with the need to analyze and explain differences (i.e., errors, in this discussion) between observed data and model predictions. This requires assessments of the accuracy and validity of observed model input data, parameter values, system representation, and observed output data. Figure 2 schematically compares the model and the natural system with regard to inputs, outputs, and sources of error. Clearly there are possible errors associated with each of the categories noted above, i.e., input, parameters, system representation, output. Differences in each of these categories can have dramatic impacts on the conclusions of the model validation process. 

Based on the performed work, a sensitivity analysis has been carried out of the model parameters. The nonlinearity of the parameters has been mapped out. Then the model has been evaluated at different working temperatures, current rates and at different values for the state of charge. In the temperature range 25-40 °C, the simulation results are in good agreement with the experimental ones. However, the error increases as the temperature decreases. In order to keep the error in an acceptable range, the thermal model has been associated to an electrical model. Based on such approach (electrothermal model), the error for operations at 0 °C is in the range of 0-1 °C. Then a number of validation tests have confirmed the performances and the accuracy of the model. 

Finally, the prediction error model assumes that the parameter values do not change with respect to time, that is, they are time invariant. A quick and simple test of the invariance of the model is to split the data into two parts and cross validate the models using the other data set. If both models perform successfully, then the parameters are probably time invariant, at least over the time interval considered. 

In summary, the support vector machine (SVM) and partial least square (PLS) methods were used to develop quantitative structure activity relationship (QSAR) models to predict the inhibitory activity of nonpeptide HIV-1 protease inhibitors. Cenetic algorithm (CA) was employed to select variables that lead to the best-fitted models. A comparison between the obtained results using SVM with those of PLS revealed that the SVM model is much better than that of PLS. The root mean square errors of the training set and the test set for SVM model were calculated to be 0.2027, 0.2751, and the coefficients of determination (R2) are 0.9800, 0.9355 respectively. Furthermore, the obtained statistical parameter of leave-one-out cross-validation test (Q ) on SVM model was 0.9672, which proves the reliability of this model. Omar Deeb is thankful for Al-Quds University for financial support. 

After the determination of the best-fit parameter values, the validity of the selected model should be tested does this model describe the available data properly or are there still indications that a part of the data is not explained by the model, indicating remaining model errors We need the means to assess whether or not the model is appropriate, that is, we need to test the goodness-af-fit against some useful statistical standard. 

The model was built with one LVs (Y explained variance in cross validation, LOO, 84.38%). Analogously to the unfold-PLS case, the robustness and the predictive capability of the model were tested by leaving one producer out (RMSEP-LOP) procedure. In all the six cases, the models built with one LV gave the best results in terms of lowest root mean squares cross-validation error. Considering the RMSEP-LOP values for each sensory parameter for the different models (data not reported), they have the same trend of the respective unfolding analysis with numerical values smaller than the previous one. 

PLS is best described in matrix notation where the matrix X represents the calibration matrix (the training set, here physicochemical parameters) and Y represents the test matrix (the validation set, here the coordinates of the odor stimulus space). If there are n stimuli, p physicochemical parameters, and m dimensions of the stimulus space, the equations in Figure 6a apply. The C matrix is an m x p coefficient matrix to be determined and the residuals not explained by the model are contained in E. The X matrix is decomposed as shown in Figure 6b into two small matrices, an n x a matrix T and an a x p matrix B where a << n and a << p. F is the error matrix. The computation of T is such that it both models X and correlates with T and is accomplished with a weight matrix W and a set of latent variables U for Y with a corresponding loading matrix B. ... 

By fitting experimental data for different deformation modes to these functions, the three network parameters of unfilled polymer networks Gc, Ge, and ne/Te can be determined. The validity of the concept can be tested if the estimated fitting parameters for the different deformation modes are compared. A plausibility criterion for the proposed model is formulated by demanding that all deformation modes can be described by a single set of network parameters. The result of this plausibility test is depicted in Fig. 44, where stress-strain data of an unfilled NR-vulcanizate are shown for the three different deformation modes considered above. Obviously, the material parameters found from the fit to the uniaxial data provide a rather good prediction for the two other modes. The observed deviations are within the range of experimental errors. 

Figures 11 and 12 illustrate the performance of the pR2 compared with several of the currently popular criteria on a specific data set resulting from one of the drug hunting projects at Eli Lilly. This data set has IC50 values for 1289 molecules. There were 2317 descriptors (or covariates) and a multiple linear regression model was used with forward variable selection the linear model was trained on half the data (selected at random) and evaluated on the other (hold-out) half. The root mean squared error of prediction (RMSE) for the test hold-out set is minimized when the model has 21 parameters. Figure 11 shows the model size chosen by several criteria applied to the training set in a forward selection for example, the pR2 chose 22 descriptors, the Bayesian Information Criterion chose 49, Leave One Out cross-validation chose 308, the adjusted R2 chose 435, and the Akaike Information Criterion chose 512 descriptors in the model. Although the pR2 criterion selected considerably fewer descriptors than the other methods, it had the best prediction performance. Also, only pR2 and BIC had better prediction on the test data set than the null model.

The principal reason that a test set is necessary for validation is that empirical model-building methods cannot readily distinguish between noise and information in data sets, so the methods are prone to adjusting the model parameters to reduce error beyond the point warranted by the information contained in the data. This problem is called overtraining and can be countered by a variety of techniques such as descriptor reduction and early stopping, and readers interested in those topics are referred to the more detailed reviews of numerical methods cited in each of the following sections. 

To illustrate the validity of the models presented in the previous section, results of validation experiments using lab-scale BSR modules are taken from Ref. 7. For those experiments, the selective catalytic reduction (SCR) of nitric oxide with excess ammonia served as the test reaction, using a BSR filled with strings of a commercial deNO catalyst shaped as hollow extrudates (particle diameter 1.6 or 3.2 mm). The lab-scale BSR modules had square cross sections of 35 or 70 mm. The kinetics of the model reaction had been studied separately in a recycle reactor. All parameters in the BSR models were based on theory or independent experiments on pressure drop, mass transfer, or kinetics none of the models was later fitted to the validation experiments. The PDFs of the various models were solved using a finite-difference method, with centered differencing discretization in the lateral direction and backward differencing in the axial direction the ODEs were solved mostly with a Runge-Kutta method [16]. The numerical error of the solutions was... 

The study of elementary reactions for a specific requirement such as hydrocarbon oxidation occupies an interesting position in the overall process. At a simplistic level, it could be argued that it lies at one extreme. Once the basic mechanism has been formulated as in Chapter 1, then the rate data are measured, evaluated and incorporated in a data base (Chapter 3), embedded in numerical models (Chapter 4) and finally used in the study of hydrocarbon oxidation from a range of viewpoints (Chapters 5-7). Such a mode of operation would fail to benefit from what is ideally an intensely cooperative and collaborative activity. Feedback is as central to research as it is to hydrocarbon oxidation Laboratory measurements must be informed by the sensitivity analysis performed on numerical models (Chapter 4), so that the key reactions to be studied in the laboratory can be identified, together with the appropriate conditions. A realistic assessment of the error associated with a particular rate parameter should be supplied to enable the overall uncertainty to be estimated in the simulation of a combustion process. Finally, the model must be validated against data for real systems. Such a validation, especially if combined with sensitivity analysis, provides a test of both the chemical mechanism and the rate parameters on which it is based. Therefore, it is important that laboratory determinations of rate parameters are performed collaboratively with both modelling and validation experiments. 

In most cases, the fixed effect parameters are the parameters of interest. However, adequate modeling of the variance-covariance structure is critical for assessment of the fixed effects and is useful in explaining the variability of the data. Indeed, sometimes the fixed effects are of little interest and the variance components are of primary importance. Covariance structures that are overparameterized may lead to poor estimation of the standard errors for estimates of the fixed effects (Altham, 1984). However, covariance matrices that are too restrictive may lead to invalid inferences about the fixed effects because the assumed covariance structure does not exist and is not valid. For this reason, methods need to be available for testing the significance of the variance components in a model. 

Most importantly, however, the use of the experimental errors allows an objective judgement of the agreement between model and data, i.e., the validity of the conceptual model that was adopted to describe the data. The model selection is based on the x test. The expected minimum value of is the number of degrees of freedom v = n - m, where n is the number of data points and m is the number of free parameters. The probability p for to be higher than a given value due to random analytical errors, although the model description is correct, can be obtained from the x -distribution with v degrees of freedom. If p is lower than some cut-off value pc (pc = 0.01 proved to be appropriate), the model is rejected. 

---