Model independent tests

10 Phenomenology of semi-leptonic reactions 10.1 Model independent tests 

What is the evidence for the existence of neutral current events involving hadrons  

Is this evidence consistent with the structure implied by (9.3.1), in particular its relationship to charged current events  

In the first place the very existence of neutral cvurrent (NC) events involving hadrons is highly significant. All of the reactions t p, 

A recent measurement (Ahrens et al., 1987) gives for the elastic events, 

---

Finally, a model has to be tested using an independent data set with compounds yet completely unknown to the model the test set. The complete process of building a prediction model is depicted in Figure 10.1-1 as a flow chart. 

EC verification provides an alternative to the model of establishing a certified production QA system. Independent testing of either all devices, or a statistically representative sample of each batch, is conducted by or on behalf of the Notified Body, which then issues a certificate of conformity for the tests conducted. This is not a popular option due to the costs involved. The procedure is not capable of providing adequate assurance as to the sterility of devices. Instead, an assurance of sterility must be based on the application of a production QA system to the sterilisation process. 

Although all the underlying assumptions (local linearity, statistical independence, etc.) are rarely satisfied, Bartlett s jf-test procedure has been found adequate in both simulated and experimental applications (Dumez et al., 1977 Froment, 1975). However, it should be emphasized that only the x2-test and the F-test are true model adequacy tests. Consequently, they may eliminate all rival models if none of them is truly adequate. On the other hand, Bartlett s x2-test does not guarantee that the retained model is truly adequate. It simply suggests that it is the best one among a set of inadequate models ... 

Verification is the complement of calibration model predictions are compared to field observations that were not used in calibration or fidelity testing. This is usually the second half of split-sample testing procedures, where the universe of data is divided (either in space or time), with a portion of the data used for calibration/fidelity check and the remainder used for verification. In essence, verification is an independent test of how well the model (with its calibrated parameters) is representing the important processes occurring in the natural system. Although field and environmental conditions are often different during the verification step, parameters determined during calibration are not adjusted for verification. 

As we hinted above, the most common use of the term validation involves simply retaining some samples separately from the main set of calibration samples and using those as a more-or-less independent test of the accuracy of the calibration model obtained. However, this definition is not universally agreed to. When the subject came up in the on-line discussion group, the following comment was made by Richard Kramer of the discussion group [1] ... 

For Wolf-Rayet stars of type WN, rotation makes smoother changes of abundances, due to internal mixing. For WN stars, the transition phase, with still H present, becomes longer due to rotation and this increases the late WN phase (WNL) where H is usually present. The CNO abundances at the end of the WN phase are the same for rotating and non-rotating models, because they are model independent and determined just by CNO nuclear equilibrium. Indeed, CNO abundances in WN stars provide a unique test of the physics of the CNO cycle. 

The most direct, model independent, way to test the validity of the mixing solution is to measure the 3He abundance in the ejecta of low-mass stars, i.e. in planetary nebulae (PNe). The search for 3He in the ejecta of PNe via the 8.667 GHz spin-flip transition of 3He+, painstakingly carried out by Rood and coworkers at the Green Bank radiotelescope since 1992 (see summary of results in Balser et al. 1997), has produced so far one solid detection (NGC 3242, see Rood, Bania, Wilson 1992 confirmed with the Effelsberg radiotelescope by... 

The resulting dependences of [Th/Eu] on stellar age are shown in Fig. 10.5, together with a selection of observational data from del Peloso et al. for disk stars Westin et al. (2000) for two of the r-process enriched ultra-metal-poor (UMP) halo stars and Cowan et al. (2002) for a third one. The range among disk stars is little more than can be expected from uncertainties in the determination, while the UMP stars show more or less the expected deficiency. For BD +17 0 3248, Cowan et al. used theoretical production ratios for Th relative to Eu, Ir and Pt, based on certain nuclear models, to deduce a Galactic-model-independent age of 13.8 4 Gyr, which happens to fit our two production-ratio-independent model curves simple inflow and modFowler T = 15 quite nicely, but the error bars are large so that what we have is more a test of consistency than an independent chronometer. 

Model-independent techniques compare data pairs observed at corresponding time values, where time is only a class effect, as in a paired -test or in an ANOVA. A data-poor set of only two or three observations, originating from routine quality control of an immediate-release dosage form, cannot be treated other than model independent. 

If the data are recorded at corresponding time values, an alternative is to treat them in a way similar to paired differences as in a paired f-test or in an ANOVA, where time is not considered as continuous independent variable but only as a class effect. The result is a model-independent index, which... 

For formulations not meeting the criterion for very fast release of drug substance, similarity of profiles may be evaluated by model-independent or model-dependent methods as stated in the Guidance for Industry—Dissolution Testing of IR Solid Oral Dosage Forms (1,2). 

The most common approach for the comparison of dissolution profiles is model-independent approach using the similarity factor f2. The pre-requisites for using the /2-test are the following ... 

Alternative methods and algorithms may be used, such as the model-independent approach to compare similarity limits derived from multi-variate statistical differences (MSD) combined with a 90% confidence interval approach for test and reference batches (21). Model-dependent approaches such as the Weibull function use the comparison of parameters obtained after curve fitting of dissolution profiles. See Chapters 8 and 9 for further discussion of these methods. 

The basis of all performance criteria are prediction errors (residuals), yt - yh obtained from an independent test set, or by CV or bootstrap, or sometimes by less reliable methods. It is crucial to document from which data set and by which strategy the prediction errors have been obtained furthermore, a large number of prediction errors is desirable. Various measures can be derived from the residuals to characterize the prediction performance of a single model or a model type. If enough values are available, visualization of the error distribution gives a comprehensive picture. In many cases, the distribution is similar to a normal distribution and has a mean of approximately zero. Such distribution can well be described by a single parameter that measures the spread. Other distributions of the errors, for instance a bimodal distribution or a skewed distribution, may occur and can for instance be characterized by a tolerance interval. 

Typically, the final part of QSAR model development is the model validation [17, 18], when the predictive power of the model is tested on an independent set of compounds. In essence, predictive power is one of the most important characteristics of QSAR models. It can be defined as the ability of a model to predict accurately the target property (e.g., biological activity) of compounds that were not used for model development. The typical problem of QSAR modeling is that at the time of the model development a researcher only has, essentially, training set molecules, so predictive ability can only be characterized by statistical characteristics of the training set model and not by true external validation. 

Conceptually, the value for a given sample reflects the extremeness of that sample s response within the PCA model space, whereas the Q valne reflects the amonnt of the sample s response that is outside of the PCA model space. Therefore, both metrics are necessary to fnlly assess the abnormality of a response. In practice, before one can nse a PCA model as a monitor, one mnst set a confidence limit on each of these metrics. There are several methods for determining these confidence limits [30,31], bnt these nsually require two sets of information (1) the set of and Q values that are obtained when the calibration data (or a suitable set of independent test data) is applied to the PCA model, and (2) a user-specified level of confidence (e.g. 95%, 99%, or 99.999%). Of conrse, the latter is totally at the discretion of the nser, and is driven by the desired sensitivity and specificity of the monitoring application. 

As a theoretical model, diastereotopically distinct isotopic substitution in 1,3-butadiene was utilized by Bach and coworkersl9b to provide an independent test of the ability of high-level ab initio calculations to accurately predict a transition structure for epoxidation. The calculated KIE for deuterium substitution at the a-carbon in the TS for epoxidation of 1,3-butadiene (Figure 24) (Ha) is 0.99, in excellent agreement with the experimental value for an aryl-substituted styrene. The KIE for diastereotopically distinct hydrogen (Ht,) on the /J-carbon cis to the vinyl substituent is 0.80, while that for Hc is predicted to be 0.82. The calculated KIE is 0.66 for this transition structure with Hc, reflecting the extensive... 

HadCM2 and CGCM models fail. They failed an independent test designed to verily whether my original criticism—thatthe models were no better than noise, or random numbers—was correct So, in summary, the USNA was sufficiently concerned about my criticism that, in spite of its public brush-off, it specifically tested my hypothesis and independently verified the finding that the... 

Savolainen et al. investigated the role of Raman spectroscopy for monitoring amorphous content and compared the performance with that of NIR spectroscopy [41], Partial least squares (PLS) models in combination with several data pre-processing methods were employed. The prediction error for an independent test set was in the range of 2-3% for both NIR and Raman spectroscopy for amorphous and crystalline a-lactose monohydrate. The authors concluded that both techniques are useful for quantifying amorphous content however, the performance depends on process unit operation. Rantanen et al. performed a similar study of anhydrate/hydrate powder mixtures of nitrofurantoin, theophyllin, caffeine and carbamazepine [42], They found that both NIR and Raman performed well and that multivariate evaluation not always improves the evaluation in the case of Raman data. Santesson et al. demonstrated in situ Raman monitoring of crystallisation in acoustically levitated nanolitre drops [43]. Indomethazine and benzamide were used as model... 

Cherkasov, 2005 a (79) for descriptors only Artificial neural networks (ANN)3 44 (77) Random peptides chosen according to two amino acid frequency distributions Sets A and B contained 933 and 500 peptides, respectively (see text for details, unpublished data) Training and validation within one set, independent testing on second set 1433 Set A models predicted activity with up to 83% accuracy on Set B Set B models predicted up to 43% accuracy on Set A (see text for details) nd... 

It is evident that the dipolar interaction increases the activation energy and slows the relaxation time of moment reversal. To explain the dependence quantitatively a simple model was proposed [26], inspired by the references contained in [22], where the relaxation time considered is for independent clusters and the dipolar interaction modifies Ueff. In the model, a test cluster of uniaxial anisotropy is surrounded by clusters with their anisotropy axis randomly oriented. The susceptibility is dominated by the largest particles in a sample since it is proportional... 

Instead of validating the predictions internally, it is possible to test the predictions against an independent data set, often called a test set. Computationally the procedure is similar to cross-validation. For example, a model is obtained using / samples, and then the predictions are calculated using an independent test set of L samples, to give... 

The model overestimated final liver chromium concentrations, but bone and kidney concentrations were well-predicted. This was not a completely independent test of the model s validity since data from this study were used to set parameters for fractional uptake of chromium into bone. 

---