Repeatability, statistical validation

Figure 10.6. Principle of DIGE analysis separation of control and treated sample on one gel and statistical validation using more than three repeated experiments. Printed by kind permission of GE Healthcare (formerly Amersham Biosciences). (See color insert.)...

Now we can really see why the CSTR operated at steady state is so different from the transient batch reactor. If the inlet feed flow rates and concentrations are fixed and set to be equal in sum to the outlet flow rate, then, because the volume of the reactor is constant, the concentrations at the exit are completely defined for fixed kinetic parameters. Or, in other words, if we need to evaluate kab and kd, we simply need to vary the flow rates and to collect the corresponding concentrations in order to fit the data to these equations to obtain their magnitudes. We do not need to do any integration in order to obtain the result. Significantly, we do not need to have fast analysis of the exit concentrations, even if the kinetics are very fast. We set up the reactor flows, let the system come to steady state, and then take as many measurements as we need of the steady-state concentration. Then we set up a new set of flows and repeat the process. We do this for as many points as necessary in order to obtain a statistically valid set of rate parameters. This is why the steady-state flow reactor is considered to be the best experimental reactor type to be used for gathering chemical kinetics. 

In practice, the initial estimation of the control limits may require an iterative process in securing control limits that are statistically valid. In particular, it may be necessary to estimate the limits initially. If a given A or R plots beyond the initial control limits, it is recommended that the cause of the extraordinary point be resolved and then removed from the calculation. That is, the control limits should be recalculated with the out-of-control point(s) removed. In some extreme cases, this iterative process may be repeated several times. Once the observed data are contained within the trial limits, continued production should be used to collect more data to further validate the limits. Once process stability is obtained, that is control is maintained for an extended period, the new control limits should become the operational standard. Furthermore, once the trial limits are established, the AT T Runs Rules should be deployed in totahty.to insure protection against process disturbances. 

For each of the seven reactor criticals, calculations were repeated using valid starting seeds. Results of these calculations are given in Table 8. All calculations were repeated at least once both Surry EF and TMIBZ were recalculated three times since the results of these cases seem the most inconsistent relative to the other five critical calculations. The Ak column of Table 9 shows the difference between the nominal value of k computed here relative to the result reported in Table 7. Thus, these calculations were found to be consistent with the results of the original base cases within statistical uncertainty bounds. Hence, all values of are considered to be spatially converged. 

The WEKA software suite [23] has been used in carrying out the experiments. The results were evaluated using Accuracy (Acc). For the training and validation steps, we used k-fold cross-validation with k = 10. Cross-validation is a robust validation method for variable selection [24]. Repeated cross-validation (as calculated by the WEKA environment) allows robust statistical tests. We also use the measurement provided automatically by WEKA Coverage of cases (0.95 level). 

The nanotribological properties of the SAM films generated from CnOH, CnCHs, and CsCH3 SAM films were characterized through the measurement of frictional forces with a plasma-cleaned silicon nitride tip as a function of load under distilled water. Representative friction-versus-load plots obtained by AFM are presented in Fig. 4a. The force-versus-distance curves obtained from the same three SAM films are shown in Fig. 4b. In order to ensure a statistically valid comparison, the measurements were repeated several times in varying order using the same tip/... 

Sometimes, experiments are repeated with a particular set of levels for all the factors to check the statistical validation and repeatability by the replicate data. This is called replication. To get rid of any bias, allocation of experimental material and the order of experimental runs are randomly selected. This is called randomisation. To arrange the experimental material into groups, or blocks, that should be more homogeneous than the entire set of material is called blocking. So, when experiments are carried ont these things should be remembered. There are several methodologies for design of experiments. Some DOE methods are presented below. 

One performs so many repeat measurements at each concentration point that standard deviations can be reasonably calculated, e.g., as in validation work the statistical weights w, are then taken to be inversely proportional to the local variance. The proportionality constant k is estimated from the data. 

In most studies, phytoestrogen intake has been estimated by direct methods that evaluate food intake either by recall (food-frequency questionnaires -FFQs) or by record (food diary), and subsequently by composition databases based on information of this kind. Food-frequency questionnaires are widely administered to subjects involved in epidemiological studies. Their validity and reproducibility is considered sufficient when statistically correlated to data obtained from dietary records (a properly-completed and comprehensive food diary) and from analysis of blood and urine samples (Kirk et ah, 1999 Huang et al, 2000 Yamamoto et al, 2001 Verkasalo et al, 2001). FFQs can be repeated several times a year and may be administered to large populations. Such an approach provides an easy and low-cost method of assessing the... 

Table 14 can be regarded as providing a reasonable overall picture, even if the results cannot applied to any particular case. However, if the underlying principle is accepted, it becomes clear that improvements in a single stage, for example the reduction of instrument variation, has a negligible beneficial effect (if this variation was not outside the normal range ). Even if the contribution of repeatability is re-duced to zero, the cumulative uncertainty is reduced by 10% only, i.e. from 2.2 to y(0.0)2 (0.8)2 (1.0)2 + (1.5)2 = 2.0. This statistical view of errors should help to avoid some unnecessary efforts to improve, e.g., calibration. Additionally, this broad view on all sources of error may help to detect the most important ones. Consequently, without participation in proficiency tests, any method validation will remain incomplete. 

Cross validation and bootstrap techniques can be applied for a statistically based estimation of the optimum number of PCA components. The idea is to randomly split the data into training and test data. PCA is then applied to the training data and the observations from the test data are reconstmcted using 1 to m PCs. The prediction error to the real test data can be computed. Repeating this procedure many times indicates the distribution of the prediction errors when using 1 to m components, which then allows deciding on the optimal number of components. For more details see Section 3.7.1. 

A widely used approach to establish model robustness is the randomization of response [25] (i.e., in our case of activities). It consists of repeating the calculation procedure with randomized activities and subsequent probability assessments of the resultant statistics. Frequently, it is used along with the cross validation. Sometimes, models based on the randomized data have high q values, which can be explained by a chance correlation or structural redundancy [26]. If all QSAR models obtained in the Y-randomization test have relatively high values for both and LOO (f, it implies that an acceptable QSAR model cannot be obtained for the given dataset by the current modeling method. 

A two-component phase Doppler interferometer (PDI) was used to determine droplet size, velocity, and number density in spray flames. The data rates were determined according to the procedure discussed in [5]. Statistical properties of the spray at every measurement point were determined from 10,000 validated samples. In regions of the spray where the droplet number density was too small, a sampling time of several minutes was used to determine the spray statistical characteristics. Results were repeatable to within a 5% margin for mean droplet size and velocity. Measurements were carried out with the PDI from the spray centerline to the edge of the spray, in increments of 1.27 mm at an axial position (z) of 10 mm downstream from the nozzle, and increments of 2.54 mm at z = 15 mm, 20, 25, 30, 35, 40, 50, and 60 mm using steam, normal-temperature air, and preheated air as the atomization gas. 

This distribution, together with the numerical value of the statistic, allows an assessment of how unusual the data are, assuming that the hypothesis is valid. The p value is the probability that the observed value of the statistic (or values even more extreme) occur. The data are declared significant at a particular level (a) if p < a, the data are considered sufficiently unusual relative to the hypothesis and the hypothesis is rejected. Standard, albeit arbitrary, values of a are taken as 0.05 and 0.01. Let us suppose that a particular data set gives p = 0.02. From the frequentist vantage, this means that, if the hypothesis were true and the whole experiment were to be repeated many times under identical conditions, in only 2% of such trials would the value of the statistic be more unusual or extreme than the value actually observed. One then prefers to believe that the data are not, in fact, unusual and concludes that the assumed hypothesis is untenable. 

In chapter 2 I introduced the statistics of repeated measurements. Here I describe how these statistics are incorporated into a quality control program. In a commercial operation it is not always feasible to repeat every analysis enough times to apply t tests and other statistics to the results. However, validation of the method will give an expected repeatability precision (sr), and this can be used to calculate the repeatability limit (r), the difference between duplicate measurements that will only be exceeded 5 times in every 100 measurements. 

In any case, the cross-validation process is repeated a number of times and the squared prediction errors are summed. This leads to a statistic [predicted residual sum of squares (PRESS), the sum of the squared errors] that varies as a function of model dimensionality. Typically a graph (PRESS plot) is used to draw conclusions. The best number of components is the one that minimises the overall prediction error (see Figure 4.16). Sometimes it is possible (depending on the software you can handle) to visualise in detail how the samples behaved in the LOOCV process and, thus, detect if some sample can be considered an outlier (see Figure 4.16a). Although Figure 4.16b is close to an ideal situation because the first minimum is very well defined, two different situations frequently occur ... 

Again, the minimum and maximum loading configurations should be studied. Thermocouples will be placed both inside and outside the container at the cool spot location(s), in the steam exhaust line, and in constant-temperature baths outside the chamber. The F0 value will be calculated based on the temperature recorded by the thermocouple inside the container at the coolest area of the load. Upon completion of the cycle, the F0 value will indicate whether the cycle is adequate or if alterations must be made. Following the attainment of the desired time-temperature cycle, cycles are repeated until the user is satisfied with the repeatability aspects of the cycle validation process. Statistical analysis of the F0 values achieved at each repeated cycle may be conducted to verify the consistency of the process and the confidence limits for achieving the desired F0 value. 

The jackknife method is based on an idea similar to cross-validation. The calculation of the statistical model is repeated g times holding out 1/gth of the data each time. In the end, each element has been held out once and once only (exactly as in cross-validation). Thus, a number of estimates of each parameter is obtained, one for each calculation round. It has been proposed that the quantity... 

---