Statistical error and bias

In order to define a reasonable estimator for the statistical error, it is necessary to start from the assumption that an infinite number of independent samples k has been generated. In this case, the distribution of the estimates is Gaussian, according to the central limit theorem of uncorrelated samples. The exact average of the estimates is then given by O). It immediately follows that 

2 Conventional Markov-chain Monte Carlo sampling 

Autocorrelation functions of energy and sguare radius of gyration for data obtained in Metropoiis Monte Cario simulations of a flexible polymer model In the random-coil phase [79], 

If the Monte Carlo updates in each sample are performed completely randomly without memory, i.e., a new conformation is created independently of the one in the step before (which is a possible but typically very inefficient strategy), two measured values Om and 0 are uncorrelated, if m n. Then, the autocorrelation function simplifies to Amn = mn-Thus, the variances of the individual data and of the mean are related with each other by 

Monte Carlo and chain growth methods for molecular simulations 

---

We have seen that Lagrangian PDF methods allow us to express our closures in terms of SDEs for notional particles. Nevertheless, as discussed in detail in Chapter 7, these SDEs must be simulated numerically and are non-linear and coupled to the mean fields through the model coefficients. The numerical methods used to simulate the SDEs are statistical in nature (i.e., Monte-Carlo simulations). The results will thus be subject to statistical error, the magnitude of which depends on the sample size, and deterministic error or bias (Xu and Pope 1999). The purpose of this section is to present a brief introduction to the problem of particle-field estimation. A more detailed description of the statistical error and bias associated with particular simulation codes is presented in Chapter 7. 

As with all statistical methods, the mean-field estimate will have statistical error due to the finite sample size (X ), and deterministic errors due to the finite grid size (S ) and feedback of error in the coefficients of the SDEs Ui,p). Since error control is an important consideration in transported PDF simulations, we will now consider a simple example to illustrate the tradeoffs that must be made to minimize statistical error and bias. The example that we will use corresponds to (6.198), where the exact solution141 to the SDEs has the form ... 

Precision and Accuracy. Precision and accuracy are assay performance characteristics that describe the random (statistical) errors and systematic errors (bias) associated with repeated measurements of the same sample under specified conditions [3-5]. Precision is typically estimated by the percent coefficient of variation (% CV, also referred to as relative standard deviation or RSD) but may certainly also be reported as standard deviations. Method accuracy is expressed as the percent relative error (% RE) and is determined by the percent deviation of the weighted samples mean from samples with nominal reference values. A collection of validation sample statistics can be found in References 9,11, and 25. 

Our considerations in the last section are founded on basic elements of probability theory and we did not have to make special assumptions about the time series or data sets. Having said that and having taken into account that all data sets we can create are finite, the entire problem of statistical errors and systematic bias affects all data obtained in experiments and computer simulations. The error estimation methods discussed in this section are sufficient to judge the quality of these data. ... 

This is a class of algorithms which makes feasible on contemporary computers an exact Monte Carlo solution of the Schrodinger equation. It is exact in the sense that as the number of steps of the random walk becomes large the computed energy tends toward the ground state energy of a finite system of bosons. It shares with all Monte Carlo calculations the problem of statistical errors and (sometimes) bias. In the simulations of extensive systems, in addition, there is the approximation of a uniform fluid by a finite portion with (say) periodic boundary conditions. The latter approximation appears to be less serious in quantum calculations than in corresponding classical ones. 

Confirmation bias is a tendency to search for or interpret information in a way that confirms one s preconceptions, leading to statistical errors. Confirmation bias is a type of cognitive bias (i.e., a bias of the internal mental processes such as memory, thought, perception, and problem solving) and represents an error of inductive inference toward confirmation of the hypothesis under study. 

There are two types of measurement errors, systematic and random. The former are due to an inherent bias in the measurement procedure, resulting in a consistent deviation of the experimental measurement from its true value. An experimenter s skill and experience provide the only means of consistently detecting and avoiding systematic errors. By contrast, random or statistical errors are assumed to result from a large number of small disturbances. Such errors tend to have simple distributions subject to statistical characterization. 

Analytical chemists make a distinction between error and uncertainty Error is the difference between a single measurement or result and its true value. In other words, error is a measure of bias. As discussed earlier, error can be divided into determinate and indeterminate sources. Although we can correct for determinate error, the indeterminate portion of the error remains. Statistical significance testing, which is discussed later in this chapter, provides a way to determine whether a bias resulting from determinate error might be present. 

The "feedback loop in the analytical approach is maintained by a quality assurance program (Figure 15.1), whose objective is to control systematic and random sources of error.The underlying assumption of a quality assurance program is that results obtained when an analytical system is in statistical control are free of bias and are characterized by well-defined confidence intervals. When used properly, a quality assurance program identifies the practices necessary to bring a system into statistical control, allows us to determine if the system remains in statistical control, and suggests a course of corrective action when the system has fallen out of statistical control. 

In this chapter we use the terms precision and accuracy in relation to the finite sampling variance and bias, respectively. Also, we describe the overall quality of an estimator - the mean square error - by the term reliability. Note the difference between our terminology and that in some statistics literature where accuracy is used to describe the overall quality (i.e., the reliability in this chapter). The decomposition of the error into the variance and bias allows us to use different approaches for studying the behavior of each term. 

Perhaps the most challenging part of analyzing free energy errors in FEP or NEW calculations is the characterization of finite sampling systematic error (bias). The perturbation distributions / and g enable us to carry out the analysis of both the finite sampling systematic error (bias) and the statistical error (variance). 

Since the bias function should enhance the sampling of pathways with important work values it can be made to depend on the work only, ir[z 2 ) = n W( (. Z))]. To minimize the statistical error in the free energy difference the bias function needs to be selected such that both the statistical errors of the numerator and the denominator of (7.44) are small. Ideally, the bias function should have a large overlap with both the unbiased work distribution P(W) and the integrand of (7.36), P (W) exp (—j3W). Just as Sun s work-biased ensemble Pa[z( ), the biased path ensemble )] can... 

The section following shows a statistical test (text for the Comp Meth MathCad Worksheet) for the efficient comparison of two analytical methods. This test requires that replicate measurements be made on two different samples using two different analytical methods. The test will determine whether there is a significant difference in the precision and accuracy for the two methods. It will also determine whether there is significant systematic error between the methods, and calculate the magnitude of that error (as bias). 

The behavior of the detection algorithm is illustrated by adding a bias to some of the measurements. Curves A, B, C, and D of Fig. 3 illustrate the absolute values of the innovation sequences, showing the simulated error at different times and for different measurements. These errors can be easily recognized in curve E when the chi-square test is applied to the whole innovation vector (n = 4 and a = 0.01). Finally, curves F,G,H, and I display the ratio between the critical value of the test statistic, r, and the chi-value that arises from the source when the variance of the ith innovation (suspected to be at fault) has been substantially increased. This ratio, which is approximately equal to 1 under no-fault conditions, rises sharply when the discarded innovation is the one at fault. 

In the statistics literature, one usually distinguishes between the estimated mean () and the true (unknown) mean (). Here, in order to keep the notation as simple as possible, we will not make such distinctions. However, the reader should be aware of the fact that the estimate will be subject to statistical error (bias, variance, etc.) that can be reduced by increasing the number of notional particles (Vp. 

As discussed in Section 6.8, the estimation errors can be categorized as statistical, bias, and discretization. In a well designed MC simulation, the statistical error will be controlling. In contrast, in FV methods the dominant error is usually discretization. 

The results of these interlaboratory studies are reported in USEPA Method Validation Studies 14 through 24 (14). The data were reduced to four statistical relationships related to the overall study 1, multilaboratory mean recovery for each sample 2, accuracy expressed as relative error or bias 3, multilaboratory standard deviation of the spike recovery for each sample and 4, multilaboratory relative standard deviation. In addition, single-analyst standard deviation and relative standard deviation were calculated. 

Accuracy. The more accurate the sampling method the better. Given the very large environmental variability, however, sampling and analytical imprecision is rardy a significant contribution to overall error, or width of confidence limits, of the final result. Even highly imprecise methods, such as dust count methods, do not add much to overall variability when the variability between workers and overtime is considered. An undetected bias, however, is more serious because such bias is not considered by the statistical analysis and can, therefore, result in gross unknown error. 

The practices described by the method provide instructions for sampling coal from beneath the exposed surface of the coal at a depth (approximately 24 in., 61 cm) where drying and oxidation have not occurred. The purpose is to avoid collecting increments that are significantly different from the majority of the lot of coal being sampled due to environmental effects. However, samples of this type do not satisfy the minimum requirements for probability sampling and, as such, cannot be used to draw statistical inferences such as precision, standard error, or bias. Furthermore, this method is intended for use only when sampling by more reliable methods that provide a probability sample is not possible. 

Analytical measurements should be made with properly tested and documented procedures. These procedures should utilise controls and calibration steps to minimise random and systematic errors. There are basically two types of controls (a) those used to determine whether or not an analytical procedure is in statistical control, and (b) those used to determine whether or not an analyte of interest is present in a studied population but not in a similar control population. The purpose of calibration is to minimise bias in the measurement process. Calibration or standardisation critically depends upon the quality of the chemicals in the standard solutions and the care exercised in their preparation. Another important factor is the stability of these standards once they are prepared. Calibration check standards should be freshly prepared frequently, depending on their stability (Keith, 1991). No data should be reported beyond the range of calibration of the methodology. Appropriate quality control samples and experiments must be included to verify that interferences are not present with the analytes of interest, or, if they are, that they be removed or accommodated. 

Shortfalls of the common control-based HTS data analysis approach have been suggested (Gribbon et al., 2005 Malo et al., 2006). For example plate-based normalization using controls (Table 14.1) intrinsically assumes a random error distribution for all wells in a plate because of the positional bias of the controls dictated by the formats of common screening libraries. Edge effects are particularly relevant in cell-based assays and vary from plate to plate (Malo et al., 2006). Classical HTS analysis relies on non-robust statistics means and standard deviations are greatly influenced by outliers. 

The pharmaceutical physician may not be expected to be a specialist statistician, and statistics are not the subject of this chapter. However, the ability to talk to and understand statisticians is absolutely essential. Sine qua non Involve a good statistician from the moment a clinical trial is contemplated. Furthermore, the pharmaceutical physician should be confident of a sound understanding of the concepts of type I and type II error, and the probabilities a and P (e.g. Freiman et al., 1978). This is one of your best defences against bias. 

---