Precision of Calculations

Computations, whether by hand calculator or by computer, should be done with awareness of certain principles concerning the effects of arithmetic operations. These follow directly from the principles underlying the propagation of errors, discussed in Part B of this chapter. 

When a number of numerical values are added, the precision of the result can be no greater than that of the least precise numerical value involved. Thus, generally, the number of decimal places in the result should be the same as the number of decimal places in the component with the fewest  

When the difference between two numbers yields a result that is relatively small, that difference not only has a precision limited to that of the less precise number but has also a relative precision much less than either number  

The less precise of the two numbers has a relative precision of about 1 in 67,000 (0.0015 percent), while the difference has a relative precision of about 1 in 50 (2 percent). The effect can be particularly devastating in an intermediate calculation on a computer, where the user of the computer may be blissfully unaware of the loss of relative precision. 

The uncertainty in 121, the least relatively precise figure, is about 0.8 percent. One in 377 is about 0.27 percent, which implies a greater relative precision, but if the result is rounded to 3.8 X 10, with a relative precision of 2.6 percent, too much significance will be lost. The principles are no different when division is involved  

---

Anderson, G.M., 1976. The accuracy and precision of calculated mineral dehydration equilibria, in D.G. Fraser, ed., Thermodynamics in Geology, Proceedings of NATO Advanced Study Institute. Oxford, D. Reidel Publishers, pp. 115-136. 

A precise calculation of the volume or surface area of a solid body of regular geometric shape can only be made when its length, breadth and thickness are known. For particulate solids in general, these three dimensions can never be precisely measured. Therefore, before a brief account is given of some of the methods of calculation available, a word of warning is necessary. It must be fully appreciated that the precision of calculation is always far greater than that of measurement of the various quantities used in the mathematical expressions. An equation, especially a complex one, always has a look of absolute dependability, but in this particular connection it most certainly leads to a false sense of security. All calculated volume or surface area data must be used with caution. 

The charges on the boundary are found Ifom the electrostatic polarization of the dielectric medium on the surface of the cavity due to the potential derived Ifom the charge distribution of the solute and Ifom other (induced) charges on the surface. The induced surface charge is evaluated iteratively at each step of the SCF procedure to solve the Schrodinger equation [ 11.1.78]. It has been reported that a simultaneous iteration of the surface charge with the Fock procedure reduces substantially the computation time without the loss in the precision of calculations. 

Sun CC (2007) Thermal expansion of organic crystals and precision of calculated crystal density. J Pharm Sci 96 1043-1052... 

The following experimental data are generally considered essential in developing an accurate equation of state ideal gas heat capacities Cf,% expressed as functions of temperature T, vapour pressure and density p data in all regions of the thermodynamic surface. Precise speed of sound w data in both the liquid and vapour phases have recently become important for the development of equations of state. The precision of calculated energies can be improved if the following data are also available Cy,m p, T) (isochoric heat capacity measurements), Cp,m(p, T) (isobaric heat capacity measurements), T) (enthalpy differences), and Joule-Thomson coefficients. 

Validity of equation (3.58) is limited to the interval in which the empirical relationship for molar heat are valid, the precision of the correlation of molar heat values being decisive for the precision of calculated equilibrium constant data. For practical purposes this precision is usually satisfactory. For illustration. Fig. 2 shows the course of the temperature dependence of equilibrium constants for some reactions which are of significance in industry. 

THE ACCURACY AND PRECISION OF CALCULATED MINERAL DEHYDRATION EQUILIBRIA... 

Yang et al. described the development of a sensitive method for the accurate and precise quantitative determination of TBT and DBT in sediments by species-specific isotope dilution ICP-MS. Using GC for sample introduction and analyte separation, detection with quadrupole ICP-MS and sector-field ICP-MS were compared. A more than two-fold improvement in precision of calculated °Sn/ Sn ratios was obtained for both TBT and DBT in standards using GC-ICP-SF-MS as compared to GC coupled with quadrupole ICP-MS. Superior limits of detection were obtained for SF-ICP-MS coupling because of the improved signal-to-background ratio. 

Calculation of the integral through Eqs. (4.94) and (4.98) imply that instead of evaluating the function at Zj, = -1 and Zj = 1 (using function values at base points), which is the case in the trapezoidal rule, function values at Zo = -l/ 3 and z, = l/v 3 should be used in the Gauss quadrature method. This results in improving the precision of calculation, as illustrated in Fig, 4,5. This is roughly equivalent to the application of five-point trapezoidal rule. 

---