Rounding-off procedure

The above rounding-off procedure applies to one-step calculations. In chain calculations, that is, calcnlations involving more than one step, we use a modified procedure. Consider the following two-step calcnlation ... 

Comment The valne differs slightly from the one listed in Table 15.3 because of the rounding-off procedure we used in the calculation. 

Rounding-off procedure Added reference to ASTM E 29, Standard Practice for using Significant Digits in Test Data to Determine Conformance With Specifications. 

All numerical computations inevitably involve round-off errors. This error increases as the number of calculations in the solution procedure is increased. Therefore, in practice, successive mesh refinements that increase the number of finite element calculations do not necessarily lead to more accurate solutions. However, one may assume a theoretical situation where the rounding error is eliminated. In this case successive reduction in size of elements in the mesh should improve the accuracy of the finite element solution. Therefore, using a P C" element with sufficient orders of interpolation and continuity, at the limit (i.e. when element dimensions tend to zero), an exact solution should be obtaiiied. This has been shown to be true for linear elliptic problems (Strang and Fix, 1973) where an optimal convergence is achieved if the following conditions are satisfied ... 

In real life, other problems involving discrete variables may not be so nicely posed. For example, if cost is a function of the number of discrete pieces of equipment, such as compressors, the optimization procedure cannot ignore the integer character of the cost function because usually only a small number of pieces of equipment are involved. You cannot install 1.54 compressors, and rounding off to 1 or 2 compressors may be quite unsatisfactory. This subject will be discussed in more detail in Chapter 9. 

The process of achieving agreement between an observed value and the repeatability (2.3.7) of the analytical procedure. The maximum rounding off interval is equal to the largest decimal unit determined to be smaller than half... 

Values which contain the information consistent with either the repeatability or reproducibility of the analytical procedure. Significant values are obtained by using the described method for rounding off (Section 8.3.1). 

Different rounding off rules are needed for addition (and its reverse, subtraction) and multiplication (and its reverse, division). In both procedures we round off the answers to the correct number of significant figures. 

An alternative procedure is to round off the individual numbers before performing the arithmetic operation, retaining only as many columns to the right of the decimal as would give a digit in every item to be added or subtracted. Examples (2), (3), and (4) above would be done as follows ... 

Apply this correction to all flash point determinations between 200 and 325 F (93.3 and 168C). Round off corrections to the nearest yhole number according to Recommended Practice E 29 Note 8 The calibration procedure provided in this method will cancel out the effect of barometric pressure if calibration and tests are run at same pressure... 

Regression Analysis. The GLM (General Linear Models) procedure of SAS was used to fit the experimental data to Equation 3. This procedure provides estimates of coefficients and intercept GLM also tests hypotheses and indicates the overall quality of the correlation. Output from the GLM procedure is shown in Tables IV, V, and VI numbers, which are listed to 6 decimal places in the original output, have been rounded off to If- places. 

In practice, we do not know the exact solution of the problem, and thus we cannot detennine the magnitude of tlie error involved in the numerical method. Knowing that the global discretization error is proportional to the step size is not much help either since there i.s no easy way of determining the value of the proportionality constant. Besides, the global discretization error alone is meaningless without a true estimate of the round-off error. Therefore, we recommend (he following practical procedures to assess the accuracy of the results obtained by a numerical method. 

---