Precision
Ron Graham
with Erk Jensen
We make two common errors regarding numerical precision:

  1. We accept calculated numbers, such as in spreadsheets, as they are without regard to the significant digits in the inputs used to calculate them.
  2. We present highly precise numbers as though they demonstrate how well we've found an answer. Precision can reflect a level of attention to detail that just isn't there.

These errors arise from the fact that precision has little bearing on the accuracy of a number, i.e. whether it's right or wrong. All precision tells us is the extent of the number's right- or wrong-ness. When we present highly precise numbers, we reflect a sense of accuracy and attention to detail that may not really be there; we may also distract our audience from the real story the numbers tell.

Excess precision in presented numbers can have unforeseen effects.

  • The human brain is limited in its ability to process information. As we sort and interpret the numbers we see, we feel the effects of
    • how many numbers are presented
    • how many significant digits there are in each
    • the range of the numbers
    • the relative importance of each
    • the speed at which they must be processed
  • Some tolerances are not achievable at finite cost. A tolerance is just one example of the practical effects of precision. Consider this example of a machined part: the cost of casting is a hyperbolic function of the tolerance. As tolerance approaches zero, cost approaches infinity. A small decrease in tolerance (or, increase in precision) can mean a very large increase in machining cost.
  • Programming cost can be affected by precision. There are defaults in C++ and other languages for the amount of storage given types of numbers have designated. That storage cost could be worth saving, especially in real-time programming.

Example

Here's an eigenvalue problem solved using Matlab:

» format long
» a
  a =

         -300           1           0
       -30000           0          -1
      1000000           0           0

» eig(a)
  ans =

   1.0e+002 *

  -1.00000480119569 + 0.00000831603109i
  -1.00000480119569 - 0.00000831603109i
  -0.99999039760862

Although Matlab is very precise (especially using "long" format), solving a problem like this with that sort of precision is something like solving "1+1" with your pocket calculator and accepting 1.99999 as more correct than 2!

In this case the eigenvalues are the solution of the equation

1000000 + 30000 x + 300 x^2 + x^3 = 0
or (x+100)^3 = 0

and they are EXACTLY 100, all three of them -- no ambiguity, no need for precision!

References

DeGarmo, E. P. et. al. Materials and Processes in Manufacturing. Prentice-Hall, 1997. ISBN 0-02328-621-0
Prata, S. C++ Primer Plus. Waite Group, 1998. ISBN 1-57169-162-6


What You Can Do

  1. Don't copy a calculator result, digit-for-digit.
  2. Prescribe output formats for spreadsheets or executable programs.
  3. Beware of direct comparisons of very large and very small numbers. Do the numbers you're comparing come from the same context?
  4. Present numbers as you wish them to be interpreted. If you are not interested in many decimal places, don't make it seem to your audience that you are.
  5. In programming, consider the possibility of assigning only the bits you know you'll need to numbers that would otherwise tend to clumsy rounding or truncation.

[Table of Contents] [Previous] [Next]