HP-UX Floating-Point Guide

Chapter 3 87
Factors that Affect the Results of Floating-Point Computations
Floating-Point Coding Practices that Affect Application Results
Taking the Difference of Similar Values
Calculations can lose precision when a program attempts to take the
difference between two values that are similar in magnitude and also
have some degree of inaccuracy to begin with. If the operands have M
bits of insignificance, and the fractions are the same for the first N
significant bits, then the difference between these two values will have
M+N bits of insignificance because of the cancellation of significant bits
during the subtraction.
For example, suppose that a single-precision application performs a
computation using two different algorithms and then takes the
difference between them to check the similarity of the results. Assume
that the true result should be 1.0 and that both actual results have up to
four bits of insignificance. If the actual results are
0x3f80001f
0x3f80003e
the magnitude of the difference is
0x36780000
However, the two values were the same for the first eighteen bits of their
fractions, which were canceled during the subtraction. This leaves six
bits in the difference, four of which are insignificant. So only two bits are
significant, and the remaining 22 bits are insignificant. (Although the
fraction field has only 23 bits, we include the fraction implicit bit in our
count.)
Figure 3-1 shows how this problem commonly manifests itself in
practice. Suppose you have two very large values:
A = 1.350107523E50
B = 1.350106321E50
If you subtract B from A, the result is
C = 0.0000001202E50
which is normalized to 1.202E44. Suppose there are already 4 digits of
insignificance in A and B. Normalizing the result from 0.0000001202E50
to 1.202E44 adds another 6 digits of insignificance. So the result has 10
digits of insignificance in all, though the operands had only 4.