KnowledgeBase Archive

An Archive of Early Microsoft KnowledgeBase Articles

View on GitHub

Q61436: Problem of Testing Floating-Point Equality, IF n=VAL("n")

Article: Q61436
Product(s): See article
Version(s): 4.00 4.00b 4.50
Operating System(s): MS-DOS
Keyword(s): ENDUSER | SR# S900416-32 B_BasicCom | mspl13_basic
Last Modified: 14-FEB-1991

A floating-point constant passed in a string to the VAL function can
return a slightly different floating-point result compared to the same
floating-point constant in an IF statement, resulting in an apparent
inequality. This floating-point difference may seem like a software
problem, but it is actually a design limitation. This behavior is
demonstrated in the program below.

To reliably test for floating-point equality (in any binary
floating-point format, such as IEEE or Microsoft Binary Format), you
must subtract the two floating-point numbers being compared and test
whether their difference is less than a value at the limits of
significance for single or double precision.

The example below applies to Microsoft QuickBASIC versions 4.00,
4.00b, and 4.50 for MS-DOS; to Microsoft BASIC Compiler versions 6.00
and 6.00b for MS-DOS and MS OS/2; and to Microsoft BASIC Professional
Development System (PDS) versions 7.00 and 7.10 for MS-DOS and MS
OS/2.

You might expect the conditions in the IF statements in the program
below to both be true and PRINT their messages. Instead, only the
second IF is true and PRINTs its message:

   a$ = "0.1"
   IF .1 <= VAL(a$) THEN PRINT "This is false, and does not print."
   s = VAL(a$)   ' This time use a temporary variable s.
   IF .1 <= s THEN PRINT "This is true and does print."

There is an inequality in the first IF statement because the numeric
constant .1 in the IF is assigned at compile time, whereas the VAL(A$)
function is calculated at run time and different numeric storage is
allocated internally so that a tiny difference exists between the
numbers at the limits of single precision. In the second IF statement,
the comparison of .1 and "s" is compiled differently than in the first
IF, and the comparison luckily expected a "true" result. However, you
should avoid both of the above numeric comparison techniques for
testing the equality of floating-point numbers.

Instead, to reliably test for floating-point equality (in any binary
floating-point format), you must subtract the two floating-point
numbers being compared and test whether their difference is less than
a value that is about 7 significant digits smaller than the value
being compared for single precision (in other words, divide by 10^7 to
find the comparison value), or about 15 significant digits smaller
than the value being compared for double precision (divide by 10^15).
For example:

A$ = ".1"
IF .1 - VAL(A$) <= .1 / 10^7 THEN PRINT "Equal within single precision"

B$ = ".1#"
IF .1# - VAL(B$) <= .1# / 10^15 THEN PRINT "Equal within double precision"

References:

Note that many numbers in decimal (base 10) notation do not have an
exact representation in the binary (base 2) floating-point storage
format used in BASIC's SINGLE and DOUBLE precision data types. This
often causes BASIC to return floating-point results different than you
might expect, as explained in separate articles found with the
following query:

   floating and point and format and QuickBASIC

THE INFORMATION PROVIDED IN THE MICROSOFT KNOWLEDGE BASE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. MICROSOFT DISCLAIMS ALL WARRANTIES, EITHER EXPRESS OR IMPLIED, INCLUDING THE WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL MICROSOFT CORPORATION OR ITS SUPPLIERS BE LIABLE FOR ANY DAMAGES WHATSOEVER INCLUDING DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL, LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF MICROSOFT CORPORATION OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. SOME STATES DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES SO THE FOREGOING LIMITATION MAY NOT APPLY.