xsl-list
[Top] [All Lists]

Re: Floating point error

2006-01-25 10:42:34
  <product>
    <quantity>3000</quantity>
    <price>6.4</price>
  </product>
 (the actual quantity and price may vary, but they're always
 valid numbers).

6.4 can't be stored exactly as a binary floating point number (for the
same reason that you can't store 1/3 as a finite-precision decimal) so
in principle there is always the potential for rounding error, but given
that you are just apparently looking for 4 or 5 significant figures of
accuracy I'd have expected you to get the exact answer here. (I get
19200 with saxon) I assume that you are using xalan and that xalan just
hands this over to the underlying JVM to do double arithmetic.

xslt2 (eg saxon8) offers decimal arithmetic which can then store 6.4
exactly.

not sure what to recommend as a workaround, you could try multiplying
all your prices by 100 (and dividing by 100 at the end) so that it all
looks like integer arithmetic which then in principle could be exact
(although XSLT1 always uses floating point arithmetic, even for
integers)

David



________________________________________________________________________
This e-mail has been scanned for all viruses by Star. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:
http://www.star.net.uk
________________________________________________________________________

--~------------------------------------------------------------------
XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list
To unsubscribe, go to: http://lists.mulberrytech.com/xsl-list/
or e-mail: <mailto:xsl-list-unsubscribe(_at_)lists(_dot_)mulberrytech(_dot_)com>
--~--



<Prev in Thread] Current Thread [Next in Thread>