xsl-list
[Top] [All Lists]

RE: single precision bug

2004-08-17 06:20:38

A better inspection learns me that the XSLT processor makes a 
mistake here
totalDebit - sum(./debitAmount) gives me a value of:

-0.00000000005820766091346741 whereas this should be 0.

MsXML, Saxon,Xalan, XT all have this error only Sablotron returns the
correct value.

What do you all think of this?


Since the XSLT/XPath specification is very precise about how numeric
arithmetic should work, it's more likely that MSXML, Saxon, Xalan, and xt
are getting it right and Sablotron is getting it wrong.

Generally, floating point arithmetic is likely to involve rounding errors
when converting between decimal and binary representations. Ideally you
would do this arithmetic in decimal (you can do this in 2.0). Alternatively,
try converting the amounts to pence/cents by multiplying by 100 and
rounding, before doing the arithmetic comparison.

Michael Kay



<Prev in Thread] Current Thread [Next in Thread>