Andrew wrote:
I've following xml:
<aa>
<plan SUMM="700000.1"/>
</aa>
execute sum(//plan/@SUMM) and you'll get 700000.09999...- what the ...?
but if you change 7 on 1 or 2 all works as expected- 1 preserved.
tested on .NET/xmlspy.
Computers usually don't represent floating point numbers the same
way humas do. Internally, a binary representation is used, which
can't excatly represent fractions which aren't a power of 1/2.
Therefore if you convert 0.1 into the internal representation and
back to decimal, you may experience roundoff errors.
The common workaround is to use the format() function to tell
the processor to round to a specific precision.
J.Pietschmann
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list