Hi David,
I'm comparing data loaded into a system with the values extracted from that
system. Not all the numbers are the same precision, but it would seem that the
system is rounding slightly less than XSL treats as double precision.
I'm already casting the two values to xs:double to carry out the comparison,
but that is still regarding the values as different. The comparision I'd like
to make ideally is "Is the value A the same as B if A is rounded to the same
precision that B is stated.". I've considered comparing as float, but I think
that may not really have sufficient precision.
I also note that the xpath function "round" doesn't have any precision
associated with it, so I can't just count the decimal places of the shorter
value and round the longer to that precision.
Yours,
Clint.
-----Original message-----
To: xsl-list(_at_)lists(_dot_)mulberrytech(_dot_)com;
CC: Clint Redwood <clint(_at_)screwtape(_dot_)co(_dot_)uk>;
From: David Carlisle <davidc(_at_)nag(_dot_)co(_dot_)uk>
Sent: Thu 09-02-2012 09:40
Subject: Re: [xsl] Comparing numbers with different precisions
On 09/02/2012 09:24, Clint Redwood wrote:
Hi!
I've a slightly awkward comparison, where I have two numbers of
differing precisions, and I want to regard them as the same if the
less preicse is the more precise rounded down to the same precision.
Examples are: 17.166666666666668 ~= 17.1666666666667 and
8.333333333333334 ~= 8.33333333333333
Unfortunately, the precision is not always the same for all the
numbers I'm trying to compare.
I just wondered if anyone on this list had done anything similar, and
had a good way of doing it, while I try and figure out a suitable way
to do it.
Yours,
Clint Redwood.
How big are your precisions? in particular do you want xpath to treat
then as doubles (which will mean that anything after 16 or so places
will be lost anyway) and you can just use double precision numeric
functions such as round to deal with the rounding, or do you need
arbitrary precision decimals or (equivalently srings0 and expect to do
teh rounding "by hand".
David
________________________________________________________________________
The Numerical Algorithms Group Ltd is a company registered in England
and Wales with company number 1249803. The registered office is:
Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.
This e-mail has been scanned for all viruses by Star. The service is
powered by MessageLabs.
________________________________________________________________________
--~------------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
To unsubscribe, go to: http://lists.mulberrytech.com/xsl-list/
or e-mail:
<mailto:xsl-list-unsubscribe(_at_)lists(_dot_)mulberrytech(_dot_)com>
--~--
--~------------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
To unsubscribe, go to: http://lists.mulberrytech.com/xsl-list/
or e-mail: <mailto:xsl-list-unsubscribe(_at_)lists(_dot_)mulberrytech(_dot_)com>
--~--