xsl-list
[Top] [All Lists]

Re: [xsl] Comparing numbers with different precisions

2012-02-09 04:20:14
On 09/02/2012 09:21, Clint Redwood wrote:
Hi!

I've a slightly awkward comparison, where I have two numbers of differing 
precisions, and I want to regard them as the same if the less preicse is the 
more precise rounded down to the same precision.

Examples are:
17.166666666666668 ~= 17.1666666666667
and
8.333333333333334 ~= 8.33333333333333

Unfortunately, the precision is not always the same for all the numbers I'm 
trying to compare.

I just wondered if anyone on this list had done anything similar, and had a 
good way of doing it, while I try and figure out a suitable way to do it.

Yours,

Clint Redwood.


Presumably by "precision" you mean the number of decimal digits in the lexical form of the number. That means that trailing zero digits change the result, and since trailing zeroes are ignored for all the system-supplied numeric data types, it means you are outside the range of what the system data types can do.

Drafts of XSD 1.1 included a "precision decimal" data type that was intended to meet this need - but it only defined the lexical and value space of the type, not the arithmetic rules for comparing and calculating with these values (which is why it was dropped from the final spec).

I think in your position I would represent the values as elements of the form <value precision="12">17.16666667</value>, and then write your own function library with operations for comparing and calculating on these values.

Michael Kay
Saxonica

--~------------------------------------------------------------------
XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list
To unsubscribe, go to: http://lists.mulberrytech.com/xsl-list/
or e-mail: <mailto:xsl-list-unsubscribe(_at_)lists(_dot_)mulberrytech(_dot_)com>
--~--