[XSL-LIST Mailing List Archive Home] [By Thread] [By Date]

RE: [xsl] Floating point precision error

---

---

I used system-property() to query for vendor, version and vendor-URL
and this is what I get:
Vendor is Apache Software Foundation-
1-
http://xml.apache.org/xalan-j
So I'm using xalan, XSLT 1.0, after all.

Michael, if you don't mind me asking, how do you explicitly
set the precision? It's not something to do with xs:float() and
xs:double() is it? I tried them out but it seems they're not
supported.

Oh and as I mentioned, I only got this on SunOS. Would it have
something to do with the OS? How can I tell if it's something
inherent in that machine (how it processes floating-point in
general) and not XSL processor or Java?

Thanks,

On 26 Jan 2006 at 8:49, Michael Kay wrote:

> I agree, this result is way out. It looks to me suspiciously as if
> it's carried out using single-precision rather than double-precision
> floating point.
> 
> Saxon (both 6.5.5 and 8.6.1) give
> 
> <?xml version="1.0" encoding="utf-8"?>6.4_
>   3000_
>   19200_  
> 
> dom4j doesn't include an XSLT processor of its own, as far as I know.
> Use xsl:system-property('xsl:vendor') to find out which processor you
> are using.
> 
> I changed the code to explicitly use single-precision float and the
> result was
> 
> <?xml version="1.0" encoding="UTF-8"?>6.4_
>   3000_
>   19200.000286102295_  
> 
> which is still closer than the result you are seeing.
> 
> Michael Kay
> http://www.saxonica.com/
> 
> 
> 
> > -----Original Message-----
> > From: Thang Le [mailto:tle@xxxxxxxxxxx] 
> > Sent: 26 January 2006 01:16
> > To: xsl-list@xxxxxxxxxxxxxxxxxxxxxx
> > Subject: [xsl] Floating point precision error
> > 
> > This is something I don't understand. Given this XSL snippet:
> > 
> > <xsl:value-of select="$nodes[1]/unitamount" />_
> > <xsl:value-of select="$nodes[1]/quantity" />_
> > <xsl:value-of select="($nodes[1]/quantity * $nodes[1]/unitamount)"
> > />_
> > 
> > I'm getting the printout as:
> > 6.4_
> > 3000_
> > 19199.908447265625_ 
> > 
> > The literal values in XML is 6.4 and 3000. If I round the result to
> > 2 decimal places, I still get 19199.91, a far cry from what it
> > should be (19200).
> > 
> > I'm using dom4j 1.6, so I guess that's XSLT 1.0? Also this 
> > only happens 
> > on SunOS and not on Windows. Still, the error is huge and obvious.
> > Isn't there a way to force the calculation to be more exact?
> > 
> > Thanks,
> > 
> > Thang Le
> > 
> > -- 
> > NeoMail - http://neomail.sourceforge.net

---