[XSL-LIST Mailing List Archive Home] [By Thread] [By Date]

Re: [xsl] "Heap" of trouble handling input file of 500 MByte

---

---

On Sat, 2011-02-19 at 19:47 +0000, thehulk@xxxxxxxxxxx wrote:
> Hello,
> 
> Thanks mainly to this list, I am successfully processing 6,335 of my
> 6,337 input files. The 6,335 are under 250 MByte each. The two problem
> cases are each just under 500 MByte. 
> 
> Are there any tips or tricks or tools which will make this possible on
> my 32-bit Windows XP SP3 machine? 
> 
> I am using Java code and the Javax.xml.* classes to do the transform.
I would not be surprised to find a DOM implementation using 5 or 10
times as much memory as the document, if it's markup-heavy, and for
500MBytes that could mean 5 gigabytes of memory; if you then do an
in-memory DOM to DOM transform, you'll need memory to store the output.

You could try Saxon 9 with its new experimental streaming mode, but if
you are doing sorting, that my not help.

Or you could try Saxon, msxml or libxml2 and see if they are better.

Or just pick up a 64-bit computer with (say) 10 G of RAM, and install a
64-bit Linux on it :-)

It's hard to say more without seeing the nature of the XML input and the
XSLT you're using.

Liam

-- 
Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/
Pictures from old books: http://fromoldbooks.org/

---