[XSL-LIST Mailing List Archive Home]
[By Thread]
[By Date]
Hi
I've had what look like memory problems using xsltproc with the document function.
I've tried many variations (e.g. storing the document in a variable etc.) but all the symptoms seem to point to xsltproc keeping all loaded docuements in memory and not releasing memory when they go out of scope.
I'm not sure from Jeni's comment whether XSLT processors are supposed to hold all files in memory - or whether she is just emphasizing that they aren't on disc,
The details below are from a query I made on the Perl XML lists.
I was expecting the memory consumed by the document stored in variable 'tdoc' to be released after each 'item' had been processed
but I could watch the memory on my machine climb towards its limit of 512mB - and then totally lock up!
I had a simple XSLT script:
1 <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
14 <xsl:output method="xml" encoding="UTF-8" omit-xml-declaration='yes' indent='yes'/>
15
16 <!-- ============================================================ -->
17 <xsl:template match='/LIST'>
18 <LIST>
19 <xsl:message>Start of List</xsl:message>
20 <xsl:apply-templates select='item'/>
21 </LIST>
22 </xsl:template>
23 <!-- ============================================================ -->
24 <xsl:template match='item'>
25
26
27 <xsl:variable name='tdoc' select='document(@href)'/>
28 <!--
29 -->
30 <ok>item <xsl:value-of select='position()'/></ok>
31
32
33 </xsl:template>
34 <!-- ============================================================ -->
35 </xsl:stylesheet>
~
~
which reads a simple XML file:
1 <LIST>
2 <item href="out/tm_1283901457.xml"/>
3 <item href="out/tm_2822501119.xml"/>
4 <item href="out/tm_2822601119.xml"/>
5 <item href="out/tm_2822701119.xml"/>
6 <item href="out/tm_2822801119.xml"/>
7 <item href="out/tm_2822901119.xml"/>
...
Each 'out/tm_...' document is about 10K in size.
If I run this stylesheet against the XML document :
In xsltproc
(and possibly xalan - haven't run many tests and have been trimming down the XSLT to try and find the minimum to produce the error)
it crashes my server - or just seems to hang forever.
In saxon, sablotron, msxsl it works as expected.
(Same behaviour under Windows XP and RedHat Linux 7.3.)
It works fine in all processors if the 'tdoc' variable definition is commented out.
Windows versions:
ppm> query XML
Querying target 1 (ActivePerl 5.8.3.809)
could not find ParserDetails.ini in D:/Perl/site/lib/XML/SAX
1. XML-LibXML [1.58] Interface to Gnome libxml2 xml parsing and DOM library
2. XML-LibXML-Common [0.13] Routines and Constants common for XML::LibXML and XML::GDOME
3. XML-LibXSLT [1.57] Interface to Gnome libxslt library
4. XML-NamespaceSupport [1.08] a simple generic namespace support class
5. XML-Parser [2.34] A Perl module for parsing XML documents
6. XML-SAX [0.12] Simple API for XML
7. XML-Simple [2.09] Easy API to read/write XML (esp config files)
Jeni Tennison wrote:
/
Re: [xsl] slow xsltproc XInclude processing w/complex document?
Subject: Re: [xsl] slow xsltproc XInclude processing w/complex document? From: Mike Trotman <mike.trotman@xxxxxxxxxxxxx> Date: Tue, 06 Jul 2004 23:42:37 +0100 |
Hi
I've had what look like memory problems using xsltproc with the document function.
I've tried many variations (e.g. storing the document in a variable etc.) but all the symptoms seem to point to xsltproc keeping all loaded docuements in memory and not releasing memory when they go out of scope.
I'm not sure from Jeni's comment whether XSLT processors are supposed to hold all files in memory - or whether she is just emphasizing that they aren't on disc,
The details below are from a query I made on the Perl XML lists.
I was expecting the memory consumed by the document stored in variable 'tdoc' to be released after each 'item' had been processed
but I could watch the memory on my machine climb towards its limit of 512mB - and then totally lock up!
I had a simple XSLT script:
1 <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
14 <xsl:output method="xml" encoding="UTF-8" omit-xml-declaration='yes' indent='yes'/>
15
16 <!-- ============================================================ -->
17 <xsl:template match='/LIST'>
18 <LIST>
19 <xsl:message>Start of List</xsl:message>
20 <xsl:apply-templates select='item'/>
21 </LIST>
22 </xsl:template>
23 <!-- ============================================================ -->
24 <xsl:template match='item'>
25
26
27 <xsl:variable name='tdoc' select='document(@href)'/>
28 <!--
29 -->
30 <ok>item <xsl:value-of select='position()'/></ok>
31
32
33 </xsl:template>
34 <!-- ============================================================ -->
35 </xsl:stylesheet>
~
~
which reads a simple XML file:
1 <LIST>
2 <item href="out/tm_1283901457.xml"/>
3 <item href="out/tm_2822501119.xml"/>
4 <item href="out/tm_2822601119.xml"/>
5 <item href="out/tm_2822701119.xml"/>
6 <item href="out/tm_2822801119.xml"/>
7 <item href="out/tm_2822901119.xml"/>
...
5854 <item href="out/tm_3407901268.xml"/> 5855 <item href="out/tm_3408001268.xml"/> 5856 </LIST>
Each 'out/tm_...' document is about 10K in size.
If I run this stylesheet against the XML document :
In xsltproc
(and possibly xalan - haven't run many tests and have been trimming down the XSLT to try and find the minimum to produce the error)
it crashes my server - or just seems to hang forever.
In saxon, sablotron, msxsl it works as expected.
(Same behaviour under Windows XP and RedHat Linux 7.3.)
It works fine in all processors if the 'tdoc' variable definition is commented out.
Windows versions:
U:\jobs\ISIS\DataTranslator>xsltproc -V Using libxml 20610, libxslt 10107 and libexslt 805 xsltproc was compiled against libxml 20610, libxslt libxslt 10107 was compiled against libxml 20610 libexslt 805 was compiled against libxml 20610
ppm> query XML
Querying target 1 (ActivePerl 5.8.3.809)
could not find ParserDetails.ini in D:/Perl/site/lib/XML/SAX
1. XML-LibXML [1.58] Interface to Gnome libxml2 xml parsing and DOM library
2. XML-LibXML-Common [0.13] Routines and Constants common for XML::LibXML and XML::GDOME
3. XML-LibXSLT [1.57] Interface to Gnome libxslt library
4. XML-NamespaceSupport [1.08] a simple generic namespace support class
5. XML-Parser [2.34] A Perl module for parsing XML documents
6. XML-SAX [0.12] Simple API for XML
7. XML-Simple [2.09] Easy API to read/write XML (esp config files)
Linux Versions [miket@dl02 DataTranslator]$ xsltproc -V Using libxml 20507, libxslt 10024 and libexslt 715 xsltproc was compiled against libxml 20501, libxslt 10024 and libexslt 715 libxslt 10024 was compiled against libxml 20501 libexslt 715 was compiled against libxml 20501 [miket@dl02 DataTranslator]$
cpan> m XML::LibXML Module id = XML::LibXML DESCRIPTION Interface to the libxml library CPAN_USERID PHISH (Christian Glahn <christian.glahn@xxxxxxxxxx>) CPAN_VERSION 1.58 CPAN_FILE P/PH/PHISH/XML-LibXML-1.58.tar.gz DSLI_STATUS RmhO (released,mailing-list,hybrid,object-oriented) MANPAGE XML::LibXML - Perl Binding for libxml2 INST_FILE /usr/lib/perl5/site_perl/5.6.1/i386-linux/XML/LibXML.pm INST_VERSION 1.58
cpan> m XML::LibXSLT Module id = XML::LibXSLT CPAN_USERID MSERGEANT (Matt Sergeant <matt@xxxxxxxxxxxx>) CPAN_VERSION 1.57 CPAN_FILE M/MS/MSERGEANT/XML-LibXSLT-1.57.tar.gz MANPAGE XML::LibXSLT - Interface to the gnome libxslt library INST_FILE /usr/lib/perl5/site_perl/5.6.1/i386-linux/XML/LibXSLT.pm INST_VERSION 1.57
Jeni Tennison wrote:
Hi Paul,
I've been running some tests on a document that includes nested
Xinclude directives. The document is complex: upwards of 1500 files,
nested to a depth of up to 4 levels. Total size of content is about
4.8MB.
As you probably know, XSLT holds the files that it loads in memory. You haven't said how much memory your machines have, but it might simply be that holding all those files (and the flattened result document) in memory is causing problems.
Another thing is that you're traversing every single node in those documents. Every node visit takes time, because the processor has to work out what to do with the node, so cutting down the node visits would be good. You could try, for example, changing the identity template that you're using at the moment for:
<xsl:template match="node()"> <xsl:copy-of select="." /> </xsl:template>
<xsl:template match="*[.//xi:include]"> <xsl:copy> <xsl:copy-of select="@*" /> <xsl:apply-templates /> </xsl:copy> </xsl:template>
Overall, though, if speed is an issue, you would be much better off using a SAX Filter to do the transformation: that way you wouldn't be storing the documents in memory, each node would have to be visited only once, and the output can stream out (if that helps). Mind you, you say that the XInclude resolution is only a test, so perhaps you can't do your real transformation using SAX...
Cheers,
Jeni
--- Jeni Tennison http://www.jenitennison.com/
--+------------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
To unsubscribe, go to: http://lists.mulberrytech.com/xsl-list/
or e-mail: <mailto:xsl-list-unsubscribe@xxxxxxxxxxxxxxxxxxxxxx>
--+--
-- Datalucid Limited 8 Eileen Road South Norwood London SE25 5EJ United Kingdom
/ tel :0208-239-6810 mob: 0794-725-9760 email: mike.trotman@xxxxxxxxxxxxx
UK Co. Reg: 4383635 VAT Reg.: 798 7531 60
/
Current Thread |
---|
|
<- Previous | Index | Next -> |
---|---|---|
Re: [xsl] slow xsltproc XInclude pr, Jeni Tennison | Thread | Re: [xsl] slow xsltproc XInclude pr, Paul DuBois |
seeking clarification of CSS2 spec, fo | Date | Re: [xsl] seeking clarification of , M. David Peterson |
Month |