Time to transform 3122 dita files with WebHelp

[ann.jensen]

Hi,
I am transforming my DITA project which contains 3122 DITA files within Oxygen XML Author using the WebHelp transformation engine.
As my file count has increased the time to transform seems to be getting much longer. The last transform I ran took 39 minutes which seems excessively long.
The build command being run is

Code: Select all

Executing:

"c:\program files (x86)\oxygen xml author 18\jre/bin/java" -Xmx1024m "-Doxygen.org.apache.xerces.xni.parser.XMLParserConfiguration=org.ditang.relaxng.defaults.RelaxDefaultsParserConfiguration" -classpath "C:\Program Files (x86)\Oxygen XML Author 18\tools\ant/lib/ant-launcher.jar" "-Dant.home=C:\Program Files (x86)\Oxygen XML Author 18\tools\ant" org.apache.tools.ant.launch.Launcher -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\lib\commons-io-2.4.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\classes" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\oxygen-annotations.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\oxygen-css-pretty-printer.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\oxygen-css-validator.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\oxygen-text-search.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\oxygen-token-markers.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\oxygenAuthor.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\resolver.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\oxygen-token-markers.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\org.eclipse.wst.xml.xpath2.processor_1.2.0.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\xml-apis.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\xercesImpl.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\commons-io-1.3.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\commons-logging-1.2.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\log4j.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\lib\commons-codec-1.9.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\jing.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\saxon9ee.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\saxon.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\xmlgraphics-commons-2.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\fop.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\batik-all-1.8.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\js.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\poi-3.10-FINAL-20140208.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\nekohtml.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\xml-apis-ext.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\avalon-framework-api-4.3.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\avalon-framework-impl-4.3.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\jeuclid-core.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\jeuclid-fop.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\jai_tiff.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\jh.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\lib\jsearch.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\lucene-analyzers-common-4.0.0.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\lucene-core-4.0.0.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\lucene-queries-4.0.0.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\lucene-queryparser-4.0.0.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\lib\lucene-suggest-4.0.0.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.oxygenxml.webhelp\lib\xhtml-indexer.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.oxygenxml.highlight\lib\xslthl-2.1.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\lib\dost-patches.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\lib" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\lib\dost.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\asm-4.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\asm-analysis-4.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\asm-tree-4.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\asm-util-4.1.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\markdown-1.1.0.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\parboiled-core-1.1.6.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\parboiled-java-1.1.6.jar" -lib "C:\Program Files (x86)\Oxygen XML Author 18\frameworks\dita\DITA-OT2.x\plugins\com.elovirta.dita.markdown\pegdown-1.4.2.jar" -f "C:\Program Files (x86)\Oxygen XML Author 18\frameworks/dita/DITA-OT2.x/build.xml" "-Dtranstype=webhelp-responsive" "-Dbasedir=C:\DevBranches\DocService_devel\AGS_Documentation" "-Doutput.dir=C:\DevBranches\DocService_devel\AGS_Documentation/out/webhelp-responsive" "-Dwebhelp.responsive.template.name=bootstrap" "-Dwebhelp.responsive.variant.name=tree" "-Dwebhelp.responsive.skin.name=ags" "-Ddita.temp.dir=C:\DevBranches\DocService_devel\AGS_Documentation/temp/webhelp-responsive" "-Dargs.filter=C:\DevBranches\DocService_devel\AGS_Documentation\ditavals\m_ags_filter.ditaval" "-Ddita.input.valfile=C:\DevBranches\DocService_devel\AGS_Documentation\ditavals\m_ags_filter.ditaval" "-Dclean.output=yes" "-Dargs.input=C:\DevBranches\DocService_devel\AGS_Documentation\supermap_AGS.ditamap" "-Ddita.dir=C:\Program Files (x86)\Oxygen XML Author 18\frameworks/dita/DITA-OT2.x" "-Dargs.breadcrumbs=yes" "-Dwebhelp.logo.image=C:\DevBranches\DocService_devel\AGS_Documentation/Aspen_Grove_Solutions.png" "-Dwebhelp.top.menu.depth=3" "-Dwebhelp.show.side.toc=no" "-Dwebhelp.show.top.menu=yes" "-DbaseJVMArgLine=-Xmx1024m"

Any advice appreciated,
Regards,
Ann

[Radu]

Hi Ann,

We already have an issue logged on our side to profile and see if we can reduce the publishing time.
For example the commercial Saxon XSLT library which comes with Oxygen (Saxon 9.6) seems to take more than the old Saxon 9.1 distributed with the DITA OT in order to process an XSLT transformation.
So about 20% of the processing time could be explained by the fact that this newer Saxon 9.6 library is used by default to produce the output.

But 39 minutes is still too much.
For example our user's manual has about 1500-200 DITA topics and published to webhelp responsive from Oxygen with a DITAVAL filter applied to it, it takes 5 minutes. So this does not scale right.
Is the computer on which you are publishing a very fast one, do you have an SSD hard drive?
One possibility could be that at some point the publishing process may attempt to connect to a remote HTTP location. If you disable the network card and publish, does the publishing break at some point?
Do you use lots of keyrefs and conkeyrefs or mostly use hrefs to refer to topics in the map and to create references between topics?
If possible (if you have enough internal RAM memory) you should also try to edit the transformation scenario and increase the Xmx memory that you have set for Oxygen (right now it is 1024 megabytes) so some higher value like 2000 MBs. It might be possible that the publishing process may encounter a lower memory limitation and stall in order to try to retrieve enough memory to continue.

Regards,
Radu

[ann.jensen]

Hi Radu,
Thanks for your feedback. About 5 minutes was the standard transformation time up until a few weeks ago. I know that the file count is growing but the rate of slow-down of the transformation doesn't correlate.
I do use the same conrefs a lot throughout the files (for system names used regularly). Would this have a massive impact on transformation time?
I can only raise the RAM to 1400MB for some reason.
Regards,
Ann

[Radu]

Hi Ann,

Than indeed maybe some change in the DITA content possibly triggered this problem. I need more data to work with. Would it be possible for you to share with us the DITA project (support@oxygenxml.com)? Or at least send us the entire ANT console output received from the transformation.

Regards,
Radu