How long should WebHelp transform take?

[ann.jensen]

Hi,

Having recently increased by ANT_OPTS env variable from 1024 to 2048, my external Oxygen WebHelp transformation started failing again with error

Code: Select all

     [echo] Using directory C:\DITA\dita-ot-2.2.2\plugins\com.oxygenxml.webhelp/lib

Error: The following error occurred while executing this line:

C:\DITA\dita-ot-2.2.2\plugins\org.dita.base\build_preprocess.xml:42: java.lang.OutOfMemoryError: GC overhead limit exceeded

I have 64 bit Java installed on my machine which my JAVA_HOME env variable references.
I have now increased ANT_OPTS to -Xmx3000 for a transform time of 10 mins or -Xmx4000 for a transform time of 5 mins.

We are only 6 months into our content development which will grow each week and I am starting to get concerned about the memory that WebHelp needs to transform our content.
What areas of my DITA can I analyse to see if there is something I can improve on in order to reduce my transform time?
I can share the output of my build again if that helps inform my use of conrefs etc.?

Any advice appreciated,
Regards,
Ann

[Radu]

Hi Ann,

Could you tell me again how many topics you have?
Also have you tried to set the parameter "webhelp.reload.stylesheet" to "true" when publishing? It should drastically reduce memory consumption.

Regards,
Radu

[ann.jensen]

Hi Radu,
This is the current count for the main file types in my project:
- 1767 .dita
- 999 .png
- 154 .ditamap

I have added that property that you suggested and the transform seems to take about the same amount of time with ANT_OPTS set to -Xmx3000. The transform still fails with ANT_OPTS set to -Xmx2000.
This is curious as setting JVM Arguments to -Xmx2000m inside Oxygen XML Author results in a successful transform in 3 minutes They are both using the same JDK but obviously use different libraries.
Any advice appreciated,
Regards,
Ann

[Radu]

Hi Ann,

Indeed DITA OT transformations performed from inside Oxygen use by default the newer Saxon 9 XSLT processor libraries which come with Oxygen. They also use another version of the Xerces XML parser library.
So there are differences in libraries.
It seems the parameter "webhelp.reload.stylesheet" made no difference on your side. Are you using the latest WebHelp plugin (corresponding to Oxygen 18.0)? Because the parameter did not exist in previous WebHelp plugin versions.
We also have an issue on our side to profile the memory and CPU usage while publishing so we'll also have some possible future publishing improvements in memory and speed.

Regards,
Radu

[ann.jensen]

Hi Radu,
Yes I am using the WebHelp-Responsive target that was released with Oxygen 18.
That is great that you are doing profiling of transforms to learn more and It will be great to hear about any tweaks that can be made to reduce memory requirements.
Also, do you have any information on what kind of memory allocation other companies are setting for transforming DITA on a centralised server using Oxygen WebHelp.
Thanks,
Ann

[Radu]

Hi Ann,

I can tell you about our DITA-based Oxygen User's Manual.
It has about 1600 topics, 1400 images and 10 DITA Maps. We use a mixture of conrefs, conkeyrefs, keyrefs and plan hrefs. So we are not consistent in using only indirect addressing, I think we have more direct than indirect addressing used.
To publish to Oxygen Webhelp using an integration server with DITA OT 2.3 + our plugin installed we set to the transformation -Xmx1024m so only one gigabyte is enough for us.

Regards,
Radu

[ann.jensen]

Hi Radu,
I just checked my file count and it currently stands at 1588 .dita files, 970 .png images and 130 .ditamaps.
We are using DITA-OT 2.2.2 and are transforming using dita-ot-2.2.2\bin\dita.bat command and an associated webhelp.properties file
On my own machine I have set %ANT_OPTS% in my environment variables to -Xmx3000m and my transform to webhelp from the command line still takes 12 minutes. If I reduce memory any lower, it will fail with out of memory error. I have plenty of free memory on my machine.
I have to get to the bottom of why my transforms are so slow and require so much memory as the file count is just going to continue increasing and the fear is we will run out of memory.
Do you think the number of ditamaps that I am using could be a problem?
Thanks and regards,
Ann

[Radu]

Hi Ann,

I'm not sure where the problem is located, probably somehow it's related to how your DITA content is structured.
Would it be possible for you to share your entire DITA project with us? You can upload large ZIP files using this form:

https://www.oxygenxml.com/techSupport.html

Otherwise I do not quite know how to build an equivalent test project on my side.

Regards,
Radu

[ann.jensen]

Hi Radu,
Unfortunately it contains all our internal system documentation which I don't think I am in a position to share.
If you think of anything else that might be impacting our memory usage, please let me know.
Regards,
Ann

[ann.jensen]

Because I am using conrefs to render the various system names that I am documenting, there are over 1000 content references pointing to a single file while defines all my reusable system names.
Would 1015 conrefs in a project with 6000 dita files cause memory issues?
Thanks in advance,
Ann

[Radu]

Hi Ann,

I do no know, if you want to test this you can replace part of the conrefs with a fixed string using our Find/Replace in Files tool, then publish.
For our user's manual we use keyrefs to refer to the product name (<ph keyref="productName"/>) with the key being defined in the DITA Map and bound to the product name keyword like:

Code: Select all

  <keydef keys="productName" product="oxygen">

    <topicmeta>

      <keywords>

        <keyword>Oxygen XML Editor</keyword>

      </keywords>

    </topicmeta>

  </keydef>

Regards,
Radu

[ann.jensen]

Hi Radu,
When compared to yours I see that I have used many more ditamaps than you; I have 134 ditamaps and 1619 dita files.
Do you know if having a lot of ditamaps effects memory usage?
Thanks in advance,
Ann

[Radu]

Hi Ann,

I do not have knowledge of such a behavior when there are lots of DITA Maps but I have not tested something similar on my side.

Regards,
Radu

[ann.jensen]

Thanks to Radu, my memory problems when transforming to webhelp outside of Oxygen XML Author have been resolved and it is now transforming with %ANT_OPTS% set to Xmx2048m in under 7 minutes.

The various properties that he suggested seem to have done the job. I have these set in an external .properties file as described in:

Code: Select all

 http://www.dita-ot.org/dev/user-guide/build-using-dita-properties-file.html.

I then run from command line as follows:

Code: Select all

[code]dita.bat -v -i "C:\DevBranches\DocService_devel\AGS_Documentation\supermap_AGS.ditamap" -f "webhelp-responsive" -filter "C:\DevBranches\DocService_devel\AGS_Documentation\ditavals\m_ags_filter.ditaval" -o "C:\DevBranches\DocService_devel\AGS_Documentation\out\webhelp-responsive" -propertyfile "webhelp.properties"

[/code]

The contents of my .properties are shown here:

Code: Select all

webhelp.responsive.template.name=bootstrap

webhelp.responsive.variant.name=tree

webhelp.responsive.skin.name=ags

args.copycss=yes

clean.output=yes

args.breadcrumbs=yes

webhelp.logo.image=C:\DevBranches\DocService_devel\AGS_Documentation\Aspen_Grove_Solutions.png

webhelp.top.menu.depth=3

webhelp.show.side.toc=no

webhelp.show.top.menu=yes

conserve-memory=true 

args.grammar.cache=no

args.css=C:\Dita\dita-ot-2.2.2\plugins\com.oxygenxml.webhelp\templates\dita\bootstrap\variants\tree\ags\skin.css

webhelp.responsive.template.name=bootstrap

webhelp.responsive.variant.name=tree

generate-debug-attributes=false

webhelp.reload.stylesheet=true

webhelp.fragment.head=C:\DevBranches\DocService_devel\AGS_Documentation\googleAnalytics.html

Hope this helps someone else,
Regards,
Ann

[Radu]

Hi Ann,

Thanks for updating the forum thread, I also updated the DITA Users List discussion that you started.

Regards,
Radu