Debugging java.lang.OutOfMemoryError issues

[chrispitude]

I have a DITA-OT project file with about 30 deliverables. When I run the "All Deliverables" transformation, the largest book sometimes crashes as follows:

Code: Select all

     [exec]      [java] INFO  LoggingEventListener - Rendered page #4202.
     [exec]      [java] INFO  LoggingEventListener - Rendered page #4203.
     [exec]      [java] INFO  LoggingEventListener - Rendered page #4204.
     [exec]      [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     [exec]      [java] 	at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
     [exec]      [java] 	at java.base/java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:185)
     [exec]      [java] 	at com.oxygenxml.chemistry.e.eb.b(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.b(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.processInternal(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry$1.b(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry$1.run(Unknown Source)
     [exec]      [java] 	at ro.sync.security.SandboxCore.runWithConfirmation(SandboxCore.java:258)
     [exec]      [java] 	at ro.sync.security.ConnectionsSandbox.runWithConnectConfirmation(ConnectionsSandbox.java:263)
     [exec]      [java] 	at ro.sync.security.Sandbox.runWithConnectConfirmation(Sandbox.java:355)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.main(Unknown Source)
     [exec] Error: The following error occurred while executing this line:
     [exec] C:\Program Files\Oxygen XML Editor 25 - 2022081006\frameworks\dita\DITA-OT3.x\plugins\com.oxygenxml.pdf.css\build.xml:670: Java returned: 1

BUILD FAILED
C:\Users\chrispy\git\dita-ams\prj\frameworks\dita\dita_project\build_dita_project_preprocessed.xml:25: exec returned: 1

Total time: 52 minutes 21 seconds

My Windows 10 laptop has 16GB physical memory. With -Xmx8192m, this PDF deliverable always crashes. With -Xmx10240m, it only sometimes crashes.

Another writer has a Windows 10 laptop with 16GB physical memory, and the transformation always crashes for her even with -Xmx10240m (although we only ran it a few times, so "always" is based on a small dataset).

Part of the difficulty in debugging this is the lack of information. For example,

• Does a deliverable require more memory when published in a set of "All Deliverables" versus published by itself?
• Does a transformation's -X memory limit include what Oxygen itself uses, or do the transformation and Oxygen each have their own limit? (related docs here)
• How do I differentiate between an insufficient -X limit versus the -X limit not getting reached due to other applications consuming memory on the machine?
• Does the -X limit consider available Windows 10 virtual memory, or just physical memory?

Interestingly, the PDF transformation always crashes after the last "Rendered page #..." line for the document (that value is always the total page count of the document), so the crash seems to be after the page rendering step.

As an enhancement idea, perhaps Oxygen could somehow print the highest memory usage for both failed transformations and successful transformations. This would provide visibility into whether the -X limit was reached and thus is insufficient, versus not reached due to unavailable system resources.

I can provide a testcase upon request. Thanks!

[julien_lacour]

Hi Chris,

For analysis, could you provide us the part of the console that calls Chemistry, it starts with:

Code: Select all

     [exec] pdf-css.chemistry:
     [exec] 
     [exec] pdf-css.chemistry.embedded:
     [exec]      [echo] =============================
     [exec]      [echo] Oxygen PDF Chemistry
     [exec]      [echo] Installation directory: ...

Regarding your questions:

• The transformation using DITA project file should be equivalent to the transformation scenario with the Prefer using the "dita" command option unchecked.
• The -Xmx parameter set is only used by the transformation and not by Oxygen itself.
• The OutOfMemory error mean that the -X limit has been reached, the other applications should not interfere here.
• The -X memory is Java virtual memory, the link between it and Windows memory is a little complicated as the operating system itself manage how the memory is used.

I added an issue to enhance the console log, we will add the original available memory and in case of error the total consumed memory. Thank you for the request!

Regards,
Julien

[chrispitude]

Thanks Julien! I appreciate the answers, that should help us intelligently perform further investigation. Also, the plan to add additional memory information to both successful and failing transformations will be very helpful in the future.

Here is the requested portion of the log:

Code: Select all

     [exec] pdf-css.chemistry.embedded:
     [exec]      [echo] =============================
     [exec]      [echo] Oxygen PDF Chemistry
     [exec]      [echo] Installation directory: C:\Program Files\Oxygen XML Editor 25 - 2022081006\frameworks\dita\DITA-OT3.x\bin\../../../..
     [exec]      [echo] Processing: C:\Users\chrispy\git\dita-ams\temp\stage1.xml.pp
     [exec]      [echo] Output file: C:\Users\chrispy\git\dita-ams\out\final\olh_cc\pdf/cc_tclref.pdf
     [exec]      [echo] =============================
     [exec]      [java] INFO  OxygenPDFChemistry - oXygen PDF Chemistry version 25.0 build 2022-08-10T02:52:53Z
     [exec]      [java] INFO  OxygenPDFChemistry - Using catalogs: [file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/config/catalogs/catalog.xml, file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/catalog-dita.xml, file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/docbook/catalog.xml]
     [exec]      [java] INFO  x - CSS: file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/plugins/com.oxygenxml.webhelp.responsive/oxygen-webhelp/app/topic-page-print.css
     [exec]      [java] INFO  x - CSS: file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/plugins/com.oxygenxml.pdf.css/css/print/p-dita.css
     [exec]      [java] INFO  x - CSS: file:/C:/Program%20Files/Oxygen%20XML%20Editor%2025%20-%202022081006/frameworks/dita/DITA-OT3.x/plugins/com.oxygenxml.editlink/pdf-css.css
     [exec]      [java] INFO  x - CSS: file:/C:/Users/chrispy/git/dita-ams/dita/_common/_template/../../../prj/frameworks/dita/css/synopsys.css
     [exec]      [java] INFO  OxygenPDFChemistry - Dumping styled content to: C:\Users\chrispy\git\dita-ams\out\final\olh_cc\pdf\cc_tclref.style.xml
     [exec]      [java] INFO  OxygenPDFChemistry - Dumping FO content to: C:\Users\chrispy\git\dita-ams\out\final\olh_cc\pdf\cc_tclref.fo
     [exec]      [java] INFO  LoggingEventListener - Rendered page #1.
     [exec]      [java] INFO  LoggingEventListener - Rendered page #2.
...omitted...
     [exec]      [java] INFO  LoggingEventListener - Rendered page #4203.
     [exec]      [java] INFO  LoggingEventListener - Rendered page #4204.
     [exec]      [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     [exec]      [java] 	at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
     [exec]      [java] 	at java.base/java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:185)
     [exec]      [java] 	at com.oxygenxml.chemistry.e.eb.b(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.b(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.processInternal(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry$1.b(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry$1.run(Unknown Source)
     [exec]      [java] 	at ro.sync.security.SandboxCore.runWithConfirmation(SandboxCore.java:258)
     [exec]      [java] 	at ro.sync.security.ConnectionsSandbox.runWithConnectConfirmation(ConnectionsSandbox.java:263)
     [exec]      [java] 	at ro.sync.security.Sandbox.runWithConnectConfirmation(Sandbox.java:355)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.process(Unknown Source)
     [exec]      [java] 	at com.oxygenxml.chemistry.OxygenPDFChemistry.main(Unknown Source)
     [exec] Error: The following error occurred while executing this line:
     [exec] C:\Program Files\Oxygen XML Editor 25 - 2022081006\frameworks\dita\DITA-OT3.x\plugins\com.oxygenxml.pdf.css\build.xml:670: Java returned: 1

BUILD FAILED
C:\Users\chrispy\git\dita-ams\prj\frameworks\dita\dita_project\build_dita_project_preprocessed.xml:25: exec returned: 1

[julien_lacour]

Hi Chris,

Starting with Oxygen 25.0 you should have more information about memory consumption in the DITA Map PDF - based on HTML5 & CSS scenarios.

Regards,
Julien

[chrispitude]

Hi Julien,

Thanks very much for this fix! This will be very helpful the next time I am debugging publishing issues for writers working with some of our larger reference books.

[chrispitude]

Hi Julien,

When I publish in Linux with -Xmx20G set in our global Java options file, I see the following in the log:

Code: Select all

[java] INFO  OxygenPDFChemistry - Max memory available: 18204Mb

which is great! When I publish in Oxygen in Windows using framework-defined transformations (with the memory option set in Advanced > JVM Arguments tab):

image.png

then the log always shows the following:

Code: Select all

[java] INFO  OxygenPDFChemistry - Max memory available: 4034Mb

Here is a testcase:

oxygen_chemistry_memory.zip

I hope I am not doing something obviously wrong...

[andrei_pomacu]

Hi Chris,

In Windows, in the JVM Arguments tab you need to use -DbaseJVMArgLine=-Xmx{memory}.
So instead of a simple -Xmx8g you need to use -DbaseJVMArgLine=-Xmx8g or -DbaseJVMArgLine=-Xmx8192m

Regards,
Andrei

[chrispitude]

Hi Andrei,

I confirmed that this works - thank you! This new memory reporting feature is already providing value.

There are many mentions of -Xmx in the Oxygen documentation:

# document(s) found for -Xmx

but no mention of setting it via -DbaseJVMArgLine=. I don't know if it's worth internally modifying a specification when running in Windows versus updating all the documentation instances to mention the Windows requirement.

[julien_lacour]

Hi Chris,

Actually this indication is present at the end of the Increasing Memory Allocation for Java topic:

If the Chemistry PDF CSS processor fails with an Out Of Memory Error, try adding the baseJVMArgLine parameter to the DITA-OT command line. For example:

Code: Select all

-DbaseJVMArgLine=-Xmx984m

But still I will add a Documentation Task to improve these topics and their display.

Regards,
Julien

[chrispitude]

Hi Julien,

Now I am unsure exactly what -Xmx and -DbaseJVMArgLine=-Xmx each control:

• Does -DbaseJVMArgLine=-Xmx require that -Xmx also be specified? (Is it configuring the memory of a launched subprocess?)
• Is -DbaseJVMArgLine=-Xmx specific to PDF Chemistry, or does it apply to other transtypes too?
• Is -DbaseJVMArgLine=-Xmx specific to transformations run from the Oxygen GUI, or does it also apply to batch Publishing Engine jobs?
• Is -DbaseJVMArgLine=-Xmx specific to Windows, or does it apply to linux too?

I think all the questions would be answered by a description of what's happening inside the transformation.

[julien_lacour]

Hi Chris,

• Does -DbaseJVMArgLine=-Xmx require that -Xmx also be specified? (Is it configuring the memory of a launched subprocess?)
  The -DbaseJVMArgLine parameter can be set alone, without any additional -Xmx (except the one appearing after =).
• Is -DbaseJVMArgLine=-Xmx specific to PDF Chemistry, or does it apply to other transtypes too?
  It is specific to Chemistry or any other transtype running a Java sub-process (I don't think there are other other Java sub-processes by default but you can have your own).
• Is -DbaseJVMArgLine=-Xmx specific to transformations run from the Oxygen GUI, or does it also apply to batch Publishing Engine jobs?
  You can either set the -DbaseJVMArgLine in Oxygen or as a parameter in the dita command.
• Is -DbaseJVMArgLine=-Xmx specific to Windows, or does it apply to linux too?
  The -DbaseJVMArgLine works fine for both Windows and Linux, for example:

  Code: Select all

  ./dita -v -f pdf-css-html5 -DbaseJVMArgLine=-Xmx2048m -i ~/OxygenXMLEditor/samples/dita/flowers/flowers.ditamap -o ~/OxygenXMLEditor/samples/dita/flowers/out/pdf-css-html5
  

Regards,
Julien

[chrispitude]

Hi Julien,

This is good information! So the PDF Chemistry transformation internally calls a Java subprocess, and this Java subprocess sometimes needs additional memory provided.

Can you confirm that -Xmx must still provide enough memory for regular DITA-OT preprocessing up through merged map creation?

I think I will need to set both -Xmx and -DbaseJVMArgLine=-Xmx based on experimentation with our largest books, so that neither limit is set too low or unnecessarily high. Hopefully each limit's failure has its own distinctive message somehow. I can try to figure that out in the experimentation too.

[julien_lacour]

Hi Chris,

In fact, when transforming DITA to PDF, Oxygen PDF Chemistry is the Java sub-process called by our com.oxygenxml.pdf.css plugin (when reaching the "pdf-css.chemistry" target in the console, to be precise).

I confirm that setting -Xmx may be needed by the regular DITA-OT processing (through the com.oxygenxml.pdf.css plugin) which creates the merged.xml and merged.html files.

You may need to set both for your largest books depending on where the process fails: in the merging steps or after calling Oxygen PDF Chemistry.

Regards,
Julien

[chrispitude]

Hi Julien,

In addition to reporting available memory at the beginning, is it also possible to report peak memory usage at the end? This would be faster (and more reliable!) than deriving values using trial-and-error with very large documents!

I cannot seem to pass this parameter to PDF Chemistry deliverables published from a DITA-OT project. In the following testcase:

oxygen_chemistry_memory_v2.zip

I define the following transformations in a DITA-OT Project framework extension:

• Publish DITA-OT Project (all deliverables) - 6GB, 6GB
• Publish DITA-OT Project (select deliverable) - 6GB, 6GB

These transformations set the following JVM arguments:

Code: Select all

-Xmx6g -DbaseJVMArgLine=-Xmx6g

However, building a PDF deliverable from the project.xml file using this "6GB, 6GB" transformation shows the following:

Code: Select all

     [exec]      [java] INFO  OxygenPDFChemistry - oXygen PDF Chemistry version 25.0 build 2022-09-29T13:05:39Z
     [exec]      [java] INFO  OxygenPDFChemistry - Max memory available: 4034Mb

Is there a special trick for controlling the memory when using DITA-OT project file transformations?

[julien_lacour]

Hi Chris,

The thing is that when running DITA transformation through DITA-OT Project file under the hood a Java process is called (and not the dita command). This is why the Java usage may differ, for this use-cases the dita command is a better choice (maybe through a bat or sh script).

Regarding the Java peak usage, even this can be tricky as we cannot really control when Java Garbage Collector will run and free memory.

Regards,
Julien

[chrispitude]

Hi Julien,

Does this mean that there is no way to control the preprocessing memory limit or PDF Chemistry subprocess memory limit when running DITA transformations from a DITA-OT project file transformation?

image.png

For a DITA-OT project file transformation, if the DITA-OT transformation is invoked internally, does that mean -DbaseJVMArgLine=-Xmx now controls the preprocessing memory limit (and then the PDF Chemistry subprocess always uses the default)?

I agree that running transformations from a command line provides more flexibility. However,

• Our writers use Oxygen on Windows laptops, and Windows does not provide a sufficient environment for command-line operation.
• We do not have Oxygen Publishing Engine licenses for our writers; they use Oxygen for their local test builds. (We do use Oxygen Publishing Engine in linux for final production publishing.)
• Now that our DITA-OT project files capture all profiling conditions and publication-specific parameters, writers publish more often from DITA-OT project files (instead of from the DITA Maps Manager) for their test builds.

I don't think this is a serious issue for us yet. We only have a couple problematically large books, and we are mindful of them. Hopefully they do not grow too much in the near term. But I would like to understand what -Xmx and -DbaseJVMArgLine=-Xmx do (or don't do!) when specified for a DITA-OT project file transformation.

[julien_lacour]

Hi Chris,

I added your vote to the issue related to JVM Memory when running DITA-OT Project files (EXM-51733).
The scope of this issue will be to transmit "JVM Arguments" from Scenario Options as ANT_OPTS environment variable.
I will notify this thread as soon as it will be resolved.

Until then you will need to follow these steps:

1. Open the OXYGEN_INSTALL_DIR\frameworks\dita\dita_project\build_dita_project.xml file in Oxygen
2. Add the ANT_OPTS declaration like follows:

   Code: Select all

   ...
   <env key="JAVA_HOME" value="${java.home}"/>
   <env key="ANT_OPTS" value="-DbaseJVMArgLine=-Xmx6G"/>
   ...
   

3. Set the -Xmx with the needed value, it will be passed to Oxygen PDF Chemistry.

Regards,
Julien

[Cosmin Duna]

Hi,
As an update, we have released Oxygen 25.1, and a new parameter has been included in the DITA-OT transformation scenarios that allows for setting the JVM memory allocation for publishing deliverables: https://www.oxygenxml.com/doc/versions/ ... l=jvm.args

Best regards,
Cosmin

[chrispitude]

Excellent, thank you Cosmin! I will update our environment.