XSLT resource Problem in 12.1(crashes)

Having trouble installing Oxygen? Got a bug to report? Post it all here.
sderrick
Posts: 264
Joined: Sat Jul 10, 2010 4:03 pm

XSLT resource Problem in 12.1(crashes)

Post by sderrick »

Oxygen 12.1 running on Ubuntu 10.04

I have some xslt scripts that process TEI documents and generate large files. 20+ MB.

Using the saxon executable from the command line it takes about 15 seconds to process. I monitored my system(dual cpu, 3GB ram) while running the scripts and it uses about 300MB ram and the cpu usage ramps up to an average of 30%.

Using Oxygen to execute the script it runs for about 3.5 minutes! Then crashes and complains that 1 GB of memory is not enough. The system ramps up to about 75% cpu usage and Java shows its using 1.1+ GB of ram.

WTF?? Whats wrong with the XSLT processor you guys are using? :shock:
sorin_ristache
Posts: 4141
Joined: Fri Mar 28, 2003 2:12 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sorin_ristache »

Hello,

Thank you for your report. The difference from 300 MB to 1 GB can be explained in part by the memory overhead of the JVM but it still seems a large difference. What is the exact version number of Saxon that you run in the command line and what is the Saxon version that you set in the transformation scenario in Oxygen?

What schema does the TEI XML file declare? Is it TEI P4 or TEI P5? Where is the schema located: on your local Ubuntu computer or on the Web? If it is TEI P4 and the TEI P4 DTD is fetched from (a slow server on) the Web by the XSLT transformation in Oxygen that may explain (part of) the time difference.

If you use Saxon 9 in Oxygen and the validation of the XML source is enabled in the user preferences that may also explain (at least part of) the time difference.

Is it Ubuntu on 64 bits or on 32 bits?

The simple way of finding some answers is to send us both the XSLT script and the TEI XML document for reproducing the large time and memory difference.


Thank you,
Sorin
sderrick
Posts: 264
Joined: Sat Jul 10, 2010 4:03 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sderrick »

sorin wrote: What is the exact version number of Saxon that you run in the command line{/quote]
9.1.0.7
sorin wrote:what is the Saxon version that you set in the transformation scenario in Oxygen?
9.3.0.4
sorin wrote:What schema does the TEI XML file declare?
<TEI xmlns="http://www.tei-c.org/ns/1.0" xml:id="tfccs.main.sh">
sorin wrote: Is it TEI P4 or TEI P5?
TEI P5
sorin wrote:Where is the schema located: on your local Ubuntu computer or on the Web?
web
sorin wrote:If you use Saxon 9 in Oxygen and the validation of the XML source is enabled in the user preferences that may also explain (at least part of) the time difference.
not sure what setting that is
sorin wrote:Is it Ubuntu on 64 bits or on 32 bits?
32 bit 10.04
sorin wrote:The simple way of finding some answers is to send us both the XSLT script and the TEI XML document for reproducing the large time and memory difference.
Ok I'll do that

thanks

Scott
sderrick
Posts: 264
Joined: Sat Jul 10, 2010 4:03 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sderrick »

Oh, you mentioned the JVM via Oxygen using an inordinate amount of memory. On the command line which is finishing in 17 seconds vs Oxygens 3+ minutes and crashing, I am using the saxon9-9.1.0.7.jar. Using the same jar file only uses about 400KB of memory vs 1.5GB for Oxygen and using the same jar file only uses about 40% of my CPU vs Oxygen using almost 90%.

I don't think you can blame the JVM, since both our using the same JVM.

Scott
sorin_ristache
Posts: 4141
Joined: Fri Mar 28, 2003 2:12 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sorin_ristache »

Thank you for the files. I tested the XSLT 2.0 transformation in Oxygen 12.1 on both Windows and Linux and it was always finished in less than 15 seconds with a total memory taken by the JVM of around 500 MB, but no more than 200 MB for the Java heap inside the JVM (reported in the Help -> About dialog in the Oxygen application). The CPU of the Windows and Linux computers where I tested the transformation may be faster than your CPU but that does not explain the memory difference (500 MB in my test vs. 1.5 GB in your test) and also the time difference seems very large (about 12 - 13 seconds in Oxygen in my test vs. 3+ minutes in your test).

You said you used Saxon 9.3.0.4 in Oxygen so I assume you tested with Oxygen version 12.1 which is the same version that I used.

Did you have other (large) files also opened in Oxygen when you tested the transformation? Please check the Java memory reported in the Help -> About dialog before starting the transformation in Oxygen. What are the used memory and total memory values in the Help -> About dialog? Do you get the OutOfMemory error when only the two files (sh-tei.xml and extract_word_list.xsl) are opened in Oxygen at the moment of starting the transformation?


Regards,
Sorin
sderrick
Posts: 264
Joined: Sat Jul 10, 2010 4:03 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sderrick »

Of course I have other files loaded in the editor! Who wouldn't? All are pretty small, css files and other scripts.

I ran the script again.

Oxygen about box reports 55MB of memory used by JVM with just the two files loaded, xml and xsl. My system monitor says the JVM Oxygen is running in is using 125MB, hmmm....

I start the transform, the memory in the system monitor rises to 1.5GB. Oxygen UI is frozen and completely unusable so I can't check the about box. After about 3+ minutes the process crashes with an out of memory error saying 1.5GB just aint enough! I sent in the report on the crash. After the crash the about box says it is using 1.5GB of memory

Scott
sderrick
Posts: 264
Joined: Sat Jul 10, 2010 4:03 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sderrick »

I would really like to solve this problem.

I realize now that any time I run any script that lasts mroe than a second or two Oxygen is completely unresponsive until the script finishes..

Maybe setting the memory available in Oxygen's startup script to 1.5 GB was a bad idea?
sorin_ristache
Posts: 4141
Joined: Fri Mar 28, 2003 2:12 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sorin_ristache »

Before starting Oxygen please rename the folder with your Oxygen user preferences which is $HOME/.com.oxygenxml. Start Oxygen 12.1 and make sure only the XML and XSLT file necessary for the transformation are opened. A new folder $HOME/.com.oxygenxml will be created automatically with default values for all the user preferences. Please make sure that you set the same output file for the transformation (the Save As field in the Output tab of the dialog box for editing a transformations scenario) as in the command line transformation that you said takes only 15 seconds. (For example if you set in Oxygen a file path located on a slow network share the total time necessary for finishing the transformation may be very long regardless of the actual transformation time). Run the same transformation again in Oxygen 12.1 and see how much memory and time it takes.

If it is finished successfully (in less than 30 seconds and less than 500 MB or 600 MB of memory for the entire JVM) then the problem was an user preference in the folder that you renamed. In this case please send us a zip archive with the folder that you renamed (the old $HOME/.com.oxygenxml folder). We will try to find the user preference that caused the problem and let you know how it can be fixed.

If you get the same OutOfMemory error after 3+ minutes and the transformation is not finished please do the following:
  • replace your file [Oxygen12.1-install-folder]/lib/oxygen.jar with this file,
  • add a text file called log4j.properties in [Oxygen12.1-install-folder] with the following content:

    Code: Select all

    log4j.rootCategory= info, R2

    log4j.category.ro.sync.ui.application.ThreadWatchDog=debug

    log4j.appender.R2=org.apache.log4j.RollingFileAppender
    log4j.appender.R2.File=oxygen.log
    log4j.appender.R2.MaxFileSize=12000KB
    log4j.appender.R2.MaxBackupIndex=20
    log4j.appender.R2.layout=org.apache.log4j.PatternLayout
    log4j.appender.R2.layout.ConversionPattern=%d %r %p [ %t ] %c - %m%n
  • restart Oxygen 12.1
  • make sure the new Oxygen build number is 2011021011 in the Help -> About dialog
  • run the same transformation again
  • send us a zip archive with all the files oxygen.log* (that is oxygen.log, oxygen.log.1, oxygen.log.2, etc) that Oxygen generated in [Oxygen12.1-install-folder]

Thank you,
Sorin
sderrick
Posts: 264
Joined: Sat Jul 10, 2010 4:03 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sderrick »

Sorin,

Still had excessive resource drain and crash, after renaming my options folder.

Installed new oxygen.jar and created file. Ran transform which did crash after 3 minutes.

No log files were generated? I sent in the log4j.properties file I created and a picture of the about box in a problem report.

Scott
sderrick
Posts: 264
Joined: Sat Jul 10, 2010 4:03 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sderrick »

Sorin,

Log file not generated because I have Oxygen installed on /opt directory

I reinstalled in my home directory(completely new install), copied new .jar, setup log file generation, ran transform, crashed and then sent you the log file via the link provided.
sorin_ristache
Posts: 4141
Joined: Fri Mar 28, 2003 2:12 pm

Re: XSLT resource Problem in 12.1(crashes)

Post by sorin_ristache »

I just sent you a reply by email requesting more data and a more detailed log file generated on your computer. That should be more useful to figure out what went wrong on your Ubuntu machine. Please send the requested data.


Thank you,
Sorin
Post Reply