Out of Memory Opening Very Large .xml file

Post by **jtwhaley** » Thu Mar 09, 2006 4:50 pm

Hi everyone,

I'm attempting to open up a 1GB .xml that was extracted from another system. In doing so, I'm getting a OutOfMemoryError. I edited oxygen.bat and changed the line invoking the jva to read:

Code: Select all

java -Xmx2GB  -cp %CP% ro.sync.exml.Oxygen %1 %2 %3 %4 %5 %6 %7

previous values of -Xmx tried were 256M and 512M.

My system's actual memory is only 512. Is there some other parameter I can alter to allow this monstrous xml file to be opened?

Thanks,
Jason[/code]

Post by **george** » Thu Mar 09, 2006 4:58 pm

Hi Jason,

That is a large file indeed and clearly oXygen cannot handle a 1GB file on that machine. You should give oXygen maximum 256 MB of memory (more that this may cause the performace to decrease as your system will start using the disk for memory) and process documents up to 20 MB.

Best Regards,
George

Post by **george** » Thu Mar 09, 2006 4:59 pm

And as a suggestion, it is possible to split that file in smaller fragments using either entities or XInclude. You can do that for instance if you write a program at SAX level and serialize the document in multiple files.

Best Regards,
George

Post by **jtwhaley** » Thu Mar 09, 2006 5:04 pm

Thanks George,

It could be possible to split those xml files, but we will be working with them frequently and it would be a bit of a hassle do that each time.

Do you have any other suggestions?

Post by **george** » Thu Mar 09, 2006 5:08 pm

In the last release of oXygen, 7.1 we added support for eXist and Berkeley DB XML databases. You may consider dropping your large file into such a database and then use XQuery to obtain data from your file.

Best Regards,
George