Best practice: many, many files - in Oxygen, in filesystem in database

[bbfkaes]

Dear all,
my question to everyone working with oxygen and xml is, how to work with a large amount of files. I mean large in number, several thousand.
For example, when you write a transformation, which converts a large 1 GB data-file into 200.000 seperate data-files.
There are several obstacles in the filesystem and in applications (editors, git, databases), which make it difficult to work with a large number, e.g. writing may be very fast, deleting is very slow, listing in Oxygen is very very slow, jumping in oxygen from one opened file to another is very very slow.

What is best practice to handle that? I develop and test with small datasets. Large dirs should be outside of git and oxygen? In database only production-ready results? Distribute files into dir-trees?

Thanks for your hints!

[Radu]

Hi,

The Oxygen XML Forum is not a very good place where to ask the community questions as most people on the forum register to receive notifications only for their posts.
So maybe the Oxygen XML Users List is a better place:
https://www.oxygenxml.com/mailman/listinfo/oxygen-user
There is also an XML Slack channel on which lots of XML experts are registered, there is an invitation URL here:
https://www.xml.com/news/2020-04-slack- ... community/

Here are some of my suggestions:

1) Increase the maximum amount of memory allocated to Oxygen as much as possible to at least 4GBs but preferably more, maybe 10Gbs if your system allows:
https://www.oxygenxml.com/doc/versions/ ... emory.html
In the Oxygen main menu Help->About you will find the "Total memory" label available.
2) Avoid opening and editing at the same time more than one huge file in Oxygen.
3) When a huge file is opened in Oxygen, Oxygen will propose to open it with the huge file editing support enabled, please follow its advice to do so:
https://www.oxygenxml.com/doc/versions/ ... ditor.html
4) When using XSLT transformations, avoid opening the XML documents at all, configure the XSLT transformations by defining transformation scenarios directly from the Oxygen project view. Make sure that in the transformation scenario edit dialog in the Output tab you have unchecked the "Show in results view as ->XML" and "Open in Editor" checboxes.

The alternative is to set up some server side processing using the Saxon Home Edition XSLT processor or use some command line scripts to run the transformation from the command line.

Regards,
Radu