Mac OS X: Where is the Lucene index stored?

[joerg.hambuch]

Hello,

the documentation http://www.oxygenxml.com/doc/ug-standal ... pects.html contains the path to index files storage on Windows systems. Unfortunately I can't find the index files on my Mac. I tried to find it in /Users/myusername/Library/Preferences/com.oxygenxml, I used the 'find' command in teh terminal to search for 'lucene' and some of the files I found on a Windows system, e.g. _COMPATIBILITY_, *.fdx, segments.gen.

Best regards
Jörg

[adrian]

Hi,

The index folders for each project file are stored in the lucene folder from the Oxygen options folder. The location of the options folder is platform dependent, as described here:
http://www.oxygenxml.com/doc/ug-editor/ ... ences.html

So, on the Mac they are located in:
/Users/myusername/Library/Preferences/com.oxygenxml/lucene
Note that for the Oxygen XML Author product the options folder is named com.oxygenxml.author and for the Oxygen XML Developer it's com.oxygenxml.developer.

Regards,
Adrian

[joerg.hambuch]

Hello Adrian,

thanks for your quick reply. You confirm, what I thought: In the mentioned path there should be a folder called "lucene". My problem is: there is no such folder on my Mac in /Users/myuser/Library/Preferences/com.oxygenxml/

mymac:com.oxygenxml myuser$ ls -la
total 392
drwxr-xr-x 22 myuser staff 748 13 Nov 17:58 .
drwx------+ 301 myuser staff 10234 14 Nov 11:45 ..
-rw-r--r-- 1 myuser staff 2430 13 Nov 17:41 defaultLayout14.1.layout
-rw-r--r-- 1 myuser staff 2196 30 Okt 14:30 defaultLayout15.0.layout
drwxr-xr-x 4 myuser staff 136 24 Sep 12:58 extensions
-rw-r--r-- 1 myuser staff 2093 13 Nov 17:41 file.history
drwxr-xr-x 3 myuser staff 102 17 Jan 2013 hcache
-rw-r--r-- 1 myuser staff 889 9 Jan 2013 license.xml
-rw-r--r-- 1 myuser staff 60635 13 Nov 17:41 oxyOptionsSa14.1.xml
-rw-r--r-- 1 myuser staff 55781 24 Okt 12:12 oxyOptionsSa14.1.xml.tmp
-rw-r--r-- 1 myuser staff 38646 30 Okt 14:30 oxyOptionsSa15.0.xml
-rw-r--r-- 1 myuser staff 4 13 Nov 17:58 oxygen14.1.lock
-rw-r--r-- 1 myuser staff 291 30 Okt 14:30 pluginsOptions15.0.xml
-rw-r--r-- 1 myuser staff 211 13 Nov 17:58 project.history
drwxr-xr-x 7 myuser staff 238 14 Nov 10:47 projectCache
drwxr-xr-x 12 myuser staff 408 11 Jan 2013 spell
-rw-r--r-- 1 myuser staff 4153 24 Jan 2013 svnClientFlexible14.1.layout
-rw-r--r-- 1 myuser staff 2886 24 Sep 12:21 svnClientStatic14.1.layout
drwxr-xr-x 3 myuser staff 102 9 Jan 2013 usageData
drwxr-xr-x 4 myuser staff 136 20 Feb 2013 wcache
drwxr-xr-x 6 myuser staff 204 24 Sep 12:56 welcome
drwxr-xr-x 7 myuser staff 238 30 Okt 14:15 welcome_temp

Ok, now you could think, I have not created an index at all. But I do have: It contains 855503 resources and can be searched in the blink of an eye

So maybe to the background of my question: Building this (quite big) index takes about 4 seconds on my Mac (16 GB RAM, SSD). When trying to build the same index on a Windows PC with 2 GB RAM and a HDD, it takes me over 10 (ten!) hours. I think this large difference in time can't be explained by hardware only. The xpr file the index is build from is 95 MB of size.

To save time, I planned to create the index on my Mac and transfer it to some Windows machines. But how to find it on the Mac? Strange. I am getting th eimpression, my <oXygen/> XML Editor 14.1, build 2012121012 does not use Lucene at all, but does some magic indexing

Best regards
Jörg

[adrian]

Hi,

Please note that two types of indexing are available in Oxygen:
- file path indexing (introduced in v12.0)
- content indexing (introduced in v14.2)

The file path indexing is very fast (it's just a file system scan), especially with an SSD.
The content indexing takes some while even with an SSD when a lot of content must be processed.
Note that once the content index is created (in the lucene folder), Oxygen tries to preserve it as much as it can. So if you press the Reindex button, Oxygen will not re-create the index from scratch, instead it will scan the file system, check for changes since the last indexing and re-index only the changed content.

You said you use on your Mac v14.1 which doesn't have content indexing. That explains why it's so fast and why there's no lucene folder. The file path index was kept in the projectCache folder.
On the Windows PC (probably running v14.2 or newer), if you're only interested in the file path indexing (quick), you can simply disable the content indexing (slow): Options > Preferences, Open/Find Resource, Enable searching in content.

Regarding the attempt to copy the index (even though it was not the case here), the Lucene index cannot be used after being copied to another machine (especially a different OS). Absolute paths are used for the indexed files, so they most likely won't match the ones from the destination machine.

Regards,
Adrian

[joerg.hambuch]

Hi Adrian,

wow! Your crystal ball seems to be working perfectly! You are absolutely right, the Windows system is running Oxygen XML 14.2. Thank you very much for your brilliant explanation! You got it right on the point.

Best regards and thank you so much!
Jörg

[joerg.hambuch]

Hi Adrian,

you wrote "... v14.1 [...] doesn't have content indexing. That explains why it's so fast and why there's no lucene folder. The file path index was kept in the projectCache folder."

Our resources are stored in an eXist database and are accessed via WebDAV. So paths to the documents should be the same, even if accessed from different platforms (Win/Mac).

So I wonder if it is possible to create an filepath index (no content index) on 14.1 and distribute this index to several computers (Mac and Win). I tried this, but without success. On a PC where I put the created index (by copying the files from projectCache folder), Oxygen did not recognize the index files and started indexing automatically, as if there were no index at all.

I think, there might be a place in a config file where Oxygen maps index files on to projects.

Best regards
Jörg

[adrian]

Hi,

There is no path index for folders imported in the project from a remote location (HTTP/FTP/WebDAV). The import itself collects the list of folders and files, and replicates the folder structure in your project as logical folders and creates links for all the remote resources. So the all the paths for remote resources are already indexed statically in the project file at the moment the import was made. There is no dynamic path indexing for remote resources, as it is the case for folders linked from the local file system.

The index is not really meant to be moved around between machines (especially not across different operating systems). While it may be possible in some circumstances to do so, it is not practical.
Also note that the filepath index from v14.1 and earlier is not compatible with the one from v14.2 and later (the same is true the other way around).

Regards,
Adrian