Progress indicator for "Validate and check for completeness"

Are you missing a feature? Request its implementation here.
cbrzeski
Posts: 17
Joined: Wed Mar 15, 2017 1:19 am

Progress indicator for "Validate and check for completeness"

Post by cbrzeski »

We use the "Validate and check for completeness" option a lot on our maps and bookmaps, and it works great. However, it gives no indication of how far along it is, and because we have large maps, this means I have no idea when I can go home. :)
A simple indicator of how many topics have been checked/how many topics in the map would be really helpful. Thanks, Corinne
Radu
Posts: 9018
Joined: Fri Jul 09, 2004 5:18 pm

Re: Progress indicator for "Validate and check for completeness"

Post by Radu »

Hi Corinne,

The problem is that the processing itself has no idea how much time this will take. The process analyses each DITA resource starting from the main DITA Map, once it finds in a topic/map references to other DITA resources, it starts looking into those as well, so while the process continues, it may find more and more file references that it needs to look into.
So there is no telling how many resources will be finally processed.
By the way, about how many DITA topics do you have in the project and about how much time does it take to validate and check them all? Are the topic DTD-based or XML Schema based?
Do you have an SSD installed as a hard drive? A fast hard drive would help a lot.

Regards,
Radu
Radu Coravu
<oXygen/> XML Editor
http://www.oxygenxml.com
cbrzeski
Posts: 17
Joined: Wed Mar 15, 2017 1:19 am

Re: Progress indicator for "Validate and check for completeness"

Post by cbrzeski »

Thanks for getting back to me, Radu.

I'm not asking for a time estimate. Just a progress report. It shows me the ID of each topic it is checking as it goes through it, so it could tell me how many have been checked. I know how many are in the map (although that would be relatively easy for you to show as well, it seems).

The map I was working with yesterday has about 1900 topics in it. It took about 2.5 hours. Which reminds me of another suggestion: It would be really great if this ran in the background, because it keeps me from using Oxygen for anything else while it's running.

They are DITA files.

I (and our authors) get the hard drives we are given, so we won't be able to do much about that.

Do you think there's any possibility of some inefficiency in the API calls to Astoria in large maps such as this?

Thanks
Corinne
Radu
Posts: 9018
Joined: Fri Jul 09, 2004 5:18 pm

Re: Progress indicator for "Validate and check for completeness"

Post by Radu »

Hi Corinne,

I understand. I will add an issue on our internal issues list and see what we can do about this.
I forgot you were using Astoria. The problem is that for each topic Oxygen needs to request its content from Astoria and this may take a while depending on the network connection speed and so on. So a faster hard drive will not help.
Does Astoria have a file export facility? For example export the entire project on the local drive and then run the validation on the local folder? This may take way less time because downloading a huge ZIP file from Astoria is faster than Oxygen requesting each file to be downloaded individually upon validation...

Regards,
Radu
Radu Coravu
<oXygen/> XML Editor
http://www.oxygenxml.com
cbrzeski
Posts: 17
Joined: Wed Mar 15, 2017 1:19 am

Re: Progress indicator for "Validate and check for completeness"

Post by cbrzeski »

Thanks, Radu. I have used the export/check local method in the past, but the issue with these big publications is that the export itself takes a long time. Also, then I'm not working in the actual content, so I lose the convenience of being able to double-click an error in the completeness results and open the correct topic to edit.
cbrzeski
Posts: 17
Joined: Wed Mar 15, 2017 1:19 am

Re: Progress indicator for "Validate and check for completeness"

Post by cbrzeski »

Radu, I had another thought. The column headers for the completeness check results have item counts in some cases (for example, the Description column). They increment as the items are found. Perhaps you could increment a counter on the Resource column (ignoring duplicates) as each one is checked? That would provide a count that would be visible as the check runs, in the current UI, and not require a separate window. Just a thought.
Radu
Posts: 9018
Joined: Fri Jul 09, 2004 5:18 pm

Re: Progress indicator for "Validate and check for completeness"

Post by Radu »

Hi Corinne,

Thanks for the tip, we'll consider this as well.

Regards,
Radu
Radu Coravu
<oXygen/> XML Editor
http://www.oxygenxml.com
Radu
Posts: 9018
Joined: Fri Jul 09, 2004 5:18 pm

Re: Progress indicator for "Validate and check for completeness"

Post by Radu »

Hi Corinne,

We released Oxygen 19.1 a couple of days ago and its Validate and check for completeness should report how many DITA resources it has processed so far.

Regards,
Radu
Radu Coravu
<oXygen/> XML Editor
http://www.oxygenxml.com
cbrzeski
Posts: 17
Joined: Wed Mar 15, 2017 1:19 am

Re: Progress indicator for "Validate and check for completeness"

Post by cbrzeski »

Thanks Radu! We are looking forward to 19.1 support from Astoria soon, so we can take advantage of this feature. Thanks.
ckabstein
Posts: 142
Joined: Fri Apr 24, 2015 12:28 pm

Re: Progress indicator for "Validate and check for completeness"

Post by ckabstein »

It would be really great if this ran in the background, because it keeps me from using Oxygen for anything else while it's running.
Hi,

We have the same issue that cbrzeski reported. Is there a chance that the validation process may run in the background one day? Or is this already the case and I've just missed the option where this can be set?

Thanks,
Christina
oXygen XML Editor 25.0 build 2023013006
DITA OT 3.7.3
Radu
Posts: 9018
Joined: Fri Jul 09, 2004 5:18 pm

Re: Progress indicator for "Validate and check for completeness"

Post by Radu »

Hi Christina,

We have an issue registered to make the validate and check for completeness run in the background some day, it makes sense and I will try to increase its priority. And we'll update this forum thread when this happens.

Regards,
Radu
Radu Coravu
<oXygen/> XML Editor
http://www.oxygenxml.com
ckabstein
Posts: 142
Joined: Fri Apr 24, 2015 12:28 pm

Re: Progress indicator for "Validate and check for completeness"

Post by ckabstein »

Thank you so much, Radu!

Looking forward to that.

Christina
oXygen XML Editor 25.0 build 2023013006
DITA OT 3.7.3
ckabstein
Posts: 142
Joined: Fri Apr 24, 2015 12:28 pm

Re: Progress indicator for "Validate and check for completeness"

Post by ckabstein »

Hi,

any updates on this one? I got notified today that the validation of one manual with more than 2000 topics takes an hour now. :shock:
Too costly for not being able to work waiting for the validation to end.

Best regards,
Christina
oXygen XML Editor 25.0 build 2023013006
DITA OT 3.7.3
sorin_carbunaru
Posts: 398
Joined: Mon May 09, 2016 9:37 am

Re: Progress indicator for "Validate and check for completeness"

Post by sorin_carbunaru »

Hello Christina,

I checked the improvement request (internal ID: EXM-12631) and its 'Fix Version' is set to 22.0 (the version we are currently working on), whose release is scheduled for February 2020. I mentioned on that issue the fact that you asked about its status, and hopefully this will increase its priority. As promised by Radu, we will update this thread when we release a version that has the improvement you requested.

All the best wishes,
Sorin Carbunaru
oXygen XML
Radu
Posts: 9018
Joined: Fri Jul 09, 2004 5:18 pm

Re: Progress indicator for "Validate and check for completeness"

Post by Radu »

Hi Christina,

In addition to what Sorin said, a couple of remarks about this:
any updates on this one? I got notified today that the validation of one manual with more than 2000 topics takes an hour now.
One hour is a very long time, for 2000 topics I would expect a couple of minutes at most. Are your DITA resources located on some kind or remote server? Are they DTD or XML Schema/Relax NG based?
When you use the "Validate and check for completeness" action there is an "Include remote resources" checkbox. Do you have it checked on your side? If you uncheck it does the validation performance increase?

One more thing, you can also buy a special license which would allow you to run the validate and check for completeness from a command line:

https://www.oxygenxml.com/doc/versions/ ... -line.html

Regards,
Radu
Radu Coravu
<oXygen/> XML Editor
http://www.oxygenxml.com
ckabstein
Posts: 142
Joined: Fri Apr 24, 2015 12:28 pm

Re: Progress indicator for "Validate and check for completeness"

Post by ckabstein »

Hi Radu,
Are your DITA resources located on some kind or remote server? Are they DTD or XML Schema/Relax NG based?
No, we validate our local working copies from SVN repository. Our files are Relax NG based.
When you use the "Validate and check for completeness" action there is an "Include remote resources" checkbox. Do you have it checked on your side? If you uncheck it does the validation performance increase?
No, we don't use this option. But we use the Batch validate referenced DITA resources option which slows down the validation. If I turn that off, it runs faster. So I've run some more checks to see what is happening exactly when this option is activated/deactivated.

It turned out that if I activate this option, it finds more errors, for example, the typical Schematron info "The short description should be a single, concise paragraph containing one or two sentences [...]" and warnings, such as "It's preferable to use tm element instead of ™ character."
It also finds errors like "element &quot;pd&quot; not allowed yet; missing required element &quot;pt&quot;"

So turning that off is not really an option either, especially since we use two validation scenarios.

Because of this, using the command line interface to validate looks like not really being an option, since that batch validation is turned off and we only need one of the provided scripting functions.

Thanks,
Christina
oXygen XML Editor 25.0 build 2023013006
DITA OT 3.7.3
Radu
Posts: 9018
Joined: Fri Jul 09, 2004 5:18 pm

Re: Progress indicator for "Validate and check for completeness"

Post by Radu »

Hi Christina,

Right, the "Batch validate referenced DITA resources" behaves in a way as if each DITA topic is individually loaded and validated with its associated validation scenarios and this adds considerable more time to the validation process.

Regards,
Radu
Radu Coravu
<oXygen/> XML Editor
http://www.oxygenxml.com
Post Reply