Page 1 of 1

Progress indicator for "Validate and check for completeness"

Posted: Wed Mar 15, 2017 1:23 am
by cbrzeski
We use the "Validate and check for completeness" option a lot on our maps and bookmaps, and it works great. However, it gives no indication of how far along it is, and because we have large maps, this means I have no idea when I can go home. :)
A simple indicator of how many topics have been checked/how many topics in the map would be really helpful. Thanks, Corinne

Re: Progress indicator for "Validate and check for completeness"

Posted: Wed Mar 15, 2017 9:34 am
by Radu
Hi Corinne,

The problem is that the processing itself has no idea how much time this will take. The process analyses each DITA resource starting from the main DITA Map, once it finds in a topic/map references to other DITA resources, it starts looking into those as well, so while the process continues, it may find more and more file references that it needs to look into.
So there is no telling how many resources will be finally processed.
By the way, about how many DITA topics do you have in the project and about how much time does it take to validate and check them all? Are the topic DTD-based or XML Schema based?
Do you have an SSD installed as a hard drive? A fast hard drive would help a lot.

Regards,
Radu

Re: Progress indicator for "Validate and check for completeness"

Posted: Wed Mar 15, 2017 5:14 pm
by cbrzeski
Thanks for getting back to me, Radu.

I'm not asking for a time estimate. Just a progress report. It shows me the ID of each topic it is checking as it goes through it, so it could tell me how many have been checked. I know how many are in the map (although that would be relatively easy for you to show as well, it seems).

The map I was working with yesterday has about 1900 topics in it. It took about 2.5 hours. Which reminds me of another suggestion: It would be really great if this ran in the background, because it keeps me from using Oxygen for anything else while it's running.

They are DITA files.

I (and our authors) get the hard drives we are given, so we won't be able to do much about that.

Do you think there's any possibility of some inefficiency in the API calls to Astoria in large maps such as this?

Thanks
Corinne

Re: Progress indicator for "Validate and check for completeness"

Posted: Thu Mar 16, 2017 12:01 pm
by Radu
Hi Corinne,

I understand. I will add an issue on our internal issues list and see what we can do about this.
I forgot you were using Astoria. The problem is that for each topic Oxygen needs to request its content from Astoria and this may take a while depending on the network connection speed and so on. So a faster hard drive will not help.
Does Astoria have a file export facility? For example export the entire project on the local drive and then run the validation on the local folder? This may take way less time because downloading a huge ZIP file from Astoria is faster than Oxygen requesting each file to be downloaded individually upon validation...

Regards,
Radu

Re: Progress indicator for "Validate and check for completeness"

Posted: Fri Mar 24, 2017 12:17 am
by cbrzeski
Thanks, Radu. I have used the export/check local method in the past, but the issue with these big publications is that the export itself takes a long time. Also, then I'm not working in the actual content, so I lose the convenience of being able to double-click an error in the completeness results and open the correct topic to edit.

Re: Progress indicator for "Validate and check for completeness"

Posted: Fri Mar 24, 2017 10:30 pm
by cbrzeski
Radu, I had another thought. The column headers for the completeness check results have item counts in some cases (for example, the Description column). They increment as the items are found. Perhaps you could increment a counter on the Resource column (ignoring duplicates) as each one is checked? That would provide a count that would be visible as the check runs, in the current UI, and not require a separate window. Just a thought.

Re: Progress indicator for "Validate and check for completeness"

Posted: Mon Mar 27, 2017 10:51 am
by Radu
Hi Corinne,

Thanks for the tip, we'll consider this as well.

Regards,
Radu

Re: Progress indicator for "Validate and check for completeness"

Posted: Mon Oct 02, 2017 2:51 pm
by Radu
Hi Corinne,

We released Oxygen 19.1 a couple of days ago and its Validate and check for completeness should report how many DITA resources it has processed so far.

Regards,
Radu

Re: Progress indicator for "Validate and check for completeness"

Posted: Mon Mar 12, 2018 9:58 pm
by cbrzeski
Thanks Radu! We are looking forward to 19.1 support from Astoria soon, so we can take advantage of this feature. Thanks.

Re: Progress indicator for "Validate and check for completeness"

Posted: Wed Feb 27, 2019 4:33 pm
by ckabstein
It would be really great if this ran in the background, because it keeps me from using Oxygen for anything else while it's running.
Hi,

We have the same issue that cbrzeski reported. Is there a chance that the validation process may run in the background one day? Or is this already the case and I've just missed the option where this can be set?

Thanks,
Christina

Re: Progress indicator for "Validate and check for completeness"

Posted: Thu Feb 28, 2019 12:00 pm
by Radu
Hi Christina,

We have an issue registered to make the validate and check for completeness run in the background some day, it makes sense and I will try to increase its priority. And we'll update this forum thread when this happens.

Regards,
Radu

Re: Progress indicator for "Validate and check for completeness"

Posted: Thu Feb 28, 2019 12:33 pm
by ckabstein
Thank you so much, Radu!

Looking forward to that.

Christina

Re: Progress indicator for "Validate and check for completeness"

Posted: Wed Aug 07, 2019 5:43 pm
by ckabstein
Hi,

any updates on this one? I got notified today that the validation of one manual with more than 2000 topics takes an hour now. :shock:
Too costly for not being able to work waiting for the validation to end.

Best regards,
Christina

Re: Progress indicator for "Validate and check for completeness"

Posted: Thu Aug 08, 2019 10:01 am
by sorin_carbunaru
Hello Christina,

I checked the improvement request (internal ID: EXM-12631) and its 'Fix Version' is set to 22.0 (the version we are currently working on), whose release is scheduled for February 2020. I mentioned on that issue the fact that you asked about its status, and hopefully this will increase its priority. As promised by Radu, we will update this thread when we release a version that has the improvement you requested.

All the best wishes,
Sorin Carbunaru
oXygen XML

Re: Progress indicator for "Validate and check for completeness"

Posted: Mon Aug 12, 2019 1:38 pm
by Radu
Hi Christina,

In addition to what Sorin said, a couple of remarks about this:
any updates on this one? I got notified today that the validation of one manual with more than 2000 topics takes an hour now.
One hour is a very long time, for 2000 topics I would expect a couple of minutes at most. Are your DITA resources located on some kind or remote server? Are they DTD or XML Schema/Relax NG based?
When you use the "Validate and check for completeness" action there is an "Include remote resources" checkbox. Do you have it checked on your side? If you uncheck it does the validation performance increase?

One more thing, you can also buy a special license which would allow you to run the validate and check for completeness from a command line:

https://www.oxygenxml.com/doc/versions/ ... -line.html

Regards,
Radu

Re: Progress indicator for "Validate and check for completeness"

Posted: Tue Aug 13, 2019 3:37 pm
by ckabstein
Hi Radu,
Are your DITA resources located on some kind or remote server? Are they DTD or XML Schema/Relax NG based?
No, we validate our local working copies from SVN repository. Our files are Relax NG based.
When you use the "Validate and check for completeness" action there is an "Include remote resources" checkbox. Do you have it checked on your side? If you uncheck it does the validation performance increase?
No, we don't use this option. But we use the Batch validate referenced DITA resources option which slows down the validation. If I turn that off, it runs faster. So I've run some more checks to see what is happening exactly when this option is activated/deactivated.

It turned out that if I activate this option, it finds more errors, for example, the typical Schematron info "The short description should be a single, concise paragraph containing one or two sentences [...]" and warnings, such as "It's preferable to use tm element instead of ™ character."
It also finds errors like "element "pd" not allowed yet; missing required element "pt""

So turning that off is not really an option either, especially since we use two validation scenarios.

Because of this, using the command line interface to validate looks like not really being an option, since that batch validation is turned off and we only need one of the provided scripting functions.

Thanks,
Christina

Re: Progress indicator for "Validate and check for completeness"

Posted: Wed Aug 14, 2019 8:00 am
by Radu
Hi Christina,

Right, the "Batch validate referenced DITA resources" behaves in a way as if each DITA topic is individually loaded and validated with its associated validation scenarios and this adds considerable more time to the validation process.

Regards,
Radu