xinclude 1.1 and Calabash within Oxygen

[dcramer]

Hi there,

I've written an xproc pipeline that uses features of xinclude 1.1 to fix duplicate xml:ids as described here:

https://norman.walsh.name/2011/10/03/transclusion
https://www.w3.org/TR/xinclude-11/#attribute-copying

When I run the pipeline from the command line (using xmlcalabash 1.1.16-97), it works fine. However, when I run the same pipeline using an xproc transformation scenario from within Oxygen, the attributes on the xi:include element are not passed down to the result document.

Is there something different about Oxygen's xmlcalabash distribution that might be interfering with this feature?

Also, is support for xinclude 1.1 within validation scenarios such that I can have in-editor validation based on the result of a transclusion-fixup.xsl or xproc pipeline?

Regards,
David

[adrian]

Hi,

In short, the latest version of Oxygen available at this time, v19.0, does not yet support XInclude 1.1 features. This support will be available starting with v19.1 (this fall). We will notify this thread when v19.1 is released.

What version of Oxygen are you using (Help > About)? This determines the version of Xerces and of XML Calabash being used in Oxygen. Oxygen v19.0 includes XML Calabash 1.1.15-97.
If XML Calabash 1.1.16-97 works for you independently from Oxygen, you could configure Oxygen to use it as a custom engine in Options > Preferences, XML > XProc:
XProc Preferences
You can then configure your XProc transformation scenario to use that custom engine.

Regards,
Adrian

[adrian]

Hi,

There's one other aspect that I thought I should mention.

However, when I run the same pipeline using an xproc transformation scenario from within Oxygen, the attributes on the xi:include element are not passed down to the result document.

Oxygen processes the XIncludes by default (merges the referred content into the document at parsing time), but may not know what to do with those attributes on the xi:include element. If you want to process the XIncludes yourself in the XProc script (let Calabash do it), you can disable Oxygen's support in Options > Preferences, XML > XML Parser, 'Enable XInclude processing'. Note that this is a global setting, so it affects all Oxygen features.

Regards,
Adrian

[dcramer]

Thanks Adrian. Turning off "Enable Xinclude Processing" globally did indeed make my XProc transformation scenario work the same as when I run the pipeline from the command line.

That's good news about the support for xinclude 1.1 in 19.1! Having support for it in my pipeline without support in the editor would be very unfortunate. I look forward to the announcement.

I assume that once xinclude 1.1 is supported in Oxygen, the workaround of disabling xinclude processing in the Oxygen xml parser will no longer be necessary for xproc transformation scenarios.

Btw., I'm using "<oXygen/> XML Editor 19.0, build 2017042020".

Regards,
David

[adrian]

Hi,

I assume that once xinclude 1.1 is supported in Oxygen, the workaround of disabling xinclude processing in the Oxygen xml parser will no longer be necessary for xproc transformation scenarios.

Actually that option has nothing to do with XInclude 1.1 and adding support for XInclude 1.1 does not eliminate the need for this workaround. Oxygen (pre)processes XIncludes for all transformations (including XProc) depending on that option. We have an issue logged separately for this mater, to let XProc transformations handle XInclude.

Regards,
Adrian

[dcramer]

I'm experimenting with the xinclude 1.1 support in Oxygen 19.1 and want to understand how it works. I have a test document that xincludes the same section several times.

When I validate this document in Oxygen 19.1, it no longer complains about the repeated xml:id attributes. Yea! What's not apparent is the strategy Oxygen uses to fix the duplicated xml:id attributes and linkends to those ids.

If I transform the document with a simple identity transform in Oxygen's debugger, the xml:ids all come through unchanged, so Oxygen only takes xincludes into account for validation and it's still up to me to fix duplicates. That's what I'd expect, but I'd like to know what logic Oxygen does with IDs and IDREFs so I can be sure I'm doing more-or-less the same thing.

Thanks,
David

[dcramer]

Ok, correction: Oxygen is passing down attributes to the xincluded content, but it's not doing any fixup of the duplicate ids. Initially, Oxygen was reporting validation errors due to my extra attributes on my xi:include element that were being passed down. This was apparently causing it not to report validation errors due to duplicate ids. When I modified my schema to allow my db:idfixup and db:idprefix attributes everywhere, Oxygen started reporting the duplicate ids.

So my new question is: how can I get Oxygen to ignore these duplicate xml:ids that will be taken care of by my fixup xslts?

[Radu]

Hi David,

We did some modifications/patches to the Xerces parser in order to pass attributes defined on the xi:include directly on the included content. And this should work based on this part in the xi:include 1.1 specs:
https://www.w3.org/TR/xinclude-11/#attribute-copying

Let's try to discuss this with a small example.
So you have a Docbook 5 "section1.xml" with some simple content:

Code: Select all

<sect1 xmlns="http://docbook.org/ns/docbook" version="5.0"

    xmlns:xlink="http://www.w3.org/1999/xlink">

    <title>FS</title>

    <para>P</para>

</sect1>

and it is xi:included twice in a main document like:

Code: Select all

<article xmlns="http://docbook.org/ns/docbook" version="5.0"

    xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:abc="def">

    <title>Welcome to Docbook</title>

    <xi:include href="section1.xml" xmlns:xi="http://www.w3.org/2001/XInclude" xml:id="sect1"/>

    <xi:include href="section1.xml" xmlns:xi="http://www.w3.org/2001/XInclude" xml:id="sect2"/>

</article>

Because of the new fixes in Oxygen 19.1, after we apply an XSLT which does identity transform the result is this one:

Code: Select all

<article xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:abc="def" version="5.0">

    <title>Welcome to Docbook</title>

    <sect1 version="5.0" xml:base="section1.xml" xml:id="sect1">

    <title>FS</title>

    <para>P</para>

</sect1>

    <sect1 version="5.0" xml:base="section1.xml" xml:id="sect2">

    <title>FS</title>

    <para>P</para>

</sect1>

</article>

So the xml:id has migrated from the xi:include to the xi:included element.

Could you modify my small example in order to show me what the problem is on your side?
Also is the behavior you expect Oxygen to have something which is covered in the xi:include 1.1 specification?

Regards,
Radu

[dcramer]

I've pushed up an example illustrating the features from the xinclude 1.1 spec and the differences between the behavior of Oxygen and Calabash:

https://github.com/dwcramer/xinclude-at ... py-example

[dcramer]

Btw., I suppose all this implies an update to the xinclude insertion Action/wizard. At the very least you'll need a way to set the value for the set-xml-id attribute, perhaps with an "auto" option that defers to the ID Options?

Maybe also a way to add other attributes that are allowed on xi:include. For example, I've modified my schema to allow profiling attributes in the local-attributes namespace so we can profile based on attributes added to the xi:include element:

Code: Select all

<xi:include href="section.xml" xmlns:la="http://www.w3.org/2001/XInclude/local-attributes" la:condition="profile-this"/>

I'm also supporting idfixup and idprefix attributes so I can fix xml:ids with prefixes etc.

Regards,
David

[Radu]

Hi David,

Thanks for the samples, I know understand the situation. I will try to make a fix for the xi:include 1.1 situations which right now do not properly work (the set-xml-id attribute and the xml:* attributes which should not be propagated down the hierarchy of nodes) and we'll probably include the fix in the next Oxygen 19.1 minor bug fix release (in a couple of weeks).

About the dialog used to insert xi:includes, I'm not sure if we should increase its functionality or not. People can first insert the xi:include and then use the Attributes view to add attributes to it. Maybe if we receive enough feedback and people actually start using these features...
Most people using xi:include are using it with Docbook and lately we've had more people work with DITA which has a much better and flexible content reuse support.

About this remark:

I'm also supporting idfixup and idprefix attributes so I can fix xml:ids with prefixes etc.

This was an interesting plan which did not make it in the XInclude 1.1 specs, right?

Regards,
Radu

[dcramer]

Thanks for taking care of that. Life is much easier when Oxygen and Calabash agree about things.

The only thing requried to support things like db:idprefix is that attributes be passed down to the xincluded content. You can then implement arbitrary post-processing to fix ids, turn data into DocBook, or whatever. It ends up being like a function call for content where those attributes are arguments.

AFAICT, the only difference between the final xinclude spec and what was envisioned [1] is the use of the set-xml-id attribute to override the xml:id of the xincluded element.

But checking IDs and IDREFS during validation will have to be done with schematron to take the db:idprefix stuff into account.

David

[1] https://norman.walsh.name/2011/10/03/transclusion

[Radu]

Hi David,

I understand. ID/IDREF indeed would need to be handled with a separate validation layer, for starters the linkend attribute for example would no longer be declared as an IDREF in the schema in order not to generate ID/IDREF errors coming from the primary schema.

Regards,
Radu

[dcramer]

We're using nvdl, so I can skip id/idref checking by omitting this option:

Code: Select all

<option name="http://www.thaiopensource.com/validate/check-id-idref"/>

Using nvdl (or something we're doing) might be borking other aspects of the framework, but I'll start another thread about that.

David

[dcramer]

It ended up being easy to implement schematron-based validation for IDs and IDREFs by importing transclusion-fixup.xsl and reusing the functions and templates in my schematron rules (so if I change the logic in transclusion-fixup.xsl it will propagate to my schematron).

Doing this in schematron turned out to be more flexible. For example, I'm only checking IDREFs if the root is book or article.

David

[Radu]

Hi,

As an update the xi:include 1.1 implementation problems discovered by David should be fixed in the latest Oxygen 19.1 minor bug fix release available on our web site:

https://www.oxygenxml.com/build_history.html#2017102417

Regards,
Radu

[dcramer]

Yes, thanks! I have xinclude 1.1 working beautifully in Oxygen and my xproc-based build pipeline.

Regards,
David

[Radu]

Hi David,

Thanks for confirming this.

Regards,
Radu