JSON Schematron Validation

[scottbdr]

Hi, I just came across your JSON Schematron info and was curious to know more about what happens under the covers to make this work. Given the lack of more established JSON Schematron implementations out on the Internet, I was very interested to see what your approach was. We are currently looking at moving from XML to JSON for a metadata standard of ours for various reasons, however the lack of robust Schematron-like option is a concern. The big question is the algorithm for JSON > XML (I'm assuming the Schematron is not applied directly to the JSON... is it not?). It would seem a full understanding of this would be needed to write the Schematron. There are multiple ways of going about it and no standard... some have a focus on round-tripping, others on usability of resultant XML.

Any insights on where this sort of validation is going or info on further plans for this capability would be great. Really appreciate that Oxygen is supporting JSON, YAML right now since they exist side-by-side with XML in many contexts.

Also, was wondering about whether its possible to plug in an JSON Schema validator like AJV into Oxygen.

Thanks, Scott

[teo]

Hi Scott,

We thought at one point that it would be nice if, in addition to the usual restrictions that can be imposed on JSON files through JSON schemas, we could somehow expand the range of restrictions with Schematron style constraints.
And that's how the idea of implementing a JSON Schematron Validator engine appeared.
Of course, behind the scenes it's about the conversion of JSON to XML, but the highlighting of errors and their location is done in the validated JSON file.

You can follow the link below for more details:
https://www.oxygenxml.com/doc/versions/ ... schematron

In the section linked above there is a tip that states "Inside the samples folder, there are some files you can use to see how Schematron validation can be done with JSON files. The path of the folder containing these sample files is: [OXYGEN_INSTALL_DIR]/samples/json/schematron/".

Open "library.json" and "library.sch" (from the indicated folder) in Oxygen.
Note that Schematron rules are written in the usual way, and we believe you don't need to convert JSON to XML to write those rules.

Validation can then be done in 2 ways, using the "Validate" menu button in the toolbar and invoking either the "Validate with..." or "Configure Validation Scenario(s)" actions.
In case of the last option, "JSON Schematron Validator" must be chosen as the validation engine.
Also, the validation scenarios configuration dialog allows specifying multiple validation units.
For example, one of the units would allow validation against a JSON Schema, and a second unit would allow validation against a Schematron.

By the way, validating the sample JSON file with the Schematron in question, there seems to be a small problem with the rule in the "birth_year" context, as one of the errors presented should not actually be reported.
I have already logged the issue to be addressed and fixed in the next release.
Anyway, I think that by analyzing the indicated sample files and following the explanations regarding the validation actions, you will get a clear idea of how JSONs can be validated against Schematron in Oxygen.

Regards,
Teo

[scottbdr]

Thanks for the response. Can I assume the JSON to XML transformation for the Schematron uses the same process as the Tools> JSON > JSON to XML.. tool? I've noticed some odd differences in the outputs from that tool. For example, if I have an array with only one item, like this:

Code: Select all

{
    "contributors": [
        {
            "organization": "some org",
            "role": "producer"
        }
    ] }

I get back

Code: Select all

<contributors>
	<array>
		<organization>some org</organization>
		<role>producer</role>
	</array>
<contributors>

However, if the same structure has more than one item, I get this:

Code: Select all

<contributors>
	<organization>some org</organization>
	<role>producer</role>
<contributors>
<contributors>
	<organization>some other org</organization>
	<role>producer</role>
<contributors>

I'm assuming the <array> element is a bug. Do you know if your JSON > XML is reliably round-trip-able? I've just not been able to find much in the way of a standard for JSON > XML so it's always a bit of a mystery what algorithm I'm dealing with and what it's aims were (round-trips? easy to use XML? etc.). Thanks again.

[teo]

Please follow the link below to find more details about JSON to XML conversion:
https://www.oxygenxml.com/doc/versions/ ... o-XML.html

JSON to XML conversion is not 100% guaranteed to be round trip.
Some elements, such as <JSON> or <array> are added to the resulting XML document to ensure its validity and, as much as possible, in the case of invoking the reverse conversion (XML to JSON) to try obtaining the JSON document in its original form.
But this is not 100% guaranteed, as I said.
There may be a few situations where a certain JSON document structure prevents JSON -> XML -> JSON conversion from being a perfect round trip.
On the other hand, an XML -> JSON -> XML (round trip) conversion cannot "recreate" the original XML document if there are XML elements that contain attributes.

However, as far as I know, we already have registered a couple of improvement tasks regarding these round-trip conversions (except for XML attributes presence). So, it's very possible to bring further improvements in the near future.

[florin_nica]

Hi Scott,

Regarding your question about plugging a JSON Schema validator into Oxygen, the short answer is yes. We even recently added an external validator called json-sKema (https://github.com/erosb/json-sKema), which is available as an add-on starting with Oxygen version 26.0.

However, plugging a validator designed in Javascript, such as AJV, is a much bigger hassle. Our editor's source code is written in Java, so in order to execute the validator's specific Javascript functions, the code needs to be compiled by an external engine such as Rhino JS or Graal.

Find more details about creating an Oxygen plugin at https://www.oxygenxml.com/doc/versions/ ... ugins.html

Regards,
Florin

[scottbdr]

Thanks again for the responses. Makes sense about AJV not being a good fit for a Java implementation.

Regarding JSON > XML, the more I've read to more I understand the fundamental difficulties with a round trip or with conversion more generally. I have to say I'm still a little unhappy with the conversion I get out of Oxygen. For example, take this (admittedly odd) structure:

Code: Select all

{   
 "arrayOfStuff": [
     [1,2,3],
        {
            "foo": "bar","baz":"bux"
        },
                {
            "list": [1,2,3]
        }
    ]
}

Plugged into this online converter, you get a more consistent use of the 'element' keyword (the default rather than 'array' - it also allows that name to be set, along with the root element which is nice...).

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<root>
   <arrayOfStuff>
      <element>
         <element>1</element>
         <element>2</element>
         <element>3</element>
      </element>
      <element>
         <baz>bux</baz>
         <foo>bar</foo>
      </element>
      <element>
         <list>
            <element>1</element>
            <element>2</element>
            <element>3</element>
         </list>
      </element>
   </arrayOfStuff>
</root>

In this converter, you get a consistent lack of any array keyword, which I've seen in many converters:

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?>
<root>
	<arrayOfStuff>1</arrayOfStuff>
	<arrayOfStuff>2</arrayOfStuff>
	<arrayOfStuff>3</arrayOfStuff>
	<arrayOfStuff>
		<foo>bar</foo>
		<baz>bux</baz>
	</arrayOfStuff>
	<arrayOfStuff>
		<list>1</list>
		<list>2</list>
		<list>3</list>
	</arrayOfStuff>
</root>

Oxygen gives you a bit of both:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<arrayOfStuff>
	<array>1</array>
	<array>2</array>
	<array>3</array>
</arrayOfStuff>
<arrayOfStuff>
	<foo>bar</foo>
	<baz>bux</baz>
</arrayOfStuff>
<arrayOfStuff>
	<list>1</list>
	<list>2</list>
	<list>3</list>
</arrayOfStuff>

None of the above is the "best" way of doing it - it probably just depends on what you need on the XML side. I guess what I don't like about the Oxygen conversion is that it's use of the 'array' element can depend on the number of items in the array (one or many, as in my 'contributor' example above) - something about that just doesn't seem right.

As an Oxygen user, what I would like to see going forward given the lack of standards would be multiple options for this conversion. It's obvious there is no "right way" - just different ways that fit different needs. Specifically, the following would be on my wish list:
- One option for a straight-up XSLT 3.0 json-to-xml() conversion to map/array format (one of the few standards out there it seems)
- Options for one or more of the other more commonly used outputs (with/without map/array keyword)
- Options to name the root and map/array keyword being used

"More options" would apply to the XML > JSON conversions as well for the same reasons. I'm amazed there hasn't been much standardization in this area (at least there are some named conventions for XML>JSON). Overall, I'm very grateful for the JSON functionality Oxygen provides now. Its great you recognize how XML/JSON can be two sides of the same coin in many contexts - even if they have fundamental differences. Looking forward to seeing what comes next!

Thanks, Scott

[teo]

Hi Scott,

JSON to XML conversion based on a set of options like those proposed by you above was discussed within our team too at some point.
Therefore your comments on this topic will cause us to re-examine this approach.
Thanks for your suggestions, they will definitely be discussed as I will add them to an additional improvement task from the suite of those we have already registered.