To debug this transformation you need to apply the stylesheet (xsdDocHtml.xsl) on an intermediate file resulted during the custom XML Schema Documentation generation.
To obtain the intermediate file, in the XML Schema Documentation dialog select Custom format and then the Options button.
Here in the 'Custom format options' dialog choose xsdDocHtml.xsl stylesheet in the Custom XSL field and make sure you clear 'Delete intermediate XML files' option and check 'Copy additional resources to the output folder'.
This way you obtain a temporary file (name_of_the_XSD_file.tmp) that should be opened in oXygen as an XML file and on it apply the transformation scenario that you can further debug. Make sure in the scenario configuration you choose Saxon-PE/Saxon-EE as Transformer and in the Parameters tab insert as mainFile the name of the temporary file and tmp for intermediateXmlExtension parameter.
I hope this helps.
oXygen XML Editor and Author Support