Would it be possible/practical to reliably extend the DITA Maps Manager to allow users to easily create these book files from within Oxygen?
No, the DITA Maps Manager is pretty much DITA oriented, we have API to allow adding various toolbar and menu actions to it but it's internal model knows only DITA-based models.
Our Java-based SDK allows you to build plugins which can contribute custom views to Oxygen, views in which you can create your own book navigator trees.
It would be especially helpful if we could configure custom transformation scenarios for these open books.
This can be done, you can create a framework customization for any type of XML:https://www.oxygenxml.com/doc/versions/18.1/ug-editor/topics/author-devel-guide-intro.html
The framework customization can contribute custom transformation scenarios of type XSLT or ANT:https://www.oxygenxml.com/doc/versions/18.1/ug-editor/tasks/dg-transformation-scenarios.html
You can create for example ANT transformation scenarios which in their turn can do mostly anything, they can call scripts with various parameters and so on.
We try to make Oxygen as plug and play as possible, so all the support we have for editing DITA topics in the main editor area can be created as a customization for any XML vocabulary. But you need to create this yourselves, we can help with advice along the way though.
Also custom vocabulary views like the DITA Maps Manager are views in which we invested a lot of time and effort. We are about 3-4 people working constantly on DITA related improvements in Oxygen.
So that's the problem with using another XML vocabulary, you need to invest time to build customizations and your customizations will probably never be as good as the ones we have for DITA but they may be good enough to cover your work flow.