Has anyone used Git submodules with the Git plugin?

Having trouble installing <oXygen/>? Got a bug to report? Post it all here.
chrispitude
Posts: 322
Joined: Thu May 02, 2019 2:32 pm

Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Mon May 11, 2020 11:56 pm

Hi all,

I tried to use the Git submodule feature of the Git plugin today, but didn't work as I expected.

When I switch to a submodule, the plugin shows every file in the submodule as having an uncommited local change:
image.png
image.png (9.86 KiB) Viewed 1651 times
If I select all files and Discard them, the list becomes empty - but only for several seconds, then all files are listed as changed again.

If I pick any file and choose "Open in compare editor", the diff windows always shows zero changes.

Because all files are seen as having uncommitted local changes, I can't pull any new updates down from the submodule's remote repository:
image.png
image.png (9.36 KiB) Viewed 1651 times
Is anyone using Git submodules in their projects?

alex_jitianu
Posts: 818
Joined: Wed Nov 16, 2005 11:11 am

Re: Has anyone used Git submodules with the Git plugin?

Post by alex_jitianu » Tue May 12, 2020 6:08 pm

Hi Chris,

I haven't encountered this situation, myself. What you describe, might be a line separator issue. You can open one of those files that appear as modified in an Oxygen editor and then go to Tools->Hex Viewer. If you see 0D 0A sequence then it might be something related with the core.autocrlf setting. You can check to see if this setting is set:

Code: Select all

git config --get core.autocrlf
and try to disable it to see if anything changes. Perhaps it is better to do it it on that specific project in order not to affect the global setting:

Code: Select all

git config --local core.autocrlf false


Best regards,
Alex

chrispitude
Posts: 322
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Fri May 15, 2020 6:34 pm

crlf-test.tgz
(876 Bytes) Downloaded 52 times
Hi Alex,

So you have pointed me down the right path, and it looks like it's not related specifically to submodules at all.

We have a mixed environment with multiple Git clients:
  • Oxygen for Windows Git client
  • Git for WIndows
  • Git command line in Ubuntu linux
I created LF and CRLF versions of some DITA files:

Code: Select all

crlf-test$ ls -l */*
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:16 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:16 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 216 May 15 11:16 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 452 May 15 11:16 unix/topic_1.dita
I used them to upload an Oxygen test project (attached) on our Git server.
crlf-test.tgz
(876 Bytes) Downloaded 52 times
If I clone this repo in a linux shell, the initial checkout preserves the original format and nothing is shown as modified:

Code: Select all

$ git clone git@gitsnps.internal.synopsys.com:chrispy/crlf-test.git
Cloning into 'crlf-test'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 9 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Resolving deltas: 100% (2/2), done.
$ cd crlf-test/
$ ls -l */*
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:32 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:32 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 216 May 15 11:32 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 452 May 15 11:32 unix/topic_1.dita
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
$
But when I use the Oxygen Git plugin to clone this project on my Windows laptop, the files in the dos directory are immediately listed as modified, despite the fact that I haven't changed anything at all:
image.png
image.png (4.88 KiB) Viewed 1599 times
What's strange is that (1) those files are in dos (CRLF) format so I don't understand why it's complaining, and (2) the Git plugin converted the unix files to dos upon the initial commit:

Code: Select all

-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:25 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:25 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:25 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:25 unix/topic_1.dita
If Oxygen's Git plugin wants to convert everything to CRLF format upon checkout, why would it list the files in the dos directory as modified?

And why would it immediately list files as modified after cloning a repo, no matter what they are? That seems internally inconsistent to me.

chrispitude
Posts: 322
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Fri May 15, 2020 8:12 pm

Here's another strange finding - if I discard the files, they come back listed as "modified" when I switch focus away from Oxygen and switch back!
crlf.gif
crlf.gif (249.38 KiB) Viewed 1576 times

chrispitude
Posts: 322
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Sun May 17, 2020 6:54 pm

Okay, I think I figured it out.

On my Windows 10 machine, I have Oxygen for Windows installed, and I have Ubuntu linux installed via WSL (Windows Subsystem for Linux).
  • By default, Oxygen's Git plugin converts text files to CRLF upon checkout.
  • In that same repo directory, if I commit a CRLF file via WSL, I get the bad behavior.
Rather than look at solutions using the .git/config file, I decided to explore the .gitattributes file because it allows more detailed configuration of the files in the repo. This is what I'm testing now:

Code: Select all

# For files with no extensions, let Git detect if they're binary
* text=auto

# By default, all files with extensions are converted to LF only, so Windows does not break them
*.* text eol=lf

# Override the following filetypes to be binary
*.[gG][iI][fF] binary
*.[jJ][pP]*[gG] binary
*.[pP][nN][gG] binary
*.[eE][xX][eE] binary
*.[pP][dD][fF] binary
*.[sS][nN][aA][gG]* binary
*.[vV][sS][dD]* binary
I chose to configure binary file extensions explicitly and let everything else fall back to ASCII, but I could have done it the other way around, or I could list both ASCII and binary extensions explicitly.

There are two places this file can be placed:
  • <repo>/.git/info/attributes - affects only your local repo, nobody else
  • <repo>/.gitattributes - stored in repo, affects everyone
For now, I put it in .git/info/ to see if it breaks anything in my daily use. When I trust it, I can roll it out to the wider team.

Placing the attributes file in the repo affects future file pulls, but does not alter the current files on-disk. I was able to test the settings on the existing files by doing a "Remove from Disk" in Oxygen, then doing a "Discard" of all the file deletions in the Git plugin, which caused the files to get pulled in LF form.

Now that I have a solution to this problem, I'll go back and update my Git submodule repo accordingly and see how it goes!

chrispitude
Posts: 322
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Mon May 18, 2020 2:35 pm

Hi Alex,

I did some experimenting with Git submodules through the Oxygen Git plugin. The basic functionality is there, but some of our writers are nontechnical and anything beyond a pushbutton interface will invite confusion.

Accordingly, I have the following enhancement requests:


1. When you clone a repo in the plugin (the "+" button), Oxygen should populate submodules too. This can be done in two separate steps:

Code: Select all

# clone repo, THEN clone submodules -- in two separate steps
git clone <repo>
git submodule update --init --recursive
or a single step:

Code: Select all

# clone repo, plus its submodules
git clone <repo> --recurse-submodules

2. Submodule updates should be easier (or automatic). Currently when a submodule needs an update, the submodule root shows up in the modified files list:
image.png
image.png (8 KiB) Viewed 1516 times
This can be confusing because the submodule was changed externally, unlike other files in the list that were changed locally. In addition, the mechanics of switching to the submodule, updating, and switching back are intimidating for someone nontechnical.


I see the value in allowing explicit recursion into the submodule, for cases where multiple people are working on the submodule contents, there might be conflicts, you might need to perform conflict resolution, etc. But for the simple case where shared files are modified by one administrator and rolled out read-only to a team (DITAVAL files, CSS files, etc.), a pushbutton update would make this much easier.

Perhaps like our other discussions on auto-updating, we could provide more automation for submodule updates that have no conflicts?

alex_jitianu
Posts: 818
Joined: Wed Nov 16, 2005 11:11 am

Re: Has anyone used Git submodules with the Git plugin?

Post by alex_jitianu » Wed May 20, 2020 9:53 am

Hi Chris,

1. I agree, chances are that, for our audience, initializing the submodules automatically is better. Otherwise, you currently get an empty directory in your file system and you don't know what to do. I've added an issue to add an option for this, with the default being to automatically initialize them.

2. Yes, working with submodules can be a challenge.
Currently when a submodule needs an update, the submodule root shows up in the modified files list:
As far as I know, the submodule appears as modified after the user entered it and run git pull (or similar), right? I'll add an issue to see how we can make things easier.

jhanna
Posts: 18
Joined: Mon Aug 17, 2020 6:29 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by jhanna » Wed Sep 16, 2020 12:35 am

Please add my vote for implementing changes that would make working with submodules easier. :D

On a related note, has any consideration been given to creating a tool for working with subtrees? From what I have seen, subtrees are easier to work with than submodules for pulling updates.

Regards,
Jonathan

sorin_carbunaru
Posts: 363
Joined: Mon May 09, 2016 9:37 am

Re: Has anyone used Git submodules with the Git plugin?

Post by sorin_carbunaru » Wed Sep 16, 2020 11:18 am

Hi Jonathan,

I added your vote to the issues related to working with submodules. I also created a new feature request for adding support for subtrees. By the way, I would say that perhaps adding support for subtrees has a higher priority than improving submodules. What do you think?

@Chris, do you have anything to say about subtrees? Would you prefer them instead of submodules?

Best wishes,
Sorin Carbunaru
Oxygen XML Editor

jhanna
Posts: 18
Joined: Mon Aug 17, 2020 6:29 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by jhanna » Wed Sep 16, 2020 6:37 pm

Hi Sorin,

Since the submodule feature already works, I would agree that adding support for subtrees would take priority over submodule improvements.

Regards,
Jonathan

sorin_carbunaru
Posts: 363
Joined: Mon May 09, 2016 9:37 am

Re: Has anyone used Git submodules with the Git plugin?

Post by sorin_carbunaru » Tue Mar 23, 2021 10:41 am

Hi all,

Just wanted to say that we released version 2.5.0 of the Git Client, and this version received some improvements on submodules. You can find them in the release notes from https://github.com/oxygenxml/oxygen-git ... /tag/2.5.0.

Best wishes,
Sorin Carbunaru
Oxygen XML Editor

jhanna
Posts: 18
Joined: Mon Aug 17, 2020 6:29 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by jhanna » Tue Mar 23, 2021 7:27 pm

Hi Sorin,

Thank you for the update and for continuing to add improvements for working with submodules! I am particularly intrigued by the following improvement that was added:
When pulling the remote changes for a repository that contains submodules, the submodules are updated as well (by default). This behavior depends on the Update all submodules after pulling changes from the remote repository option from the Git Client preferences page in Oxygen.
How exactly is this supposed to work? Could you provide an example?

I tried testing this feature out by making a change to a remote repository, then pulling the changes. However, the submodules did not update. I still had to manually open the submodule (using the Submodules button in the Git Staging plugin) to update the submodule. I double-checked the preferences and the "update all submodules" option was selected.

Interestingly, I noticed that when I opened the submodule, it was in a detached HEAD state, pointing to a specific commit. I previously had the submodule pointing at the main branch. It switched to the detached HEAD state automatically after I pulled changes to the parent repository.

Best regards,
Jonathan

chrispitude
Posts: 322
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Wed Mar 24, 2021 3:57 am

Thanks Sorin, I can't wait to try the improvements!

Sorry I didn't see your question about subtrees until now. For our purposes, I think submodules are more suitable. We have three Git repos of DITA documentation. Each of them has a prj/ directory that contains all our Oxygen configuration data - frameworks, DITA plugins, external tools, and all that fun stuff. I want to create a Git repo for this prj/ stuff, then instantiate it (as a submodule) in each of the repos. This way, I can make Oxygen environment changes in one place, then roll them out to all three teams. I believe submodules are the best fit for this.

alex_jitianu
Posts: 818
Joined: Wed Nov 16, 2005 11:11 am

Re: Has anyone used Git submodules with the Git plugin?

Post by alex_jitianu » Wed Mar 24, 2021 4:57 pm

Hi Jonathan,
How exactly is this supposed to work? Could you provide an example?
First of all, the option Update all submodules after pulling changes must be set. The scenario is as follows. Lets assume we have a main repository with a second submodule:
1. User1 open main in the Git Staginf view. It uses the Submodules action on the toolbar and goes into the second submodule
2. User1 moves the submodule to a newer commit. In Oxygen it has to move from DETACHED HEAD to a brach, pull changes and maybe use the History view to switch to another commit
3. User1 returns to main and commits/pushes the submodule binding
4. User2 also works in the main repository. When it Pulls the change we also run a git submodule update. Depending on the module update behavior, its second submodule might jump to the DETACHED HEAD state, on the commit bound by User1.

Best regards,
Alex

jhanna
Posts: 18
Joined: Mon Aug 17, 2020 6:29 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by jhanna » Sun Mar 28, 2021 1:39 am

alex_jitianu wrote:
Wed Mar 24, 2021 4:57 pm

First of all, the option Update all submodules after pulling changes must be set. The scenario is as follows. Lets assume we have a main repository with a second submodule:
1. User1 open main in the Git Staginf view. It uses the Submodules action on the toolbar and goes into the second submodule
2. User1 moves the submodule to a newer commit. In Oxygen it has to move from DETACHED HEAD to a brach, pull changes and maybe use the History view to switch to another commit
3. User1 returns to main and commits/pushes the submodule binding
4. User2 also works in the main repository. When it Pulls the change we also run a git submodule update. Depending on the module update behavior, its second submodule might jump to the DETACHED HEAD state, on the commit bound by User1.
Hi Alex,

Thank you for the example. Unfortunately, I still cannot get the submodules to automatically update when pulling changes to the parent repository. Either I am doing something wrong, I am not understanding how this function is supposed to work, or there is a bug.

If I understand the Stack Overflow article you provided, I am supposed to add update = merge to the submodule in the .gitmodules file. I tried doing this and it appears to not have any effect. When I open the submodule, I still see it in a detached HEAD state, even though I previously checked out the main branch. I never experienced this issue in previous versions of the git plugin, which is why I am wondering if there is a bug. Another item to note, in previous versions of the git plugin, I would get a notification that the submodule was in a detached HEAD state. I no longer get these notifications in the current version of the plugin.

Would it be possible for you to provide a video demonstration on how this feature is supposed to work?

Best regards,
Jonathan

alex_jitianu
Posts: 818
Joined: Wed Nov 16, 2005 11:11 am

Re: Has anyone used Git submodules with the Git plugin?

Post by alex_jitianu » Tue Mar 30, 2021 1:23 pm

Hi Jonathan,

I, too, see a difference beween our Git client and the command line git client. I've setup a .gitmodule file in the main repository like this:

Code: Select all

[submodule "git_test"]
	path = git_test
	url = https://github.com/AlexJitianu/git_test.git
	branch = master
	update = merge
A git submodule update leaves the submodule in its current state, on a local branch. Meanwhile, Oxygen will switch the module to the commit recorded in the main repository. I'll add an issue to investigate this different behavior. You can disable Update all submodules after pulling changes from the remote repository in preferences until we understand why it behaves differently.
Another item to note, in previous versions of the git plugin, I would get a notification that the submodule was in a detached HEAD state. I no longer get these notifications in the current version of the plugin.
We removed that warning because a submodule is many time by design in a detached HEAD state. This is what you do if the main repo and the module have different release cycles and you want to control what content gets pulled in the main repo.


Best regards,
Alex

jhanna
Posts: 18
Joined: Mon Aug 17, 2020 6:29 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by jhanna » Wed Mar 31, 2021 7:20 pm

Hi Alex,

Thank you for the additional information. I will disable the Update Submodules feature for now. Thank you again for all the work you are putting in to support submodules!

Best regards,
Jonathan

chrispitude
Posts: 322
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Thu Apr 01, 2021 6:33 pm

alex_jitianu wrote:
Tue Mar 30, 2021 1:23 pm
We removed that warning because a submodule is many time by design in a detached HEAD state. This is what you do if the main repo and the module have different release cycles and you want to control what content gets pulled in the main repo.
Hi Alex,

I haven't had a chance to dig into submodules yet (busy with my "day job" as technical writer these days), but I wanted to note that this will likely be the case for us. Our submodule will contain all our Oxygen project/framework stuff. That submodule repo will accumulate fixes/enhancements over time, but we will control how we roll these out to the full Oxygen/DITA repo environment.

Agreed with Jonathan - thank you for all the work you are putting into submodules, and to Git support in general!

jhanna
Posts: 18
Joined: Mon Aug 17, 2020 6:29 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by jhanna » Wed Apr 21, 2021 10:36 pm

A useful feature to add would be the ability to update all submodules with a single click of a button. Currently, updating submodules is a bit cumbersome, especially if a repository contains several submodules. It is also a bit confusing since it is not a straightforward process to change the working copy back to the main repository after checking out a submodule.

Basically what I am looking for is a button that will run the command:

Code: Select all

git submodule update --recursive --remote
After that, users can stage and commit the changes as they usually would.

Would this be possible or is there a better solution?

alex_jitianu
Posts: 818
Joined: Wed Nov 16, 2005 11:11 am

Re: Has anyone used Git submodules with the Git plugin?

Post by alex_jitianu » Thu Apr 22, 2021 10:59 am

Hello,

We had a prior request for such a feature, but we didn't get the chance to implement it yet. I will add your vote for it and increase its priority. When we manage to do it and a release it, we will notify you on this thread.
Right now, I'm afraid that there is not an alternative to opening each project and performing a pull.

Best regards,
Alex

Post Reply