Has anyone used Git submodules with the Git plugin?

Having trouble installing <oXygen/>? Got a bug to report? Post it all here.
chrispitude
Posts: 201
Joined: Thu May 02, 2019 2:32 pm

Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Mon May 11, 2020 11:56 pm

Hi all,

I tried to use the Git submodule feature of the Git plugin today, but didn't work as I expected.

When I switch to a submodule, the plugin shows every file in the submodule as having an uncommited local change:
image.png
image.png (9.86 KiB) Viewed 355 times
If I select all files and Discard them, the list becomes empty - but only for several seconds, then all files are listed as changed again.

If I pick any file and choose "Open in compare editor", the diff windows always shows zero changes.

Because all files are seen as having uncommitted local changes, I can't pull any new updates down from the submodule's remote repository:
image.png
image.png (9.36 KiB) Viewed 355 times
Is anyone using Git submodules in their projects?

alex_jitianu
Posts: 770
Joined: Wed Nov 16, 2005 11:11 am

Re: Has anyone used Git submodules with the Git plugin?

Post by alex_jitianu » Tue May 12, 2020 6:08 pm

Hi Chris,

I haven't encountered this situation, myself. What you describe, might be a line separator issue. You can open one of those files that appear as modified in an Oxygen editor and then go to Tools->Hex Viewer. If you see 0D 0A sequence then it might be something related with the core.autocrlf setting. You can check to see if this setting is set:

Code: Select all

git config --get core.autocrlf
and try to disable it to see if anything changes. Perhaps it is better to do it it on that specific project in order not to affect the global setting:

Code: Select all

git config --local core.autocrlf false


Best regards,
Alex

chrispitude
Posts: 201
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Fri May 15, 2020 6:34 pm

crlf-test.tgz
(876 Bytes) Downloaded 13 times
Hi Alex,

So you have pointed me down the right path, and it looks like it's not related specifically to submodules at all.

We have a mixed environment with multiple Git clients:
  • Oxygen for Windows Git client
  • Git for WIndows
  • Git command line in Ubuntu linux
I created LF and CRLF versions of some DITA files:

Code: Select all

crlf-test$ ls -l */*
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:16 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:16 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 216 May 15 11:16 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 452 May 15 11:16 unix/topic_1.dita
I used them to upload an Oxygen test project (attached) on our Git server.
crlf-test.tgz
(876 Bytes) Downloaded 13 times
If I clone this repo in a linux shell, the initial checkout preserves the original format and nothing is shown as modified:

Code: Select all

$ git clone git@gitsnps.internal.synopsys.com:chrispy/crlf-test.git
Cloning into 'crlf-test'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 9 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Resolving deltas: 100% (2/2), done.
$ cd crlf-test/
$ ls -l */*
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:32 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:32 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 216 May 15 11:32 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 452 May 15 11:32 unix/topic_1.dita
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
$
But when I use the Oxygen Git plugin to clone this project on my Windows laptop, the files in the dos directory are immediately listed as modified, despite the fact that I haven't changed anything at all:
image.png
image.png (4.88 KiB) Viewed 303 times
What's strange is that (1) those files are in dos (CRLF) format so I don't understand why it's complaining, and (2) the Git plugin converted the unix files to dos upon the initial commit:

Code: Select all

-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:25 dos/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:25 dos/topic_1.dita
-rwxrwxrwx 1 chrispy chrispy 222 May 15 11:25 unix/map.ditamap
-rwxrwxrwx 1 chrispy chrispy 464 May 15 11:25 unix/topic_1.dita
If Oxygen's Git plugin wants to convert everything to CRLF format upon checkout, why would it list the files in the dos directory as modified?

And why would it immediately list files as modified after cloning a repo, no matter what they are? That seems internally inconsistent to me.

chrispitude
Posts: 201
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Fri May 15, 2020 8:12 pm

Here's another strange finding - if I discard the files, they come back listed as "modified" when I switch focus away from Oxygen and switch back!
crlf.gif
crlf.gif (249.38 KiB) Viewed 280 times

chrispitude
Posts: 201
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Sun May 17, 2020 6:54 pm

Okay, I think I figured it out.

On my Windows 10 machine, I have Oxygen for Windows installed, and I have Ubuntu linux installed via WSL (Windows Subsystem for Linux).
  • By default, Oxygen's Git plugin converts text files to CRLF upon checkout.
  • In that same repo directory, if I commit a CRLF file via WSL, I get the bad behavior.
Rather than look at solutions using the .git/config file, I decided to explore the .gitattributes file because it allows more detailed configuration of the files in the repo. This is what I'm testing now:

Code: Select all

# For files with no extensions, let Git detect if they're binary
* text=auto

# By default, all files with extensions are converted to LF only, so Windows does not break them
*.* text eol=lf

# Override the following filetypes to be binary
*.[gG][iI][fF] binary
*.[jJ][pP]*[gG] binary
*.[pP][nN][gG] binary
*.[eE][xX][eE] binary
*.[pP][dD][fF] binary
*.[sS][nN][aA][gG]* binary
*.[vV][sS][dD]* binary
I chose to configure binary file extensions explicitly and let everything else fall back to ASCII, but I could have done it the other way around, or I could list both ASCII and binary extensions explicitly.

There are two places this file can be placed:
  • <repo>/.git/info/attributes - affects only your local repo, nobody else
  • <repo>/.gitattributes - stored in repo, affects everyone
For now, I put it in .git/info/ to see if it breaks anything in my daily use. When I trust it, I can roll it out to the wider team.

Placing the attributes file in the repo affects future file pulls, but does not alter the current files on-disk. I was able to test the settings on the existing files by doing a "Remove from Disk" in Oxygen, then doing a "Discard" of all the file deletions in the Git plugin, which caused the files to get pulled in LF form.

Now that I have a solution to this problem, I'll go back and update my Git submodule repo accordingly and see how it goes!

chrispitude
Posts: 201
Joined: Thu May 02, 2019 2:32 pm

Re: Has anyone used Git submodules with the Git plugin?

Post by chrispitude » Mon May 18, 2020 2:35 pm

Hi Alex,

I did some experimenting with Git submodules through the Oxygen Git plugin. The basic functionality is there, but some of our writers are nontechnical and anything beyond a pushbutton interface will invite confusion.

Accordingly, I have the following enhancement requests:


1. When you clone a repo in the plugin (the "+" button), Oxygen should populate submodules too. This can be done in two separate steps:

Code: Select all

# clone repo, THEN clone submodules -- in two separate steps
git clone <repo>
git submodule update --init --recursive
or a single step:

Code: Select all

# clone repo, plus its submodules
git clone <repo> --recurse-submodules

2. Submodule updates should be easier (or automatic). Currently when a submodule needs an update, the submodule root shows up in the modified files list:
image.png
image.png (8 KiB) Viewed 220 times
This can be confusing because the submodule was changed externally, unlike other files in the list that were changed locally. In addition, the mechanics of switching to the submodule, updating, and switching back are intimidating for someone nontechnical.


I see the value in allowing explicit recursion into the submodule, for cases where multiple people are working on the submodule contents, there might be conflicts, you might need to perform conflict resolution, etc. But for the simple case where shared files are modified by one administrator and rolled out read-only to a team (DITAVAL files, CSS files, etc.), a pushbutton update would make this much easier.

Perhaps like our other discussions on auto-updating, we could provide more automation for submodule updates that have no conflicts?

alex_jitianu
Posts: 770
Joined: Wed Nov 16, 2005 11:11 am

Re: Has anyone used Git submodules with the Git plugin?

Post by alex_jitianu » Wed May 20, 2020 9:53 am

Hi Chris,

1. I agree, chances are that, for our audience, initializing the submodules automatically is better. Otherwise, you currently get an empty directory in your file system and you don't know what to do. I've added an issue to add an option for this, with the default being to automatically initialize them.

2. Yes, working with submodules can be a challenge.
Currently when a submodule needs an update, the submodule root shows up in the modified files list:
As far as I know, the submodule appears as modified after the user entered it and run git pull (or similar), right? I'll add an issue to see how we can make things easier.

Post Reply