Solved Help with migration to git

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Greetings all,

I have been using svn to download sources to /usr/ports. At this time, it is still working, but I would like to try transitioning to git in time, so I do not encounter a surprise when least needed.

I have found the document FreeBSD mini-git Primer, but it could as well be written in Swahili, so I would like a review of my understanding.

First I need to pull the /usr/ports tree. There is a choice between a deep clone and shallow clone. The deep clone "pulls in the entire tree, as well as all the history and branches". So regarding ports, it is latest or quarterly. Since I do follow latest and do not need quarterly, I thought of using the shallow clone: % git clone -o freebsd -b latest --depth 1 $URL /usr/ports. Afterwards, the tree is updated by: % git pull --ff-only.

Since I already have a tree in /usr/ports, can I just use the git pull command to get the revisions since last update, or do I need to delete the ports tree and start afresh?

The warning regarding the shallow clone ". . . you will lose the commit count in your uname output. This can make it more difficult to determine if your system needs to be updated when a security advisory is issued", does not appear to apply to ports.

Thank you for any help.

Kindest regards,

M
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 13,146
Messages: 39,756

I would suggest giving net/gitup a go. It's easy to use and specifically made for people with zero experience with git.

Since I already have a tree in /usr/ports, can I just use the git pull command to get the revisions since last update, or do I need to delete the ports tree and start afresh?
Delete and start over.
 

Zirias

Son of Beastie

Reaction score: 1,787
Messages: 3,052

First I need to pull the /usr/ports tree.
Last time I checked, ports weren't migrated to git yet.
The warning regarding the shallow clone ". . . you will lose the commit count in your uname output. This can make it more difficult to determine if your system needs to be updated when a security advisory is issued", does not appear to apply to ports.
Yes, this is for src. Also, if you intend to do any development yourself, I'd suggest a full clone. It really isn't THAT large.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Hi SirDice,

thank you for the recommendation, I would prefer to stay with git proper, one - I understand that there is only one developer for net/gitup; two - a new knowledge is a good think.

Hi Zirias,

thank you for the answer. I followed the first link in he cited article, and it shows a repository. The latter ones are, indeed empty. So, I will wait.

I gather that since you did not comment on the proposed commands, they are fine.

Kindest regards,

M
 

Jose

Daemon

Reaction score: 1,168
Messages: 1,383

It looks like Git for ports is still in beta. I'm using it and it works for me:

Code:
git clone -o freebsd -b latest --depth 1 $URL /usr/ports
The master branch is called "main", and will be checked out automatically if you don't specify -b. There is no branch named "latest":

Code:
git pull --ff-only
What's the point of --ff-only? Are you worried about local changes?
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Hi Jose,

thank you for the reply.

Regarding the ports "branches", since the Primer was not entirely clear to me; the ports section is very terse, just referring to src section. Thus, I did a search on term "freebsd branches ports" and the first link https://wiki.freebsd.org/Ports/QuarterlyBranch#Switching_from_quarterly_to_latest, refers to the Quarterly as a branch, as actually does your second link Indeed, the Primer states that without the -b, the main/master branch is downloaded. But, it is completely silent regarding your assertion that main is latest.

Regarding the --ff-only, the Primer recites (emphasis supplied):

will update the tree. In git, a 'fast forward' merge is one that only needs to set a new branch pointer and doesn't need to re-create the commits. By always doing a 'fast forward' merge/pull, you'll ensure that you have an identical copy of the FreeBSD tree. This will be important if you want to maintain local patches.

So, by implication the Primer suggests that if one does not use --ff-only, one will have different (antonym to identical) branches.

I do understand that the Primer is a draft, so hopefully it will be made clearer in a final version.

Kindest regards,

M
 

Jose

Daemon

Reaction score: 1,168
Messages: 1,383

Regarding the ports "branches", since the Primer was not entirely clear to me; the ports section is very terse, just referring to src section. Thus, I did a search on term "freebsd branches ports" and the first link https://wiki.freebsd.org/Ports/QuarterlyBranch#Switching_from_quarterly_to_latest, refers to the Quarterly as a branch, as actually does your second link...
That page is really not very clear. It says that "quarterly" is the name for the group of all quarterly branches, which is not terribly informative. Later on it describes the naming convention for individual quarterly branches which is
Branches are named according to the year (YYYY) and quarter (Q1-4) they are created in.
The current quarterly branch is called "branches/2021Q1", for example. That's not a very Git-like name. We're carrying the Subversion branching naming convention, apparently.

Indeed, the Primer states that without the -b, the main/master branch is downloaded. But, it is completely silent regarding your assertion that main is latest.
That is the convention, but you're right, it's not explicitly stated. You can verify that this is the case yourself by comparing the history of the main branch to the history of the current quarterly branch.

Regarding the --ff-only, the Primer recites (emphasis supplied):
Yeah, and right after your emphasis it reads "(t)his will be important if you want to maintain local patches" which is why I asked about local changes.

So, by implication the Primer suggests that if one does not use --ff-only, one will have different (antonym to identical) branches.
True, if and only if you have local changes. If you don't that flag does either nothing, or silences a Git warning, depending on your version of Git.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Hi Jose,

thank you again for the detail explanation. It is becoming clear now. the remaining thing I do not still understand is the use of the --ff-only. So, if I do not use it, it pulls the same copy as with it? If so, what is its purpose?

Kindest regards,

M
 

Jose

Daemon

Reaction score: 1,168
Messages: 1,383

...the remaining thing I do not still understand is the use of the --ff-only. So, if I do not use it, it pulls the same copy as with it? If so, what is its purpose?
OK, but this is long.

By default a git-pull is actually two operations, a git-fetch followed by a git-merge.

The fetch operation downloads changes from one or more remotes from one or more branches. Yes, that's vague because Git is very flexible. By default it downloads changes from the remote called "origin" for any local tracking branches you have. The latter is usually a single branch called "master". All of the preceding is configurable. Witness that the master branch in all Freebsd repos is called "main" instead of "master". Key takeaway is that a fetch will download all the changes you're likely to be interested in, but doesn't do anything else with them.

Now I have these changes, but I don't have any branches that have references into them, and looking at them would be awkward at best. Enter the second stage of pull, the merge. Pull automatically merges upstream changes into the branch you are currently working on. This is also called the "checked-out" branch.

The problem is that Git merges changes in the order that they happened. Imagine that you added your local patches a week ago. There have been 300 changes to the branch you're working on since then. Git will dutifully merge these for you, and the next time you do a git-log you'll think your changes are gone because they're buried under all the stuff that happened on the upstream branch after your commits. Worse than that, your local branch will now have a merge commit that does not exist in the upstream branch. This means all the commit hashes in your branch after the merge will be different from those in the upstream branch. Messy.

How does "ff-only" help with this? That option means "only make fast-forward merges when pulling". A fast-forward merge happens when the upstream branch has not changed after you made your commits. In this case, Git just moves the HEAD reference for your branch to the hash for your last commit. If your branch can't be fast-forwarded, git-pull "refuse(s) to merge and exit(s) with a non-zero status."

The default merge behavior has caused trouble for more than a few so a warning was added in Git 2.27 to try and make you aware of the pitfalls.

This is already very long so I won't get into the different behaviors you can configure for git-pull and under which circumstances they are appropriate unless you ask.
 

PMc

Daemon

Reaction score: 694
Messages: 1,394

It looks like Git for ports is still in beta. I'm using it and it works for me:
My last impression was that ports would move to git for 2021Q2. So while for src it would only become relevant when upgrading to R.13, for ports it would be relevant next month.
Your observation does not have it appear likely this is going to happen.


Code:
git clone -o freebsd -b latest --depth 1 $URL /usr/ports
The master branch is called "main", and will be checked out automatically if you don't specify -b. There is no branch named "latest":

Alright, lets see. git clone https://cgit-beta.freebsd.org/ports.git .
This one seems to do something. I can already smell it...

Aha. Now I have a repo clone and a checked out source.

Now, patching the source: I can
* just modify the source - and git nicely keeps track on the changes
* use git-add - for what purpose?
* use git-commit - for what purpose?

Then I definitely need to keep the repo clown actual. That seems to work with git-pull, but it asks for one of three options:
* merge
* rebase
* fast-forward

Then, as the repo clown should contain all the quarterlies (and they are visible with git-branch --all), but is 1:1 married with the checkout, I cannot just create a new checkout at some other place for looking into some branch. Or can I?
It seems I can switch the checkout with some git-checkout branches/2018Q3 - but that does sometimes change the source and sometimes does nothing. And it is unclear what it does to the local changes.

Then, I currently keep my local changes stored at a separate place and just patch/merge them into the source after every update or switch. It seems I can continue to do so, by reverting and re-applying the patches on demand - as this allows to have different patchsets for different machines.

It seems git would prefer to put the local patches within the repo. This would need different branches for different machines. Then, when quarterly gets switched, the delta between 2021Q1-public and 2021Q1-local-desktop1 needs to be grabbed and merged into 2021Q2-public in order to obtain 2021Q2-local-desktop1... hm...
 

Zirias

Son of Beastie

Reaction score: 1,787
Messages: 3,052

Your observation does not have it appear likely this is going to happen.

And as for git usage: You can create as many clones as you like, but why would you, just for looking at a different branch? Just keep your changes in your own local branch. And then, have a look at rebase.
 

Jose

Daemon

Reaction score: 1,168
Messages: 1,383

My last impression was that ports would move to git for 2021Q2. So while for src it would only become relevant when upgrading to R.13, for ports it would be relevant next month.
Your observation does not have it appear likely this is going to happen.
There was just a posting to freebsd-ports that reports the change should happen "in a couple of weeks." That lines up pretty well with the cutting of the 2021Q2 branch.
Now, patching the source: I can
* just modify the source - and git nicely keeps track on the changes
Yes. Git-status will show you the files that have changed in your working copy. Git-diff will show you the actual changes.
* use git-add - for what purpose?
To stage changes for a commit.
* use git-commit - for what purpose?
So that you can switch to other branches. Also, changes in your working tree exist only in the clone directory on your machine, and cannot be pushed to a remote repo.
Then I definitely need to keep the repo clown
:)
actual. That seems to work with git-pull, but it asks for one of three options:
* merge
Explained above.
This will set aside your changes, fast-forward the branch to the remote HEAD, and then re-apply your changes. You'll have to resolve any conflicts caused by this re-application.
* fast-forward
Special case of merge*. Also covered above.
Then, as the repo clown should contain all the quarterlies (and they are visible with git-branch --all), but is 1:1 married with the checkout, I cannot just create a new checkout at some other place for looking into some branch. Or can I?
Yes, you can. This will create a local tracking branch for the current quarterly branch:
Code:
git switch -t -c 2021Q1 origin/branches/2021Q1
It seems I can switch the checkout with some git-checkout branches/2018Q3 - but that does sometimes change the source and sometimes does nothing. And it is unclear what it does to the local changes.
I prefer to use git-switch because "checkout" means too many different things in Git.

What you propose will create a local branch called "branches/2018Q3" that is unrelated to the upstream branch called "branches/2018Q3". It could be completely different, too. If you issue that command while "main" is the checked-out branch, you'll actually wind up with a branch called "branches/2018Q3" that actually contains the latest changes as of your last pull.
Then, I currently keep my local changes stored at a separate place and just patch/merge them into the source after every update or switch. It seems I can continue to do so, by reverting and re-applying the patches on demand - as this allows to have different patchsets for different machines.
Yes, your Git repo will be different from all the other clones of the Freebsd ports repo. Git makes this sort of workflow easy.
It seems git would prefer to put the local patches within the repo. This would need different branches for different machines. Then, when quarterly gets switched, the delta between 2021Q1-public and 2021Q1-local-desktop1 needs to be grabbed and merged into 2021Q2-public in order to obtain 2021Q2-local-desktop1... hm...
This is not tested, but something like
Code:
# Initial setup
git switch -c local-desktop1 origin/branches/2021Q1
(make some changes)
git add (...)
git commit (...)

# Update upstream branch
git fetch branches/2021Q1
git switch local-desktop1
git rebase origin/branches/2021Q1

# New quarterly branch
git fetch branches/2021Q2
git switch local-desktop1
git rebase origin/branches/2021Q2

* Edit: The more I think about it, the more I'm convinced a "fast-forward merge" is not a merge at all. Nothing is merged. It's purely a metadata change. This is another area where the Git terminology is unnecessarily confusing.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Hi Jose,

thank you very much for your time explaining the "behind the scene" operation of git. I think that you should write the Handbook chapter on using git.

As I understand your explanation, since I do not plan on modifying the branch I downloaded, i.e., local patches, I do not need the --ff only.

Kindest regards,

M
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Hi SirDice,

I just noticed that you replied also, confirming my understanding.

Kindest regards,

M
 

olli@

Daemon
Developer

Reaction score: 1,263
Messages: 1,140

I understand that there is only one developer for net/gitup
Why would that be relevant? gitup was created by the same developer who created svnup 8 years ago, which many people (including myself) have been using successfully since then. And now that FreeBSD has switched from SVN to Git, gitup is designed to take over that role.

Of course, there may be good reasons why you would prefer to use git instead of gitup. The sole purpose of gitup is to fetch the source tree (and soon the ports tree, too) as easily as possible, and with as little overhead as possible. It is meant for users (not developers) who just want to build src or ports, nothing else.
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Hi olli@,

I am generally leery of an application(s) that has(ve0 a single developer. What if he gets tired of working on it, and with the complexity of git the application quits working. Then, I will have to learn git proper anyway, perhaps in the least opportune time.

Furthermore, there is always fun to learn something new, especially, when there is a person willing to explain the concepts, like Jose.

Kindest regards,

M
 

Jose

Daemon

Reaction score: 1,168
Messages: 1,383

I should mention that git-rebase(1) has traps for the unwary too. The first is that you'll have to manually resolve any conflicts that are introduced by changes on the branch you're rebasing too. This can be very complex and time consuming if you have a lot of changes to rebase.

Another is that rebase rewrites the commit history. This is usually fine if the branch you're rebasing is not widely shared or entirely private. Pushing a rebased shared branch is bad form, though, 'cause you could royally screw over the other people that are tracking that branch. Don't do it unless you really know what you're doing.
 

Zirias

Son of Beastie

Reaction score: 1,787
Messages: 3,052

I tend to be even stricter here: as soon as a branch is pushed anywhere or published for someone to pull (so, not strictly local in your own private repository), don't rebase.

But I'm really looking forward to using git for ports, as I *do* have local changes I want to keep, or maybe create a patch for a submission, and rebase will be the perfect tool for that. It was a bit of a hassle with subversion, creating conflicts during svn up, and having to svn add new files on a branch I could never commit on, just so I could create a patch…
 

olli@

Daemon
Developer

Reaction score: 1,263
Messages: 1,140

I am generally leery of an application(s) that has(ve0 a single developer. What if he gets tired of working on it, and with the complexity of git the application quits working.
John Mehr didn’t get tired in the past 8 years of maintaining svnup, and I’m confident that he will continue to care for gitup just the same. Of course there’s no guarantee, but even if he stops maintaining it, chances are that someone else will pick it up.
Besides, even though git itself is somewhat complex, its network API is rather static. I wouldn’t expect any changes that break compatibility in the foreseeable future.
 

PMc

Daemon

Reaction score: 694
Messages: 1,394

Yes. Git-status will show you the files that have changed in your working copy. Git-diff will show you the actual changes.

To stage changes for a commit.

So that you can switch to other branches. Also, changes in your working tree exist only in the clone directory on your machine, and cannot be pushed to a remote repo.
Ahh, thanks for mentioning, this brings up the backup topic. And while one does not want to copy the whole repo into some backup stprage, the local patches are certainly worth being cared for.

And also, git is a database, so a simple filesystem backup is not technically correct anyway. The docs seem to recommend filesystem snapshots as the solution. Unlike SVN, there seems no specific "dump" command present, and obviousely most people do just clone the data to some cloud.

So this needs some thinking, too.

This will set aside your changes, fast-forward the branch to the remote HEAD, and then re-apply your changes. You'll have to resolve any conflicts caused by this re-application.
[rebase] Sounds like that's the most practical way for some limited number of locally maintained patches.

What you propose will create a local branch called "branches/2018Q3" that is unrelated to the upstream branch called "branches/2018Q3". It could be completely different, too. If you issue that command while "main" is the checked-out branch, you'll actually wind up with a branch called "branches/2018Q3" that actually contains the latest changes as of your last pull.
Thank You - that approximately might be what had happened. :)

I prefer to use git-switch because "checkout" means too many different things in Git.
That is what I was looking for. And it can optionally merge local changes to the new branch. :)

For the most simple cases of some local patches (where no history is needed) this should already do - and the backup issue could then be solved by just saving the git-diff output into backup.

Thanks a lot for Your effort, this now begins to look entertaining :)
 

jmehr

Member

Reaction score: 59
Messages: 32

Why would that be relevant? gitup was created by the same developer who created svnup 8 years ago, which many people (including myself) have been using successfully since then. And now that FreeBSD has switched from SVN to Git, gitup is designed to take over that role.

Of course, there may be good reasons why you would prefer to use git instead of gitup. The sole purpose of gitup is to fetch the source tree (and soon the ports tree, too) as easily as possible, and with as little overhead as possible. It is meant for users (not developers) who just want to build src or ports, nothing else.

I'm definitely supporting net/gitup for the long haul. I'm grateful for the opportunity to give something back to the FreeBSD community that has given me so much for the past 21 years. And I'm having way too much fun to stop! :)
 
OP
M

mefizto

Aspiring Daemon

Reaction score: 32
Messages: 730

Hi jmehr,

thank you for the assurance, since I do not need anything sophisticated at this point, I will try net/gitup first.

Kindest regards,

M
 
Top