Ports transitioned to git.

Zirias said:
IOW, ignore it if you know you handle your local repository correctly
Click to expand...
I only have a handful of additional patch files (read: untracked files) in my ports tree, that git is not aware of. For the time being I'd like to keep it that way and I guess that shouldn't be a problem then?
 
Thanks everyone for this thread - the to-ing and fro-ing has helped to explain why some things are they way they are.

My old ports flow (as a ports user rather than developer):
Code: 
portsnap fetch update
pkg version -vIL=
Before starting with the new flow, I built net/gitup, replaced the /usr/local/etc/gitup.conf with the one referenced in this thread, mv'd /usr/ports to /usr/old-ports and made a new /usr/ports directory and started in there with the new flow:
Code: 
gitup ports
make fetchindex
pkg version -vIL=
(And then I use whisper portm**ter)

I didn't time the original gitup ports - was very roughly a minute. All this on FreeBSD 12.2p6.

EDIT: but might have to change the make fetchindex part - I'll see how I go: https://forums.freebsd.org/threads/ports-git-portsnap-warning-about-index.79689/
 
mickey said:
I only have a handful of additional patch files (read: untracked files) in my ports tree, that git is not aware of. For the time being I'd like to keep it that way and I guess that shouldn't be a problem then?
Click to expand...
It's as much a problem as it was with SVN. but as long as you don't commit these to your local branch tracking main from freebsd, --ff-only won't have any effect.

The thing is, with GIT, you have a much better option for local changes: Manage them in your own local branch. Roughly like this:

Code: 
git pull
git checkout -b local
git add mycat/addedport
git commit
[edit commit message: "add port mycat/addedport"]
git add devel/frobnicate/Makefile
git commit
[edit commit message: "devel/frobnicate: fix build with libfoobar"]
[…]

Then, to update and rebase, you can do
Code: 
git fetch origin main:main
git rebase main
 
Zirias said:
It's as much a problem as it was with SVN. but as long as you don't commit these to your local branch tracking main from freebsd, --ff-only won't have any effect.
Click to expand...
I have no intention of committing those files.
Zirias said:
The thing is, with GIT, you have a much better option for local changes: Manage them in your own local branch.
Click to expand...
I'd rather call those additions than changes, as these are all additional files that live in the corresponding port's files directory. So no file under version control is actually changed. Most of these are additional patches that get applied during the patch phase when building the port. The only exception is an .asterisk.makeopts file which is pretty much expendable, as it can easily be recreated. And from your explanation it seems that I would have to perform additional steps each and every time I update the ports tree, so I don't think it's worth it for the time being.
 
mickey said:
… I saw instructions mentioning the use of -o freebsd in the clone command, while others omitted this flag. What difference does it make?
Click to expand...

Good question. -o <name>, --origin <name> appears in git-clone(1) however I know so little about git, I can't tell the reason(s) for inclusion or omission in cases such as this.

I took my cue (to include) from things such as Warner Losh's FreeBSD mini-Git Primer (FreeBSD Journal, November/December 2020). Under Deep Clone (page 2):

% git clone -o freebsd $URL -b branch [dir]

– no use of -o in the earlier review copy at https://hackmd.io/hJgnfzd5TMK-VHgUzshA2g#Normal-Clone
bsdimp (or anyone with the knowhow) please, can you answer the question? Thanks.
 
grahamperrin said:
– no use of -o in the earlier review copy at https://hackmd.io/hJgnfzd5TMK-VHgUzshA2g#Normal-Clone
bsdimp (or anyone with the knowhow) please, can you answer the question? Thanks.
Click to expand...
There are two ways to make a Git repository appear on your filesystem. You can go into a directory and type git-init(1). You'll now have a brand-spankin' new Git repository with no commits in it. It will also not have any connections to any remote repositories. It will bask in its unique snowflake nature until you connect it to the mean and nasty Internet.

The other way is to git-clone(1) an existing Git repository from somewhere. That somewhere can be out in the Internets, on some other machine in the same network, or even somewhere else in the same filesystem. This kind of Git repo is not as pure and unsullied. It contains a reference to where it came from in the form of a remote called "origin". However, this is purely convention. You may decide that you want to call the original remote "upstream" or "booyah". That's what the -o flag is for, to specify a different name for the remote whence the repo came.

There's absolutely nothing special about the name "origin" other that it's the default. You might consider using something else for this original remote a violation of the principle of least surprise. I don't feel strongly about it one way or the other.
 
Jose said:
That is not correct. There is no generic "quarterly" branch. Each individual quarterly branch gets a unique name in the form yyyyQ[1-4]. The latest quarterly branch is 2021Q1. There is no Q2 branch yet. It's part of the migration:


Not quite. The git.freebsd.org URLs are the canonical URLs used for cloning repositories. The cgit.freenbsd.org URLs are for the Web interface to the Git repositories. They are used to browse the source online and are not to be used for cloning (won't work anyway). See https://github.com/lwhsu/freebsd-git-docs/blob/main/URLs.md


More information needed. A git pull does very little if your local clone is up-to-date with the remote. What are the exact commands you are running? But I suggest you take this subject to its own topic. Jmehr has been very responsive on these forums in the recent past.
Click to expand...

The "quarterly" section is designed to be an alias for the current quarterly branch or the previous quarterly branch if the current quarterly branch doesn't exist. The goal is to make tracking quarterly ports branches seamless and to avoid the pain of having to manually update gitup.conf every three months. Unfortunately, there's a bug in 0.90 where this is broken :( -- it's fixed for 0.91 which I'll be pushing out once proxy support is fully implemented (hopefully soon).

Unfortunately, net/gitup will always be slower than the official git client and will be more noticeable with slower hardware (and especially with slower drives).

Because net/gitup minimizes the disk space required, it only deals with shallow copies and does not save the raw packfiles sent from the Git servers. When requesting pulls, the official Git client depends on the packfile data that has been previously fetched. If these packfiles contain all of the information needed to create a local repository, then a pristine local repository can be used to reconstruct the information that existed in the original packfiles and net/gitup must walk the local tree and calculate the checksum for every file to do this, sacrificing execution speed to save disk space.
 
Hello!

Here is my 5 cents to this thread - did it on my test desktop machine. Here is my action log:

Code: 
zfs snapshot flash/usr/ports@portsnap

zfs create flash/usr/ports_git

zfs set mountpoint=/usr/ports_portsnap flash/usr/ports

zfs set mountpoint=/usr/ports flash/usr/ports_git

vi /usr/local/etc/gitup.conf

        "ports" : {
                "repository" : "/ports.git",
                "branch"     : "main",
                "target"     : "/usr/ports",
                "ignores"    : [
                        "distfiles",
                        "packages",
                ],
        },

gitup ports -v 1

# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: d73721414d8ded2d68448effc25c895057225a0e
# Branch: main
# Action: clone

portversion -vl "<"

[Reading data from pkg(8) ... - 1474 packages found - done]
Fetching the ports index ... /usr/bin/env  fetch -am -o /usr/ports/INDEX-13.bz2 https://www.FreeBSD.org/ports/INDEX-13.bz2
/usr/ports/INDEX-13.bz2                               2271 kB 2448 kBps    01s
done
[Updating the portsdb <format:bdb_btree> in /usr/ports ... - 31029 port entries found .........1000.........2000.........3000.........4000.........5000.........6000.........7000.........8000.........9000.........10000.........11000.........12000.........13000.........14000.........15000.........16000.........17000.........18000.........19000.........20000.........21000.........22000.........23000.........24000.........25000.........26000.........27000.........28000.........29000.........30000.........31000 ..... done]

Few minutes all the procedure.
 
mickey said:
I have no intention of committing those files.
Click to expand...
That's exactly the scenario ;)

mickey said:
I'd rather call those additions than changes, as these are all additional files that live in the corresponding port's files directory. So no file under version control is actually changed. Most of these are additional patches that get applied during the patch phase when building the port. The only exception is an .asterisk.makeopts file which is pretty much expendable, as it can easily be recreated.
Click to expand...
The benefit is to have your own history with meaningful commits, the ability to track your own changes and easily correct mistakes, and a much easier way to resolve any conflicts should they ever happen. It's nothing you have to do, just a nice tool. To take full advantage, you'd have to learn how to work with "rebase", e.g. from here: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

A simpler alternative, if you just continue to keep all your changes in the working copy only, is git stash. This will be obligatory if you ever run into a conflict, then you'll have to git stash before you can successfully git pull. To get your changes back afterwards, write git stash pop. In that case, the "stash" is kind of a single commit that will be rebased onto your working copy with "pop". It's your decision whether this is good enough for you. I just recommend the local branch for easy management of several unrelated changes :)

mickey said:
And from your explanation it seems that I would have to perform additional steps each and every time I update the ports tree, so I don't think it's worth it for the time being.
Click to expand...
Uhm, I don't think having to issue two commands (a merge and a rebase) for updating instead of just one pull command is really a drawback ;)
 
jmehr said:
Because net/gitup minimizes the disk space required, it only deals with shallow copies and does not save the raw packfiles sent from the Git servers. When requesting pulls, the official Git client depends on the packfile data that has been previously fetched. If these packfiles contain all of the information needed to create a local repository, then a pristine local repository can be used to reconstruct the information that existed in the original packfiles and net/gitup must walk the local tree and calculate the checksum for every file to do this, sacrificing execution speed to save disk space.
Click to expand...
Sure, walking a large tree takes time, but attributing a difference of magnitudes to it still looks like ignoring a problem…
 
I can only (again) wonder about the space savings potential of git compared to svn. In my local ports tree there are no work directories and no distfiles, both are kept on separate volumes each. Until yesterday, this has been the usage of the ports tree updated by svn:
Code: 
sys/usr/ports  used                  3,89G                                 -
sys/usr/ports  referenced            3,89G                                 -
sys/usr/ports  compressratio         2.62x                                 -
sys/usr/ports  compression           lz4                                   local
sys/usr/ports  copies                1                                     default
sys/usr/ports  dedup                 off                                   default
sys/usr/ports  logicalused           5,85G                                 -
sys/usr/ports  logicalreferenced     5,85G                                 -
Now after I deleted, recreated and populated the dataset using git clone -o freebsd --single-branch --depth 1 https://git.freebsd.org/ports.git . the usage is quite different:
Code: 
sys/usr/ports  used                  789M                                  -
sys/usr/ports  referenced            789M                                  -
sys/usr/ports  compressratio         1.74x                                 -
sys/usr/ports  compression           lz4                                   local
sys/usr/ports  copies                1                                     default
sys/usr/ports  dedup                 off                                   default
sys/usr/ports  logicalused           502M                                  -
sys/usr/ports  logicalreferenced     502M                                  -
I know svn has the annoying habit of keeping a pristine copy of everything, effectively doubling the space required, but still... o_O
 
Zirias said:
The benefit is to have your own history with meaningful commits, the ability to track your own changes and easily correct mistakes, and a much easier way to resolve any conflicts should they ever happen.
Click to expand...
I understand that it would make my life much easier if it were actually changes (that have a history) we're talking about, But the files in question are not present at all in the repo and are unified diffs against the original sources of a specific port. Nothing you would ever want to manually edit. Either those patches apply cleanly, or (if the original sources have diverged too much) you will have to manually merge the changes and then create a new diff. And given the fact we're talking about a total of 8 files relating to 4 ports, I don't think the additional overhead is justified. If at some point it gets more (and more complex) I would probably think about such a more organized solution though.
 
Zirias said:
The thing is, with GIT, you have a much better option for local changes:
Click to expand...
The "always bad thing" about switching from A to B is that your usual workflow no longer works, but you neither know nor are able to use the advantages of the new tool; you only get disadvantages at first.

One big difference I've noticed: "gitup ports" removes any own file in the ports tree directory (a clean "reset" for everything), while "portsnap fetch update" (or a "svn up") handles only files explicit versioned. This "much better option for local changes" may be there, but that will take some more time for me…

However, I'm sure I will make my peace with GIT, and as a friend of simple setups I've also switched now my own projects to GIT (sadly GIT cannot handle a directory structure without containing at least one file - that's so far the ugliest disadvantage).
 
jmos said:
One big difference I've noticed: "gitup ports" removes any own file in the ports tree directory (a clean "reset" for everything), while "portsnap fetch update" (or a "svn up") handles only files explicit versioned. This "much better option for local changes" may be there, but that will take some more time for me…
Click to expand...
With local changes, you should use git, not gitup.
 
jmos said:
Git is just the next cool must have to play with. It has zero benefit to me, but is a additional burden to deal with.)
Click to expand...
Sorry, but that's totally wrong. You do realize that it took FreeBSD 2 or so years before they came to this decision and carried out the change? See 3 years ago I wrote this guide on Git. Back then it was the next new cool thing but after 3 more years?

As for people being more inclined to participate on development because of Git... I can't tell for sure of course considering that I'm not involved with this myself, but to be honest I wouldn't be surprised if this were true. For example.. if I want to work on a project managed by Subversion then I have to check out the source and after that I'm fully tied to the repository I got the source code from.

With Git on the other hand you have way more freedom. I can work on a project and keep track of my own developments. I can provide my changes to the "powers that be" but if they refuse any of my work then it won't be gone "just like that", I still keep my own local changes. Heck, I can even work on my own stuff while still keeping up to date with the official developments.. You can't easily do that with Subversion. It gets even better: I can also provide these local changes to others.

Sure.. I can understand that it may not seem like much for an end user. But FreeBSD never was about the "user experience" in the first place, things need to work and work as efficiently as possible. That's what happened here.
 
Zirias said:
Sure, walking a large tree takes time, but attributing a difference of magnitudes to it still looks like ignoring a problem…
Click to expand...
Just traversing the ports tree ( find . >/dev/null) takes a little less than 2 minutes on one of my older machines that still has a HDD (i.e. rotating rust). And that’s without actually opening the files, let alone calculating checksums. This is UFS mounted with the noatime flag; otherwise it would probably take even longer.
 
Just for the record, on my newer machine with SSD (NVMe) the traversal takes only 2.5 seconds. Not too bad for a tree of about 40k directories and 140k files.
 
olli@ said:
Just traversing the ports tree ( find . >/dev/null) takes a little less than 2 minutes on one of my older machines that still has a HDD (i.e. rotating rust).
Click to expand...
Hm. I don't own any SSDs, my testing is on a raidz pool with 4 HDDs, and I can't reproduce any substantial(!) slowdown with gitup. But maybe ZFS' ARC helps a lot here?
 
Zirias said:
Hm. I don't own any SSDs, my testing is on a raidz pool with 4 HDDs, and I can't reproduce any substantial(!) slowdown with gitup. But maybe ZFS' ARC helps a lot here?
Click to expand...
Good question. I did my test with empty cache, because that’s the typical case when updating the ports tree. When I repeat the test immediately, the traversal is much faster, of course, because the whole ports tree is still in the cache. Also, I don’t have any RAID (I guess most users have only one disk for FreeBSD).
 
I haven’t used gitup for ports so far (I intend to continue using portsnap for the time being, as I don’t really have a good reason to change right now), so I can’t compare. But I use gitup for updating my src tree, and that is roughly as fast as svnup that I used before. So I’m happy with it.
 
olli@ said:
Good question. I did my test with empty cache, because that’s the typical case when updating the ports tree. When I repeat the test immediately, the traversal is much faster, of course, because the whole ports tree is still in the cache. Also, I don’t have any RAID (I guess most users have only one disk for FreeBSD).
 
I haven’t used gitup for ports so far (I intend to continue using portsnap for the time being, as I don’t really have a good reason to change right now), so I can’t compare. But I use gitup for updating my src tree, and that is roughly as fast as svnup that I used before. So I’m happy with it.
Click to expand...
First time running on my desktop gave:

Code: 
time gitup ports -v 1

# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 9aae35b48f216c96c71d782f0f14e3cfd6a70f90
# Branch: main
# Action: clone

Executed in   55.59 secs    fish           external
   usr time    7.04 secs  512.00 micros    7.04 secs
   sys time   14.61 secs  256.00 micros   14.61 secs

Next time:

Code: 
root@Tuna2 ~# time gitup ports -v 1
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: cf118ccf875508b9a1c570044c93cfcc82bd455c
# Want: 0997d61ae7a84a95b0e81febfac0e4da9a247ac2
# Branch: main
# Action: pull

________________________________________________________
Executed in   15.16 secs    fish           external
   usr time    3.38 secs  514.00 micros    3.38 secs
   sys time    7.19 secs  265.00 micros    7.19 secs


And just few minutes later without updates:

Code: 
time gitup ports -v 1
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 0997d61ae7a84a95b0e81febfac0e4da9a247ac2
# Want: 0997d61ae7a84a95b0e81febfac0e4da9a247ac2
# Branch: main

________________________________________________________
Executed in    8.37 secs    fish           external
   usr time    2.33 secs    0.00 millis    2.33 secs
   sys time    5.13 secs    5.16 millis    5.13 secs
 
grahamperrin said:
Thanks, I needed that.

With poudriere-ports(8) as a starting point, how would a person discover the use of option -U for the git method?
Click to expand...
It's unfortunately not documented, AFAIK. That command was posted to freebsd-ports a couple of days ago:

There are recommendations in that mail thread for alternatives to the -U switch:

I haven't tested any because as I said, I use -m null and -M /path/to/git/ports to point Poudriere at a manually-maintained Git checkout of the ports tree.
 
mickey said:
I understand that it would make my life much easier if it were actually changes (that have a history) we're talking about, But the files in question are not present at all in the repo and are unified diffs against the original sources of a specific port. Nothing you would ever want to manually edit. Either those patches apply cleanly, or (if the original sources have diverged too much) you will have to manually merge the changes and then create a new diff. And given the fact we're talking about a total of 8 files relating to 4 ports, I don't think the additional overhead is justified. If at some point it gets more (and more complex) I would probably think about such a more organized solution though.
Click to expand...
Disclaimer: I'm trying to help you. Let me know if I'm annoying you instead and I'll shut up. No need to read the following if that's the case.

Well git-diff(1) creates unidiff patches by default. Imagine this scenario:
  • You keep a local tracking branch of the quarterly branch called 2021Q1
  • You have a private local branch called mychanges with your local port changes
Workflow 1: Update the quarterly tracking branch
Code: 
git switch 2021Q1
git pull --ff-only
git switch mychanges
git rebase 2021Q1
The rebase will warn you if there are conflicts, and will leave modified files in your working tree with conflict markers. You would then resolve your conflicts and commit your changes. Your local branch now has history for the changes you've had to make to keep up with upstream.

Workflow 2: New quarterly ports branch
Code: 
git switch -c 2021Q2 -t origin/2021Q2
git switch mychanges
git rebase 2021Q2
And resolve the conflicts as before.

Workflow 3: Create unified diffs of your changes against the quarterly tracking branch
Code: 
git switch 2021Q2
git pull --ff-only
git diff mychanges..

I haven't actually tested these git command sequences, but they should be pretty close to accurate.
 
Can't find a guide on how to set up gitup to get ports.
Created a config file, but I don't quite understand if it is correct.
/usr/local/etc/gitup.conf
{
"defaults" : {
"host" : "git.freebsd.org",
"port" : 443,
"verbosity" : 1,
"work_directory" : "/var/db/gitup",
},

"ports" : {
"host" : "git.freebsd.org",
"repository" : "ports.git",
"branch" : "main",
"target" : "/usr/ports",
"ignores" : [
"distfiles",
"packages",
],
},
}
Click to expand...
 
You must log in or register to reply here.
Back
Top