Ports transitioned to git.

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

I'm not sure what the problem is here. The transition is complete. Pushing changes to mirrors on Github, Gitlab etc is entirely irrelevant for using the official repo. net/gitup will work just fine with it.
 

grahamperrin

Well-Known Member

Reaction score: 106
Messages: 389

The quote "Transition almost done …" came from a gitup area around an hour ago; please don't shoot the messenger.

I don't expect a significant difference in speed from a change of source

True. Around sixteen minutes:

Code:
root@mowa219-gjp4-8570p:~ # grep repository /usr/local/etc/gitup.conf
                "repository" : "/ports.git",
                "repository" : "/ports.git",
                "repository" : "/src.git",
                "repository" : "/src.git",
                "repository" : "/src.git",
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 848355ff18d260cf954ca13c37f4d4d266ee6d0e
# Branch: main
# Action: pull
 * /usr/ports/README
…
 - /usr/ports/www/node/files/patch-tools_genv8constants.py
7.677u 9.738s 15:57.54 1.8%     51+398k 143400+78033io 1pf+0w
root@mowa219-gjp4-8570p:~ #
 

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

The quote "Transition almost done …" came from a gitup area around an hour ago; please don't shoot the messenger.
This doesn't make sense, gitup won't tell you anything about a "transition". The repository is now available for a while, and since then, gitup will work correctly on it.

True. Around sixteen minutes:
Code:
7.677u 9.738s 15:57.54 1.8%     51+398k 143400+78033io 1pf+0w
Trying to replicate what you did (so starting from commit 4010f7bbc03638d71781ce091bf40a0907fa12fe on an empty /usr/ports, then updating…), I can't reproduce any slowness:
(1 minute 15 seconds for initial clone, 11.4 seconds for updating from 4010f7bbc03638d71781ce091bf40a0907fa12fe)
Code:
nexus# time gitup -w 4010f7bbc03638d71781ce091bf40a0907fa12fe ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: (detached)
# Action: clone
[...]
 + /usr/ports/x11/zenity/distinfo
 + /usr/ports/x11/zenity/pkg-descr
 + /usr/ports/x11/zenity/pkg-plist
gitup -w 4010f7bbc03638d71781ce091bf40a0907fa12fe ports  10,74s user 13,68s system 32% cpu 1:15,12 total
nexus# time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 7444131cd19a49d73952e741999e08e7dc5905c6
# Branch: main
# Action: pull
[...]
 - /usr/ports/www/chromium/files/patch-third__party_zlib_BUILD.gn
 - /usr/ports/www/chromium/files/patch-ui_base_data__transfer__policy_data__transfer__endpoint.h
 - /usr/ports/www/chromium/files/patch-weblayer_browser_content__browser__client__impl.h
 - /usr/ports/www/node/files/patch-tools_genv8constants.py
gitup ports  4,44s user 4,30s system 76% cpu 11,444 total
It seems you have to troubleshoot this locally. Bad network connectivity?

edit: Are you running this on -CURRENT? Do you use a GENERIC kernel? What about MALLOC_PRODUCTION? If you have all kinds of debugging stuff enabled, catastrophic performance is expected.
 

grahamperrin

Well-Known Member

Reaction score: 106
Messages: 389

For my use case:

True, at least for git(1). Abridged:

Code:
root@mowa219-gjp4-8570p:~ # time git -C /usr/ports pull --ff-only
remote: Enumerating objects: 292, done.
remote: Counting objects: 100% (292/292), done.
remote: Compressing objects: 100% (120/120), done.
remote: Total 212 (delta 149), reused 113 (delta 92), pack-reused 0
Receiving objects: 100% (212/212), 45.30 KiB | 6.47 MiB/s, done.
Resolving deltas: 100% (149/149), completed with 49 local objects.
From https://git.freebsd.org/ports
   7444131cd19a..4b13a4c45a26  main       -> freebsd/main
Updating 7444131cd19a..4b13a4c45a26
Fast-forward
 .gitignore                                                                                                                       |   2 +-
 audio/schismtracker/Makefile                                                                                                     |   2 +-
…
 create mode 100644 databases/mongodb49/pkg-plist
0.835u 1.863s 0:30.81 8.7%      2965+762k 8073+403io 364pf+0w
root@mowa219-gjp4-8570p:~ #

– around thirty seconds.
 

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

So, do you use -CURRENT with all the debugging stuff enabled? If so, that's most likely the "problem" with gitup. Even 30 seconds is a lot for a pull…
 

grahamperrin

Well-Known Member

Reaction score: 106
Messages: 389

From IRC:
First run of gitup ports: around forty-six seconds. Second run: around twelve minutes. Those were with file system compression relaxed, to zstd. I don't perceive a problem with -CURRENT.

More generally, emphatically:
  • I do not perceive a problem
– I simply do not expect gitup(1) to be as fast as git(1).

https://github.com/johnmehr/gitup/blob/2b055c47f4e5f703c7ae81ec5da5e6c41ed0ef6c/gitup.1.in#L129 explains that gitup(1) "… relies on the known remote files lists stored in /var/db/gitup and the current state of the local repository to reconstruct data …".

I know very little about git(1) but I assume that it takes a more expeditious approach to determining the state of the local repository; no need to reconstruct data.

Code:
root@mowa219-gjp4-8570p:~ # date ; uptime
Tue Apr  6 11:25:41 BST 2021
11:25AM  up 15:50, 5 users, load averages: 1.58, 1.83, 2.18
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem              Size    Used   Avail Capacity  Mounted on
copperbowl/usr/ports    190G    1.5G    189G     1%    /usr/ports
root@mowa219-gjp4-8570p:~ # zfs set compression=zstd copperbowl/usr/ports
root@mowa219-gjp4-8570p:~ # time rm -r /usr/ports/*
0.669u 7.811s 6:31.69 2.1%      10+169k 14121+0io 0pf+0w
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem              Size    Used   Avail Capacity  Mounted on
copperbowl/usr/ports    190G    847M    190G     0%    /usr/ports
root@mowa219-gjp4-8570p:~ # time du -hs /usr/ports
847M    /usr/ports
0.002u 0.000s 0:00.20 0.0%      0+0k 7+0io 1pf+0w
root@mowa219-gjp4-8570p:~ # rm -r /usr/ports/.git*
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x   2 root  wheel     3B Apr  6 11:41 /usr/ports/.
drwxr-xr-x  17 root  wheel    17B Jan  3 07:57 /usr/ports/..
-rw-r--r--   1 root  wheel   115B Apr  6 10:53 /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # rm /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 848355ff18d260cf954ca13c37f4d4d266ee6d0e
# Want: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Branch: main
 ! /usr/ports/.arcconfig is missing.
 ! /usr/ports/.gitattributes is missing.
 ! /usr/ports/.gitauthors is missing.
 ! /usr/ports/.gitignore is missing.
 ! /usr/ports/.gitmessage is missing.
 ! /usr/ports/CHANGES is missing.
…
 ! /usr/ports/x11/zenity/pkg-plist is missing.
gitup: build_repair_command: There are too many files to repair -- please re-clone the repository: Argument list too long
0.786u 0.224s 0:06.30 15.8%     53+175k 113+0io 6pf+0w
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x   2 root  wheel     2B Apr  6 11:41 /usr/ports/.
drwxr-xr-x  17 root  wheel    17B Jan  3 07:57 /usr/ports/..
root@mowa219-gjp4-8570p:~ # rm /var/db/gitup
rm: /var/db/gitup: is a directory
root@mowa219-gjp4-8570p:~ # rm /var/db/gitup/ports
root@mowa219-gjp4-8570p:~ # ls -hl /var/db/gitup
total 0
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Branch: main
# Action: clone
 + /usr/ports/.arcconfig
 + /usr/ports/.gitattributes
 + /usr/ports/.gitauthors
 + /usr/ports/.gitignore
 + /usr/ports/.gitmessage
 + /usr/ports/CHANGES
…
 + /usr/ports/x11/zenity/pkg-plist
7.054u 12.562s 0:45.50 43.0%    50+3166k 142+217743io 0pf+0w
root@mowa219-gjp4-8570p:~ # time gitup ports
load: 1.72  cmd: gitup 28070 [zio->io_cv] 283.53r 2.56u 3.55s 2% 281096k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
load: 1.89  cmd: gitup 28070 [zio->io_cv] 693.76r 5.34u 8.63s 3% 326260k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Want: 64d94165652f40041bf71bdf7a775f867e7b80a9
# Branch: main
# Action: pull
 * /usr/ports/security/openssl/Makefile
7.403u 10.415s 12:30.76 2.3%    50+165k 144157+77802io 0pf+0w
root@mowa219-gjp4-8570p:~ #

If you like, take a holistic view of https://bsd-hardware.info/?probe=d9ec051372 … if not holistic, then please at least be aware of NODEBUG.
 

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

vigole

Daemon

Reaction score: 1,199
Messages: 1,106

grahamperrin :
I've asked it on IRC, but for the record, I'll ask it here too: Are you on main branch, or on tag 0.90? gitup.conf has changed since tag 0.90.
 

grahamperrin

Well-Known Member

Reaction score: 106
Messages: 389

With an unnaturally clean installation of FreeBSD 12.2-RELEASE-p4, and the local repository up-to-date:
  • git pull took less than one second
  • pulls with gitup took between eighty-five and one hundred and twenty-five seconds.
 

Jose

Aspiring Daemon

Reaction score: 753
Messages: 885

I'm not sure. Seems like it would be a good idea. chrcol, looking at my /usr/local/etc/gitup.conf, I have this for quarterly, but don't know if it is correct...
Code:
"quarterly" : {
                "host"       : "github.com",
                "repository" : "/freebsd/freebsd-ports.git",
                "branch"     : "quarterly",
                "target"     : "/usr/ports",
                "ignores"    : [
                        "distfiles",
                        "packages",
That is not correct. There is no generic "quarterly" branch. Each individual quarterly branch gets a unique name in the form yyyyQ[1-4]. The latest quarterly branch is 2021Q1. There is no Q2 branch yet. It's part of the migration:

https://git.freebsd.org/docs.git redirects to https://cgit.freebsd.org/doc where https://git.FreeBSD.org/doc.git is the primary non-developer address for cloning. I ignore non-working links such as (public-mirror).

https://git.freebsd.org/src.git redirects to https://cgit.freebsd.org/src where https://git.FreeBSD.org/src.git is the primary non-developer address for cloning.

As the transition progresses, I expect https://git.freebsd.org/ports.git to begin redirecting to https://cgit.freebsd.org/ports
Not quite. The git.freebsd.org URLs are the canonical URLs used for cloning repositories. The cgit.freenbsd.org URLs are for the Web interface to the Git repositories. They are used to browse the source online and are not to be used for cloning (won't work anyway). See https://github.com/lwhsu/freebsd-git-docs/blob/main/URLs.md

With an unnaturally clean installation of FreeBSD 12.2-RELEASE-p4, and the local repository up-to-date:
  • git pull took less than one second
  • pulls with gitup took between eighty-five and one hundred and twenty-five seconds.
More information needed. A git pull does very little if your local clone is up-to-date with the remote. What are the exact commands you are running? But I suggest you take this subject to its own topic. Jmehr has been very responsive on these forums in the recent past.
 

suntzu00

Member

Reaction score: 33
Messages: 74

if you're using poudriere and you want to follow 'main'(replace it with the branch name if you're interested in following quarterly) :

poudriere ports -c -B main -m git -U 'https://git.FreeBSD.org/ports.git' -p MAIN

maybe someone will need this
 

grahamperrin

Well-Known Member

Reaction score: 106
Messages: 389

… What are the exact commands you are running?

For gitup: essentially, gitup ports – nothing complicated.

For git: as outlined in the guide that I linked from https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504066 on page 2. Again, nothing complicated.

But I suggest you take this subject to its own topic. …

+1

– if anyone else would like to do so. I don't have a problem with the test results, other people are welcome to use the results as points of reference in a new topic.
 

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

grahamperrin even if you're fine with it, they do show a problem though, gitup shouldn't be slower than git (at least not substantially). But as long as nobody else finds similar problems, there will probably be no thread then ;)
 

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

If there's one thing I don't see, it's respect. No, there wasn't any reason, just assumptions that are pretty unlikely. A lot of people are trying gitup, you're the only one so far reporting it to be a LOT slower than git. There is a problem for sure. If you don't have any interest in having it fixed, that's fine. But don't claim it "works that way", that's nonsense.
 

Mayhem30

Well-Known Member

Reaction score: 26
Messages: 453

On my slowest machine that has a Intel i3 CPU, 8GB ddr2 ram and 1TB HDD it takes 23 seconds for gitup ports to complete.

I'm not sure how that compares to git .. but I'm happy with those numbers, so I'm going to stick with it.
 

grahamperrin

Well-Known Member

Reaction score: 106
Messages: 389

poudriere ports -c -B main -m git -U 'https://git.FreeBSD.org/ports.git' -p MAIN
Thanks, I needed that.

With poudriere-ports(8) as a starting point, how would a person discover the use of option -U for the git method?

I chose to delete then recreate my default ports tree, encountered a bug that prevented me from destroying the filesystem for the deleted tree. Worked around by killing gvfsd-trash, which is not necessarily recommended but it suits me.

For reference only (not seeking help with this):

Code:
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default...cannot create 'copperbowl/poudriere/ports/default': dataset already exists
fail
[00:00:00] Error: Failed to create FS copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
cannot unmount '/copperbowl/poudriere/ports/default': pool or dataset is busy
root@mowa219-gjp4-8570p:~ # poudriere ports -l
PORTSTREE   METHOD TIMESTAMP PATH
portoverlay -                /usr/local/poudriere/ports/portoverlay
root@mowa219-gjp4-8570p:~ # killall gvfsd-trash
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree... done
root@mowa219-gjp4-8570p:~ # cat /usr/local/poudriere/ports/default/.git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = git://github.com/freebsd/freebsd-ports.git
        fetch = +refs/heads/main:refs/remotes/origin/main
[branch "main"]
        remote = origin
        merge = refs/heads/main
root@mowa219-gjp4-8570p:~ # poudriere ports -d default
[00:00:00] Are you sure you want to delete the ports tree default at /usr/local/poudriere/ports/default? [y/N] y
[00:00:02] Deleting portstree "default"... done
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default...cannot create 'copperbowl/poudriere/ports/default': dataset already exists
fail
[00:00:00] Error: Failed to create FS copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # killall gvfsd-trash
No matching processes were found
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # rm -r /usr/local/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # time poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree... done
11.174u 7.791s 1:05.23 29.0%    2879+737k 3024+170812io 1095pf+0w
root@mowa219-gjp4-8570p:~ # cat /usr/local/poudriere/ports/default/.git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://git.freebsd.org/ports.git
        fetch = +refs/heads/main:refs/remotes/origin/main
[branch "main"]
        remote = origin
        merge = refs/heads/main
root@mowa219-gjp4-8570p:~ # pkg query '%o %v %R' poudriere-devel
ports-mgmt/poudriere-devel 3.3.99.20210303_1 FreeBSD
root@mowa219-gjp4-8570p:~ #

(I was curious about how /usr/local/poudriere/ports/default/.git/config would differ without option -U.)
 

grahamperrin

Well-Known Member

Reaction score: 106
Messages: 389

If you don't have any interest in having it fixed,

If you must go round in circles, please don't put words in my mouth.

there wasn't any reason, just assumptions

https://github.com/johnmehr/gitup/blob/2b055c47f4e5f703c7ae81ec5da5e6c41ed0ef6c/gitup.1.in#L129 explains that gitup(1) "… relies on the known remote files lists stored in /var/db/gitup and the current state of the local repository to reconstruct data …".

That was a reference to the manual page. Not an assumption.
 

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

So, stop playing silly games now. The manual page has nothing to say about this taking time. That's your assumption, although it contradicts the experience of many who see gitup working pretty fast.
 

mickey

Aspiring Daemon

Reaction score: 299
Messages: 747

Can someone please explain what exactly the --ff-only flag on git pull does? Everything I've read so far was a bit shallow, to say the least. And if git pull --ff-only is the recommended way of updating the tree, why isn't it used in /usr/ports/Makefile:
Code:
.  else
        @echo "--------------------------------------------------------------"
        @echo ">>> Updating ${.CURDIR} from git repository"
        @echo "--------------------------------------------------------------"
        cd ${.CURDIR}; ${GIT} pull
.  endif
Also I saw instructions mentioning the use of -o freebsd in the clone command, while others omitted this flag. What difference does it make?
 

Zirias

Daemon

Reaction score: 1,184
Messages: 2,150

--ff-only will outright refuse to create a merge commit, which would be necessary if you locally added commits to the same branch. Adding commits to a branch you will never push is certainly a bad idea (if you need local commits, use a separate local branch for them, you can rebase it) – but --ff-only is just a safety measure here: Fail fast instead of writing a local history that diverges from the upstream branch more and more.

IOW, ignore it if you know you handle your local repository correctly ;)
 
Top