I don't expect a significant difference in speed from a change of source
root@mowa219-gjp4-8570p:~ # grep repository /usr/local/etc/gitup.conf
"repository" : "/ports.git",
"repository" : "/ports.git",
"repository" : "/src.git",
"repository" : "/src.git",
"repository" : "/src.git",
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 848355ff18d260cf954ca13c37f4d4d266ee6d0e
# Branch: main
# Action: pull
* /usr/ports/README
…
- /usr/ports/www/node/files/patch-tools_genv8constants.py
7.677u 9.738s 15:57.54 1.8% 51+398k 143400+78033io 1pf+0w
root@mowa219-gjp4-8570p:~ #
This doesn't make sense, gitup won't tell you anything about a "transition". The repository is now available for a while, and since then, gitup will work correctly on it.The quote "Transition almost done …" came from a gitup area around an hour ago; please don't shoot the messenger.
Trying to replicate what you did (so starting from commitTrue. Around sixteen minutes:
Code:7.677u 9.738s 15:57.54 1.8% 51+398k 143400+78033io 1pf+0w
4010f7bbc03638d71781ce091bf40a0907fa12fe
on an empty /usr/ports, then updating…), I can't reproduce any slowness:4010f7bbc03638d71781ce091bf40a0907fa12fe
)nexus# time gitup -w 4010f7bbc03638d71781ce091bf40a0907fa12fe ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: (detached)
# Action: clone
[...]
+ /usr/ports/x11/zenity/distinfo
+ /usr/ports/x11/zenity/pkg-descr
+ /usr/ports/x11/zenity/pkg-plist
gitup -w 4010f7bbc03638d71781ce091bf40a0907fa12fe ports 10,74s user 13,68s system 32% cpu 1:15,12 total
nexus# time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 7444131cd19a49d73952e741999e08e7dc5905c6
# Branch: main
# Action: pull
[...]
- /usr/ports/www/chromium/files/patch-third__party_zlib_BUILD.gn
- /usr/ports/www/chromium/files/patch-ui_base_data__transfer__policy_data__transfer__endpoint.h
- /usr/ports/www/chromium/files/patch-weblayer_browser_content__browser__client__impl.h
- /usr/ports/www/node/files/patch-tools_genv8constants.py
gitup ports 4,44s user 4,30s system 76% cpu 11,444 total
-CURRENT
? Do you use a GENERIC
kernel? What about MALLOC_PRODUCTION
? If you have all kinds of debugging stuff enabled, catastrophic performance is expected.… I'm not complaining. YMMV, depending on hardware, time of day and so on.
… catastrophic performance …
For my use case:
- … I do expect git(1) and portsnap(8) to be faster than gitup(1).
root@mowa219-gjp4-8570p:~ # time git -C /usr/ports pull --ff-only
remote: Enumerating objects: 292, done.
remote: Counting objects: 100% (292/292), done.
remote: Compressing objects: 100% (120/120), done.
remote: Total 212 (delta 149), reused 113 (delta 92), pack-reused 0
Receiving objects: 100% (212/212), 45.30 KiB | 6.47 MiB/s, done.
Resolving deltas: 100% (149/149), completed with 49 local objects.
From https://git.freebsd.org/ports
7444131cd19a..4b13a4c45a26 main -> freebsd/main
Updating 7444131cd19a..4b13a4c45a26
Fast-forward
.gitignore | 2 +-
audio/schismtracker/Makefile | 2 +-
…
create mode 100644 databases/mongodb49/pkg-plist
0.835u 1.863s 0:30.81 8.7% 2965+762k 8073+403io 364pf+0w
root@mowa219-gjp4-8570p:~ #
… I'm still testing.
So, do you use -CURRENT with all the debugging stuff enabled? …
This doesn't make sense, gitup won't tell
First run ofgitup ports
: around forty-six seconds. Second run: around twelve minutes. Those were with file system compression relaxed, to zstd. I don't perceive a problem with -CURRENT.
root@mowa219-gjp4-8570p:~ # date ; uptime
Tue Apr 6 11:25:41 BST 2021
11:25AM up 15:50, 5 users, load averages: 1.58, 1.83, 2.18
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem Size Used Avail Capacity Mounted on
copperbowl/usr/ports 190G 1.5G 189G 1% /usr/ports
root@mowa219-gjp4-8570p:~ # zfs set compression=zstd copperbowl/usr/ports
root@mowa219-gjp4-8570p:~ # time rm -r /usr/ports/*
0.669u 7.811s 6:31.69 2.1% 10+169k 14121+0io 0pf+0w
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem Size Used Avail Capacity Mounted on
copperbowl/usr/ports 190G 847M 190G 0% /usr/ports
root@mowa219-gjp4-8570p:~ # time du -hs /usr/ports
847M /usr/ports
0.002u 0.000s 0:00.20 0.0% 0+0k 7+0io 1pf+0w
root@mowa219-gjp4-8570p:~ # rm -r /usr/ports/.git*
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x 2 root wheel 3B Apr 6 11:41 /usr/ports/.
drwxr-xr-x 17 root wheel 17B Jan 3 07:57 /usr/ports/..
-rw-r--r-- 1 root wheel 115B Apr 6 10:53 /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # rm /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 848355ff18d260cf954ca13c37f4d4d266ee6d0e
# Want: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Branch: main
! /usr/ports/.arcconfig is missing.
! /usr/ports/.gitattributes is missing.
! /usr/ports/.gitauthors is missing.
! /usr/ports/.gitignore is missing.
! /usr/ports/.gitmessage is missing.
! /usr/ports/CHANGES is missing.
…
! /usr/ports/x11/zenity/pkg-plist is missing.
gitup: build_repair_command: There are too many files to repair -- please re-clone the repository: Argument list too long
0.786u 0.224s 0:06.30 15.8% 53+175k 113+0io 6pf+0w
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x 2 root wheel 2B Apr 6 11:41 /usr/ports/.
drwxr-xr-x 17 root wheel 17B Jan 3 07:57 /usr/ports/..
root@mowa219-gjp4-8570p:~ # rm /var/db/gitup
rm: /var/db/gitup: is a directory
root@mowa219-gjp4-8570p:~ # rm /var/db/gitup/ports
root@mowa219-gjp4-8570p:~ # ls -hl /var/db/gitup
total 0
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Branch: main
# Action: clone
+ /usr/ports/.arcconfig
+ /usr/ports/.gitattributes
+ /usr/ports/.gitauthors
+ /usr/ports/.gitignore
+ /usr/ports/.gitmessage
+ /usr/ports/CHANGES
…
+ /usr/ports/x11/zenity/pkg-plist
7.054u 12.562s 0:45.50 43.0% 50+3166k 142+217743io 0pf+0w
root@mowa219-gjp4-8570p:~ # time gitup ports
load: 1.72 cmd: gitup 28070 [zio->io_cv] 283.53r 2.56u 3.55s 2% 281096k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8
load: 1.89 cmd: gitup 28070 [zio->io_cv] 693.76r 5.34u 8.63s 3% 326260k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Want: 64d94165652f40041bf71bdf7a775f867e7b80a9
# Branch: main
# Action: pull
* /usr/ports/security/openssl/Makefile
7.403u 10.415s 12:30.76 2.3% 50+165k 144157+77802io 0pf+0w
root@mowa219-gjp4-8570p:~ #
% pkg query '%o %v %R' gitup
net/gitup 0.90_1 unknown-repository
%
That is not correct. There is no generic "quarterly" branch. Each individual quarterly branch gets a unique name in the form yyyyQ[1-4]. The latest quarterly branch is 2021Q1. There is no Q2 branch yet. It's part of the migration:I'm not sure. Seems like it would be a good idea. chrcol, looking at my /usr/local/etc/gitup.conf, I have this for quarterly, but don't know if it is correct...
Code:"quarterly" : { "host" : "github.com", "repository" : "/freebsd/freebsd-ports.git", "branch" : "quarterly", "target" : "/usr/ports", "ignores" : [ "distfiles", "packages",
Not quite. The git.freebsd.org URLs are the canonical URLs used for cloning repositories. The cgit.freenbsd.org URLs are for the Web interface to the Git repositories. They are used to browse the source online and are not to be used for cloning (won't work anyway). See https://github.com/lwhsu/freebsd-git-docs/blob/main/URLs.mdhttps://git.freebsd.org/docs.git redirects to https://cgit.freebsd.org/doc where https://git.FreeBSD.org/doc.git is the primary non-developer address for cloning. I ignore non-working links such as (public-mirror).
https://git.freebsd.org/src.git redirects to https://cgit.freebsd.org/src where https://git.FreeBSD.org/src.git is the primary non-developer address for cloning.
As the transition progresses, I expect https://git.freebsd.org/ports.git to begin redirecting to https://cgit.freebsd.org/ports …
More information needed. AWith an unnaturally clean installation of FreeBSD 12.2-RELEASE-p4, and the local repository up-to-date:
- git pull took less than one second
- pulls with gitup took between eighty-five and one hundred and twenty-five seconds.
git pull
does very little if your local clone is up-to-date with the remote. What are the exact commands you are running? But I suggest you take this subject to its own topic. Jmehr has been very responsive on these forums in the recent past.… What are the exact commands you are running?
gitup ports
– nothing complicated.But I suggest you take this subject to its own topic. …
– the preceding post did link to a reason.… a reason gitup keeps a dabatase. …
Thanks, I needed that.poudriere ports -c -B main -m git -U 'https://git.FreeBSD.org/ports.git' -p MAIN
-U
for the git method?root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default...cannot create 'copperbowl/poudriere/ports/default': dataset already exists
fail
[00:00:00] Error: Failed to create FS copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
cannot unmount '/copperbowl/poudriere/ports/default': pool or dataset is busy
root@mowa219-gjp4-8570p:~ # poudriere ports -l
PORTSTREE METHOD TIMESTAMP PATH
portoverlay - /usr/local/poudriere/ports/portoverlay
root@mowa219-gjp4-8570p:~ # killall gvfsd-trash
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree... done
root@mowa219-gjp4-8570p:~ # cat /usr/local/poudriere/ports/default/.git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = git://github.com/freebsd/freebsd-ports.git
fetch = +refs/heads/main:refs/remotes/origin/main
[branch "main"]
remote = origin
merge = refs/heads/main
root@mowa219-gjp4-8570p:~ # poudriere ports -d default
[00:00:00] Are you sure you want to delete the ports tree default at /usr/local/poudriere/ports/default? [y/N] y
[00:00:02] Deleting portstree "default"... done
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default...cannot create 'copperbowl/poudriere/ports/default': dataset already exists
fail
[00:00:00] Error: Failed to create FS copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # killall gvfsd-trash
No matching processes were found
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # rm -r /usr/local/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # time poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree... done
11.174u 7.791s 1:05.23 29.0% 2879+737k 3024+170812io 1095pf+0w
root@mowa219-gjp4-8570p:~ # cat /usr/local/poudriere/ports/default/.git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = https://git.freebsd.org/ports.git
fetch = +refs/heads/main:refs/remotes/origin/main
[branch "main"]
remote = origin
merge = refs/heads/main
root@mowa219-gjp4-8570p:~ # pkg query '%o %v %R' poudriere-devel
ports-mgmt/poudriere-devel 3.3.99.20210303_1 FreeBSD
root@mowa219-gjp4-8570p:~ #
-U
.)If you don't have any interest in having it fixed,
there wasn't any reason, just assumptions
https://github.com/johnmehr/gitup/blob/2b055c47f4e5f703c7ae81ec5da5e6c41ed0ef6c/gitup.1.in#L129 explains that gitup(1) "… relies on the known remote files lists stored in /var/db/gitup and the current state of the local repository to reconstruct data …".
git pull
does? Everything I've read so far was a bit shallow, to say the least. And if git pull --ff-only
is the recommended way of updating the tree, why isn't it used in /usr/ports/Makefile:. else
@echo "--------------------------------------------------------------"
@echo ">>> Updating ${.CURDIR} from git repository"
@echo "--------------------------------------------------------------"
cd ${.CURDIR}; ${GIT} pull
. endif
--ff-only
will outright refuse to create a merge commit, which would be necessary if you locally added commits to the same branch. Adding commits to a branch you will never push is certainly a bad idea (if you need local commits, use a separate local branch for them, you can rebase it) – but --ff-only
is just a safety measure here: Fail fast instead of writing a local history that diverges from the upstream branch more and more.Thread help-with-migration-to-git.79365/#post-500223Can someone please explain what exactly the --ff-only flag ongit pull
does?