gitup

grahamperrin

Well-Known Member

Reaction score: 72
Messages: 268

… suggest you take this subject to its own topic. …

👍

… I didn't time the original gitup ports - was very roughly a minute. All this on FreeBSD 12.2p6. …

Thanks. Not significantly different here, where gitup ports clones into an empty (or nearly empty) directory.

With slightly outdated 12.2-RELEASE-p4, around a minute and a half (1:31.66). Marginally quicker with things warmed up a little (the second screenshot):

2021-04-07 01:31 gitup ports 1:31.66.png


2021-04-07 01:41 gitup ports 1:27.17.png


It was easy to get below one minute for clone situations. More memory to the machine, then after deletions and a warm-up: fifty-six seconds for the second run.

2021-04-07 01:53 gitup ports 1:17.85.png


2021-04-07 01:59 gitup ports 0:55.58.png


Where the effect of gitup ports is to pull, I see diverse timings from diverse use cases (diverse hardware, and so on). Some runs are long, some not.

The diversity surprises some people; I find no problem with it. Keyword for any reader: YMMV
 
OP
J

jmehr

Member

Reaction score: 43
Messages: 21

When cloning, the performance of net/gitup should be pretty close to the performance of the official Git client when requesting a shallow clone -- the size of the pack file downloaded will be the same as a --depth 1 git clone but net/gitup runs in a single thread so there will be a performance hit when processing the pack file.

When pulling, however, net/gitup will be much slower than the official Git client. In order to minimize the amount of disk space needed, net/gitup does not retain the pack files that the official Git client needs. Since these pack files contain all the information needed to update a local copy of the repository, a pristine local copy of a repository can be used to reconstruct the missing information that once existed in the pack files and net/gitup must walk the local tree and calculate the checksum for every file to do this.
 

trev

Aspiring Daemon

Reaction score: 257
Messages: 973

The only issue I have is with a FreeBSD 12.2R VPS with 512MB memory and 1.5G swap. gitup runs out of swap and the OOM kills processes including gitup. What Is the minimum amount of memory required?

As a workaround, I'm currently rsync'ing a gitup'ped /usr/pports from another system, so not a big deal.
 
OP
J

jmehr

Member

Reaction score: 43
Messages: 21

I just did a couple of tests to check memory usage and a clone of the ports repository topped out around 545M and a pull (updating from a few days ago) was less than that. Cloning/pulling the source can take up to 2GB of memory.

I just added an issue to gitup's to-do list to implement an option to minimize its memory footprint.
 

grahamperrin

Well-Known Member

Reaction score: 72
Messages: 268

jmehr thanks for all your work in these areas.

I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?
 

scottro

Daemon

Reaction score: 730
Messages: 1,831

I notice that it doesn't create an INDEX file. (or said file isn't in the git repository). I use a package, psearch, to search for ports, that relies upon the INDEX file. I know I've seen something on the forums about said file, but don't remember if it was that the file is broken, removed,or something completely different. An INDEX-x file is also needed for make search in /usr/ports

I am able to run make fetchindex after cloning the ports, (and as I've only used gitup to get ports, not git, I don't know if this is something that would also happen if I used git rather than gitup), so it's not a major issue. Aside from that, the tool is working quite well for me. I
 

Zirias

Daemon

Reaction score: 1,095
Messages: 1,977

scottro see

In a nutshell:
  • An index is never part of the repository
  • A downloaded index can never be fully correct as soon as some options are changed locally
  • Since introduction of port flavors, index creation is broken, so even creating one locally won't guarantee a correct result
Therefore, best avoid anything relying on an index for now.
 

olli@

Daemon
Developer

Reaction score: 1,205
Messages: 1,108

Well, the INDEX file is good enough for using “make search” (and probably for psearch, too, although I don’t know this tool). So it should be safe to fetch the INDEX file or generate it yourself, if it’s just for those purposes.

Alternatively, there are online search engines for ports, e.g. Porgle. Also, FreshPorts has a search feature (although it doesn’t support regular expressions or complex search expressions).
 

scottro

Daemon

Reaction score: 730
Messages: 1,831

olli@ thanks, someone (perhaps you?), had already mentioned porgle. But yes, the only thing I use INDEX for is for psearch. (Since I have psearch,I don' use make search, but mentioned it as more people are familiar with it).
Zirias, thanks for showing me the link, I *knew* I had seen something but couldn't remember what it was.
 
OP
J

jmehr

Member

Reaction score: 43
Messages: 21

jmehr thanks for all your work in these areas.

I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?

I'm ok with either option. Discussing feature requests here does make it easier for more people to join in and contribute ideas, though. :)
 

grahamperrin

Well-Known Member

Reaction score: 72
Messages: 268

One of the ideas appeared already:
– I intended to request a --quiet option.



The overlapping idea is timely attention to relevant parts of UPDATING files.

It's easier to draw a user's attention when things are quiet.

An initial gitup ports clone can conclude with a hint for the user to see:
  • /usr/ports/UPDATING
gitup ports pulls can conclude with a hint for the user to review the same file, only if the pull included an update to the file.

An initial gitup current clone can conclude with a hint for the user to read:
  • /usr/src/UPDATING
– with special attention to common items.

And so on.
 
OP
J

jmehr

Member

Reaction score: 43
Messages: 21

The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).

I like the special attention feature as I must confess that I don't always read /usr/ports/UPDATING as much as I should. ;)

One idea I had when thinking about this was that, since the timestamps of the files in /var/db/gitup let us know when the last time gitup was run, we could also add an option that prints out the portions of /usr/ports/UPDATING since the last run (either as the full text or just the first two lines such as "20210411: AFFECTS: users or devel/py-RPyC").

Thoughts?
 

grahamperrin

Well-Known Member

Reaction score: 72
Messages: 268

The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).

FWIW I imagine minimally verbose (when no UPDATING file is updated) comprising just two lines:

Code:
working …
done.

– with working … apparent immediately after the file system scan commences (context: <https://github.com/johnmehr/gitup/issues/43#issuecomment-813783543>).

I like the special attention feature …

Side note: COMMON ITEMS is partly within the scope of <https://reviews.freebsd.org/D28062>. Given the duplication of effort, and so on, I'll be not surprised if this section eventually disappears … then it'll be unnecessary to draw special attention to the section.
 

grahamperrin

Well-Known Member

Reaction score: 72
Messages: 268

When pulling, … walk the local tree and calculate the checksum for every file to do this.

I wondered whether a simple ZFS cache device would help.

As far as I can tell: the effect, if any, is negligible.

First and second runs with the device online, third run offline:

Code:
root@mowa219-gjp4-8570p:~ # time gitup ports
load: 1.71  cmd: gitup 11603 [zio->io_cv] 640.53r 3.63u 6.07s 0% 284624k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: ce7525e65911cc709c94f07f6b64477e2e35f7a7
# Want: 72b4c9dd9b214e5c8785494ed92fc39568c15603
# Branch: main
# Action: pull
 * /usr/ports/net/quiche/Makefile
 * /usr/ports/net/quiche/distinfo
7.803u 11.844s 20:17.92 1.6%    51+171k 144380+77637io 0pf+0w
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 72b4c9dd9b214e5c8785494ed92fc39568c15603
# Want: 168a3c4e3a51e4fe8e9ef0351c2b4e8008a97452
# Branch: main
# Action: pull
 * /usr/ports/net-p2p/awgg/Makefile
 * /usr/ports/net-p2p/awgg/pkg-plist
8.349u 12.100s 21:16.81 1.6%    50+168k 144394+77637io 0pf+0w
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 168a3c4e3a51e4fe8e9ef0351c2b4e8008a97452
# Want: 1c8ce0fb8ad179fc5b346e3cd92fa61798404132
# Branch: main
# Action: pull
 * /usr/ports/audio/praat/Makefile
 * /usr/ports/audio/praat/distinfo
7.610u 10.826s 22:03.44 1.3%    51+169k 144176+77637io 0pf+0w
root@mowa219-gjp4-8570p:~ # zpool status copperbowl
  pool: copperbowl
 state: ONLINE
  scan: scrub repaired 0B in 01:40:58 with 0 errors on Thu Jan  7 02:36:19 2021
config:

        NAME          STATE     READ WRITE CKSUM
        copperbowl    ONLINE       0     0     0
          ada0p4.eli  ONLINE       0     0     0
        cache
          da1         ONLINE       0     0     0

errors: No known data errors
root@mowa219-gjp4-8570p:~ # freebsd-version -kru
14.0-CURRENT
14.0-CURRENT
14.0-CURRENT
root@mowa219-gjp4-8570p:~ #
 
Top