gitup

[Cath O'Deray]

… suggest you take this subject to its own topic. …

?

… I didn't time the original gitup ports - was very roughly a minute. All this on FreeBSD 12.2p6. …

Thanks. Not significantly different here, where gitup ports clones into an empty (or nearly empty) directory.

With slightly outdated 12.2-RELEASE-p4, around a minute and a half (1:31.66). Marginally quicker with things warmed up a little (the second screenshot):

It was easy to get below one minute for clone situations. More memory to the machine, then after deletions and a warm-up: fifty-six seconds for the second run.

Where the effect of gitup ports is to pull, I see diverse timings from diverse use cases (diverse hardware, and so on). Some runs are long, some not.

The diversity surprises some people; I find no problem with it. Keyword for any reader: YMMV

 

[jmehr]

When cloning, the performance of net/gitup should be pretty close to the performance of the official Git client when requesting a shallow clone -- the size of the pack file downloaded will be the same as a --depth 1 git clone but net/gitup runs in a single thread so there will be a performance hit when processing the pack file.

When pulling, however, net/gitup will be much slower than the official Git client. In order to minimize the amount of disk space needed, net/gitup does not retain the pack files that the official Git client needs. Since these pack files contain all the information needed to update a local copy of the repository, a pristine local copy of a repository can be used to reconstruct the missing information that once existed in the pack files and net/gitup must walk the local tree and calculate the checksum for every file to do this.

 

[trev]

The only issue I have is with a FreeBSD 12.2R VPS with 512MB memory and 1.5G swap. gitup runs out of swap and the OOM kills processes including gitup. What Is the minimum amount of memory required?

As a workaround, I'm currently rsync'ing a gitup'ped /usr/pports from another system, so not a big deal.

 

[jmehr]

I just did a couple of tests to check memory usage and a clone of the ports repository topped out around 545M and a pull (updating from a few days ago) was less than that. Cloning/pulling the source can take up to 2GB of memory.

I just added an issue to gitup's to-do list to implement an option to minimize its memory footprint.

 

[Cath O'Deray]

jmehr thanks for all your work in these areas.

I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?

 

[scottro]

I notice that it doesn't create an INDEX file. (or said file isn't in the git repository). I use a package, psearch, to search for ports, that relies upon the INDEX file. I know I've seen something on the forums about said file, but don't remember if it was that the file is broken, removed,or something completely different. An INDEX-x file is also needed for make search in /usr/ports

I am able to run make fetchindex after cloning the ports, (and as I've only used gitup to get ports, not git, I don't know if this is something that would also happen if I used git rather than gitup), so it's not a major issue. Aside from that, the tool is working quite well for me. I

 

[Cath O'Deray]

… something on the forums …

Maybe [ports][GIT][portsnap] Warning about INDEX

 

[zirias@]

scottro see

[ports][GIT][portsnap] Warning about INDEX

In the context of ports moving to GIT, as well as (expected) deprecation of portsnap, I've seen some questions regarding INDEX, which was automatically fetched by portsnap. Right now, I learned the following in an IRC channel full of devs: < XXXXXX> INDEX is currently dangerously misleading...

forums.freebsd.org

In a nutshell:

• An index is never part of the repository
• A downloaded index can never be fully correct as soon as some options are changed locally
• Since introduction of port flavors, index creation is broken, so even creating one locally won't guarantee a correct result

Therefore, best avoid anything relying on an index for now.

 

[olli@]

Well, the INDEX file is good enough for using “make search” (and probably for psearch, too, although I don’t know this tool). So it should be safe to fetch the INDEX file or generate it yourself, if it’s just for those purposes.

Alternatively, there are online search engines for ports, e.g. Porgle. Also, FreshPorts has a search feature (although it doesn’t support regular expressions or complex search expressions).

 

[scottro]

olli@ thanks, someone (perhaps you?), had already mentioned porgle. But yes, the only thing I use INDEX for is for psearch. (Since I have psearch,I don' use make search, but mentioned it as more people are familiar with it).
Zirias, thanks for showing me the link, I *knew* I had seen something but couldn't remember what it was.

 

[jmehr]

jmehr thanks for all your work in these areas.

I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?

I'm ok with either option. Discussing feature requests here does make it easier for more people to join in and contribute ideas, though.

 

[Cath O'Deray]

One of the ideas appeared already:

• Less verbose option? · Issue #59 · johnmehr/gitup

– I intended to request a --quiet option.

---

The overlapping idea is timely attention to relevant parts of UPDATING files.

It's easier to draw a user's attention when things are quiet.

An initial gitup ports clone can conclude with a hint for the user to see:

• /usr/ports/UPDATING

gitup ports pulls can conclude with a hint for the user to review the same file, only if the pull included an update to the file.

An initial gitup current clone can conclude with a hint for the user to read:

• /usr/src/UPDATING

– with special attention to common items.

And so on.

 

[jmehr]

The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).

I like the special attention feature as I must confess that I don't always read /usr/ports/UPDATING as much as I should.

One idea I had when thinking about this was that, since the timestamps of the files in /var/db/gitup let us know when the last time gitup was run, we could also add an option that prints out the portions of /usr/ports/UPDATING since the last run (either as the full text or just the first two lines such as "20210411: AFFECTS: users or devel/py-RPyC").

Thoughts?

 

[Cath O'Deray]

The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).

FWIW I imagine minimally verbose (when no UPDATING file is updated) comprising just two lines:

working …
done.

– with working … apparent immediately after the file system scan commences (context: <{link removed}>).

I like the special attention feature …

Side note: COMMON ITEMS is partly within the scope of <{link removed}>. Given the duplication of effort, and so on, I'll be not surprised if this section eventually disappears … then it'll be unnecessary to draw special attention to the section.

 

[Cath O'Deray]

When pulling, … walk the local tree and calculate the checksum for every file to do this.

I wondered whether a simple ZFS cache device would help.

(Detail moved to {link removed}.)

 

[Cath O'Deray]

jmehr FYI {link removed}.

 

[edinilsonjs]

Something is wrong with IPv6 implementation because if IPv6 isn´t compiled in kernel or is disabled, the following error is raised in FreeBSD 13.0 RELEASE:
gitup: connect_server: socket failure: Address family not supported by protocol family

 

[jmehr]

jmehr FYI Problems with gitup - doesn't delete files / old ports.

Thank you for letting me know about this one! I'm not currently subscribed to the freebsd-ports list and I just sent him an email asking for more information.

 

[bagas]

Error!

# gitup ports
gitup: connect_server: socket failure: Address family not supported by protocol family

ipv6 is completely removed from my system, I don't want to turn it on, I don't need it !!!
How to update ports is now not clear!

Previously, there were no problems with updating ports using gitup.
Why it was implemented via ipv6 is unclear.
We'll have to install a full-fledged git!
Gitup delete to hell!

 

[covacat]

Error!


ipv6 is completely removed from my system, I don't want to turn it on, I don't need it !!!
How to update ports is now not clear!

Previously, there were no problems with updating ports using gitup.
Why it was implemented via ipv6 is unclear.
We'll have to install a full-fledged git!
Gitup delete to hell!

try to set the ipv4 address in the config file (instead of git.freebsd.org)

 

[bagas]

try to set the ipv4 address in the config file (instead of git.freebsd.org)

Created a ticket.

255649 – gitup: connect_server: socket failure: Address family not supported by protocol family

bugs.freebsd.org

# host git.freebsd.org
git.freebsd.org is an alias for gitmir.geo.freebsd.org.
gitmir.geo.freebsd.org has address 139.178.72.204
gitmir.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::e6a:1
gitmir.geo.freebsd.org mail is handled by 0 .

# telnet 139.178.72.204 443
Trying 139.178.72.204...
Connected to gitmir.pkt.freebsd.org.
Escape character is '^]'.

q
HTTP/1.1 400 Bad Request
Server: nginx/1.18.0
Date: Thu, 06 May 2021 07:14:28 GMT
Content-Type: text/html
Content-Length: 157
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
Connection closed by foreign host.


I assume that the problem is in the IPv4 address (139.178.72.204).
IPv6 in my system is completely removed, I do not need IPv6.
I will not build the system with ipv6!

 

[bagas]

try to set the ipv4 address in the config file (instead of git.freebsd.org)

Specified the ipv4 address in place of the domain, the ports were updated.
Strange, then the address selection priority (gitmir.geo) stands firmly ipv6?

gitup.conf
"ports" : {
# "host" : "git.freebsd.org",
"host" : "139.178.72.204",

 

[covacat]

Specified the ipv4 address in place of the domain, the ports were updated.
Strange, then the address selection priority (gitmir.geo) stands firmly ipv6?

gitup.conf
"ports" : {
# "host" : "git.freebsd.org",
"host" : "139.178.72.204",

looking at the source i saw it uses getaddrinfo which returns all addresses of the host regardles of the family
and if for some reason the inet6 address comes first and you don't support ipv6 you have a problem

 

[bagas]

gitup-0.92 no more problem.

"ports" : {
"host" : "git.freebsd.org",
"repository" : "/ports.git",
"branch" : "main",

 

[zirias@]

IPv6 in my system is completely removed, I do not need IPv6.
I will not build the system with ipv6!

This is pretty short-sighted. We're already at the point where IPv4 cannot support the "whole internet" any more, ISPs provide IPv4 via CGNAT or DSLITE to customers (sharing a single address for many of them) and there are quite some hosts out there reachable only via IPv6 (most notable here the "beefy" machines building official FreeBSD packages). This doesn't come as a surprise, we all knew 32bit addressing won't hold …

But then:

looking at the source i saw it uses getaddrinfo which returns all addresses of the host regardles of the family
and if for some reason the inet6 address comes first and you don't support ipv6 you have a problem

This is a bug anyways. Using getaddrinfo(3) and looping over the results is the correct approach of course, but any error trying to create a socket(2) or connecting on it should be ignored, trying the next result instead. An error should only be thrown after all results failed.