gitup

grahamperrin

Well-Known Member

Reaction score: 102
Messages: 382

… suggest you take this subject to its own topic. …

👍

… I didn't time the original gitup ports - was very roughly a minute. All this on FreeBSD 12.2p6. …

Thanks. Not significantly different here, where gitup ports clones into an empty (or nearly empty) directory.

With slightly outdated 12.2-RELEASE-p4, around a minute and a half (1:31.66). Marginally quicker with things warmed up a little (the second screenshot):

2021-04-07 01:31 gitup ports 1:31.66.png


2021-04-07 01:41 gitup ports 1:27.17.png


It was easy to get below one minute for clone situations. More memory to the machine, then after deletions and a warm-up: fifty-six seconds for the second run.

2021-04-07 01:53 gitup ports 1:17.85.png


2021-04-07 01:59 gitup ports 0:55.58.png


Where the effect of gitup ports is to pull, I see diverse timings from diverse use cases (diverse hardware, and so on). Some runs are long, some not.

The diversity surprises some people; I find no problem with it. Keyword for any reader: YMMV
 
OP
J

jmehr

Member

Reaction score: 50
Messages: 27

When cloning, the performance of net/gitup should be pretty close to the performance of the official Git client when requesting a shallow clone -- the size of the pack file downloaded will be the same as a --depth 1 git clone but net/gitup runs in a single thread so there will be a performance hit when processing the pack file.

When pulling, however, net/gitup will be much slower than the official Git client. In order to minimize the amount of disk space needed, net/gitup does not retain the pack files that the official Git client needs. Since these pack files contain all the information needed to update a local copy of the repository, a pristine local copy of a repository can be used to reconstruct the missing information that once existed in the pack files and net/gitup must walk the local tree and calculate the checksum for every file to do this.
 

trev

Aspiring Daemon

Reaction score: 260
Messages: 978

The only issue I have is with a FreeBSD 12.2R VPS with 512MB memory and 1.5G swap. gitup runs out of swap and the OOM kills processes including gitup. What Is the minimum amount of memory required?

As a workaround, I'm currently rsync'ing a gitup'ped /usr/pports from another system, so not a big deal.
 
OP
J

jmehr

Member

Reaction score: 50
Messages: 27

I just did a couple of tests to check memory usage and a clone of the ports repository topped out around 545M and a pull (updating from a few days ago) was less than that. Cloning/pulling the source can take up to 2GB of memory.

I just added an issue to gitup's to-do list to implement an option to minimize its memory footprint.
 

grahamperrin

Well-Known Member

Reaction score: 102
Messages: 382

jmehr thanks for all your work in these areas.

I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?
 

scottro

Daemon

Reaction score: 767
Messages: 1,878

I notice that it doesn't create an INDEX file. (or said file isn't in the git repository). I use a package, psearch, to search for ports, that relies upon the INDEX file. I know I've seen something on the forums about said file, but don't remember if it was that the file is broken, removed,or something completely different. An INDEX-x file is also needed for make search in /usr/ports

I am able to run make fetchindex after cloning the ports, (and as I've only used gitup to get ports, not git, I don't know if this is something that would also happen if I used git rather than gitup), so it's not a major issue. Aside from that, the tool is working quite well for me. I
 

Zirias

Daemon

Reaction score: 1,181
Messages: 2,140

scottro see

In a nutshell:
  • An index is never part of the repository
  • A downloaded index can never be fully correct as soon as some options are changed locally
  • Since introduction of port flavors, index creation is broken, so even creating one locally won't guarantee a correct result
Therefore, best avoid anything relying on an index for now.
 

olli@

Daemon
Developer

Reaction score: 1,214
Messages: 1,108

Well, the INDEX file is good enough for using “make search” (and probably for psearch, too, although I don’t know this tool). So it should be safe to fetch the INDEX file or generate it yourself, if it’s just for those purposes.

Alternatively, there are online search engines for ports, e.g. Porgle. Also, FreshPorts has a search feature (although it doesn’t support regular expressions or complex search expressions).
 

scottro

Daemon

Reaction score: 767
Messages: 1,878

olli@ thanks, someone (perhaps you?), had already mentioned porgle. But yes, the only thing I use INDEX for is for psearch. (Since I have psearch,I don' use make search, but mentioned it as more people are familiar with it).
Zirias, thanks for showing me the link, I *knew* I had seen something but couldn't remember what it was.
 
OP
J

jmehr

Member

Reaction score: 50
Messages: 27

jmehr thanks for all your work in these areas.

I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?

I'm ok with either option. Discussing feature requests here does make it easier for more people to join in and contribute ideas, though. :)
 

grahamperrin

Well-Known Member

Reaction score: 102
Messages: 382

One of the ideas appeared already:
– I intended to request a --quiet option.



The overlapping idea is timely attention to relevant parts of UPDATING files.

It's easier to draw a user's attention when things are quiet.

An initial gitup ports clone can conclude with a hint for the user to see:
  • /usr/ports/UPDATING
gitup ports pulls can conclude with a hint for the user to review the same file, only if the pull included an update to the file.

An initial gitup current clone can conclude with a hint for the user to read:
  • /usr/src/UPDATING
– with special attention to common items.

And so on.
 
OP
J

jmehr

Member

Reaction score: 50
Messages: 27

The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).

I like the special attention feature as I must confess that I don't always read /usr/ports/UPDATING as much as I should. ;)

One idea I had when thinking about this was that, since the timestamps of the files in /var/db/gitup let us know when the last time gitup was run, we could also add an option that prints out the portions of /usr/ports/UPDATING since the last run (either as the full text or just the first two lines such as "20210411: AFFECTS: users or devel/py-RPyC").

Thoughts?
 

grahamperrin

Well-Known Member

Reaction score: 102
Messages: 382

The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).

FWIW I imagine minimally verbose (when no UPDATING file is updated) comprising just two lines:

Code:
working …
done.

– with working … apparent immediately after the file system scan commences (context: <https://github.com/johnmehr/gitup/issues/43#issuecomment-813783543>).

I like the special attention feature …

Side note: COMMON ITEMS is partly within the scope of <https://reviews.freebsd.org/D28062>. Given the duplication of effort, and so on, I'll be not surprised if this section eventually disappears … then it'll be unnecessary to draw special attention to the section.
 

edinilsonjs

New Member


Messages: 4

Something is wrong with IPv6 implementation because if IPv6 isn´t compiled in kernel or is disabled, the following error is raised in FreeBSD 13.0 RELEASE:
gitup: connect_server: socket failure: Address family not supported by protocol family
 

bagas

Active Member

Reaction score: 5
Messages: 217

Error!
# gitup ports
gitup: connect_server: socket failure: Address family not supported by protocol family

ipv6 is completely removed from my system, I don't want to turn it on, I don't need it !!!
How to update ports is now not clear!

Previously, there were no problems with updating ports using gitup.
Why it was implemented via ipv6 is unclear.
We'll have to install a full-fledged git!
Gitup delete to hell!
 

covacat

Active Member

Reaction score: 48
Messages: 113

Error!


ipv6 is completely removed from my system, I don't want to turn it on, I don't need it !!!
How to update ports is now not clear!

Previously, there were no problems with updating ports using gitup.
Why it was implemented via ipv6 is unclear.
We'll have to install a full-fledged git!
Gitup delete to hell!
try to set the ipv4 address in the config file (instead of git.freebsd.org)
 

bagas

Active Member

Reaction score: 5
Messages: 217

try to set the ipv4 address in the config file (instead of git.freebsd.org)
Created a ticket.

# host git.freebsd.org
git.freebsd.org is an alias for gitmir.geo.freebsd.org.
gitmir.geo.freebsd.org has address 139.178.72.204
gitmir.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::e6a:1
gitmir.geo.freebsd.org mail is handled by 0 .

# telnet 139.178.72.204 443
Trying 139.178.72.204...
Connected to gitmir.pkt.freebsd.org.
Escape character is '^]'.

q
HTTP/1.1 400 Bad Request
Server: nginx/1.18.0
Date: Thu, 06 May 2021 07:14:28 GMT
Content-Type: text/html
Content-Length: 157
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
Connection closed by foreign host.


I assume that the problem is in the IPv4 address (139.178.72.204).
IPv6 in my system is completely removed, I do not need IPv6.
I will not build the system with ipv6!
 

bagas

Active Member

Reaction score: 5
Messages: 217

try to set the ipv4 address in the config file (instead of git.freebsd.org)
Specified the ipv4 address in place of the domain, the ports were updated.
Strange, then the address selection priority (gitmir.geo) stands firmly ipv6?

gitup.conf
"ports" : {
# "host" : "git.freebsd.org",
"host" : "139.178.72.204",
 

covacat

Active Member

Reaction score: 48
Messages: 113

Specified the ipv4 address in place of the domain, the ports were updated.
Strange, then the address selection priority (gitmir.geo) stands firmly ipv6?

gitup.conf
"ports" : {
# "host" : "git.freebsd.org",
"host" : "139.178.72.204",
looking at the source i saw it uses getaddrinfo which returns all addresses of the host regardles of the family
and if for some reason the inet6 address comes first and you don't support ipv6 you have a problem
 

bagas

Active Member

Reaction score: 5
Messages: 217

gitup-0.92 no more problem.

"ports" : {
"host" : "git.freebsd.org",
"repository" : "/ports.git",
"branch" : "main",
 

Zirias

Daemon

Reaction score: 1,181
Messages: 2,140

IPv6 in my system is completely removed, I do not need IPv6.
I will not build the system with ipv6!
This is pretty short-sighted. We're already at the point where IPv4 cannot support the "whole internet" any more, ISPs provide IPv4 via CGNAT or DSLITE to customers (sharing a single address for many of them) and there are quite some hosts out there reachable only via IPv6 (most notable here the "beefy" machines building official FreeBSD packages). This doesn't come as a surprise, we all knew 32bit addressing won't hold …

But then:
looking at the source i saw it uses getaddrinfo which returns all addresses of the host regardles of the family
and if for some reason the inet6 address comes first and you don't support ipv6 you have a problem
This is a bug anyways. Using getaddrinfo(3) and looping over the results is the correct approach of course, but any error trying to create a socket(2) or connecting on it should be ignored, trying the next result instead. An error should only be thrown after all results failed.
 
Top