FreeBSD package mirror infrastructure

R

redstreams

Is there a particular reason why FreeBSD has only few package mirrors? For major Linux distros there are dozens of mirrors, same for OpenBSD. It looks like in the last few years a few mirrors were added.
I did not test it but when I tried it last time mirroring was rather slow and I think no mirror provides rsync or some way to check for new packages.
 
I don't see need to increase the amount of mirrors. On Linux you can have dozens of mirrors, but there are also so many more hundreds of users.
See: http://pkg.freebsd.org
The pkg mirror is switched based on your GeoIP Location. You can force it to use another mirror.
See /etc/pkg/FreeBSD.conf

But at your risk, no guarantees of any particular mirror.

I didn't understand your last question about check for new packages. You dont need rsync for this. "pkg update && pkg upgrade" don't satisfies you?
 
The background of the question was to create a local mirror for packages. Having more official mirrors would reduce the load on the current ones.
For mirroring all packages something like rsync would be useful but the current mirrors don't support this. iirc you can't use wget either because the mirror does not support timestamping. Also overall performance of the mirrors seemed to be slow last time I tested it. You can't even browse the package list anymore.
 
docs.freebsd.org

Mirroring FreeBSD

The all in one guide for mirroring the FreeBSD website, FTP servers, and more
docs.freebsd.org docs.freebsd.org
Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages
Click to expand...
Wouldn't using more mirrors actually reduce bandwidth requirements? In particular if other mirrors are synced preferably instead of normal FreeBSD end users?
Storage requirements are about 1TB which seems rather cheap. I don't know about the administration overhead but I *think* for some mirror operator it is mostly running something like rsync to the master mirrors every few hours. Looking at just a few examples like https://mirror.yandex.ru https://ftp.halifax.rwth-aachen.de https://mirror.bytemark.co.uk/ they store everything and the kitchen sink including FreeBSD (minus packages).
 
Parallel to rootbert https://forums.freebsd.org/profile-posts/3507/

Due to very high requirements of bandwidth, storage and adminstration
Click to expand...

Blame can take us to a point in time <https://github.com/freebsd/freebsd-...ion/content/en/articles/hubs/_index.adoc#L165> preceding which it might be unreasonably difficult to discover when, or why, the phrase was added.

Not specific to packages, the note at the head of <https://docs.freebsd.org/en/articles/hubs/> might be more relevant. With added emphasis:

We are not accepting new mirrors at this time.
Click to expand...

Why not? I'm not a member of the team, but I guess that its work in progress <https://www.freebsd.org/status/report-2021-07-2021-09/#_cluster_administration_team> is enough for members to not want attention diverted to anything else at this time.

Glancing at some progress points, another guess:
  • the fewer the mirrors, the easier it will be to coordinate when things change.

redstreams said:
The background of the question was to create a local mirror for packages. …
Click to expand...

forums.freebsd.org

How to make a local mirror for FreeBSD.

I need a local mirror server, because I need an FreeBSD cluster (run ZFS and PostgreSQL). For Debian system, the local mirror can easily be created with the apt-mirror tool. Is there some tools to create a local mirror for FreeBSD to run freebsd-update and pkg commands, so the entire cluster...
forums.freebsd.org forums.freebsd.org

fender0107401 said:
I will find a way to work around it.
Click to expand...

Performance

redstreams said:
… overall performance of the mirrors seemed to be slow last time I tested ?…
Click to expand...

forums.freebsd.org

Slow package fetching

Hello, I've trying to install some packages with pkg install, but the process is too slow today. How can I change the source? I've installed fastest_pkg from ports. But this util did not respond after start? Or Should I wait for it to finish?
forums.freebsd.org forums.freebsd.org

For anyone who's not already aware of this test script:

ports-mgmt/fastest_pkg

– kudos to anyone who's willing to draft a pull request for this enhancement:
Two related FreeBSD bugs (ERs):
 
You must log in or register to reply here.
Back
Top