What is proper method to create a local mirror of pkg package repositories now that pkg.freebsd.org returns 401 when attempting to sync

A

auralox

Hi Freebsd Forum community,

We run a large installation of Freebsd 11 and 12 servers that have historically utilized a local FreeBSD package repository mirror on-premise that we sync periodically. The method we have been using for years -- since the official repositories were poisoned a few years ago -- no longer works due to the HTTP policy change on the pkg.freebsd.org servers. We have been trying to understand the new policy but I've not been able to find much. The best quote of the "non-answer" I've found of the about the changed policy is below (source):

Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the pkg(8) process. Alternatively specific packages and their dependencies can be fetched by running something like the following:
Click to expand...

We have a pretty diverse environment of FreeBSD hosts and generations and the current recommendation to manually download and sync all the packages we use like this with the `pkg` tool is not a scalable approach. [In fact the only way I can conceive of to do this would be to analyze our configuration management system and build a list of packages from there?) Although we have a working poudriere environment I'd prefer not to build the whole ports tree myself. The method I see in the current manual as the recommended -- untenable -- approach included with the anwer quoted above is to do the following for every package to get it and its dependencies:

Code: 
% pkg fetch -d -o /usr/local/mirror vim

Does anyone have a solution for us other than either running the above command for every package we need to install or building the whole ports tree with Poudreire? Or am I missing something? I'm open to ideas and would really love to hear any other solutions folks have working.

Thanks in advance,
Josh
 
auralox said:
Does anyone have a solution for us other than either running the above command for every package we need to install
Click to expand...

There is no need to run the command for every package separately, you can compile a list (prime-list) of packages as you mentioned it earlier and let pkg(8) read that list, e.g.: pkg fetch -y -d -o /usr/local/mirror `cat prime-list`.

The list can be expanded by running pkg prime-list > prime-list on machines having packages installed. You need also two separate systems, one for 11 and 12. The packages are downloaded according the systems ABI.

auralox said:
or building the whole ports tree with Poudreire?
Click to expand...
Do you need the whole ports tree, categories like arabic, astro, chinese, games, etc.? Other solution except those already mentioned are none with the official repositories.
 
Thanks chrbr and T-Daemon, I appreciate the feedback. However, neither of your answer's are particularly encouraging.

my current solution using www/squid as a caching proxy
Click to expand...

Although a SQUID proxy would be helpful if we were concerned about bandwidth, the technical requirement here is to have a static package repository not to use local caching or bandwidth limitations. In fact, the reason we don't use the public repositories to begin with is that we don't want hosts that were built or patched at different times to be running different minor versions of software across our fleet. I don't see how we can even "fake" having static package versions with this approach, although it would get us a local copy of the packages at least.

There is no need to run the command for every package separately, you can compile a list (prime-list) of packages as you mentioned it earlier and let pkg(8) read that list, e.g.: pkg fetch -y -d -o /usr/local/mirror `cat prime-list`.
Click to expand...

Yeah, I can see why this feature got added to pkg, but this is the antitheses of a fully baked "Enterprise" solution. It adds so many steps to installing a package in a well managed environment! Let me walk you through this: If we want to add packages to a host via configuration management we'd first have to sync the package manually to the package server AND re-generate the metadata before we even try to install it? I'm flummoxed how anyone thought this was a sustainable solution for enterprise clients.

I mean I guess we have to move to a caching proxy, but this is the kind of stuff that makes me want to just stop running FreeBSD entirely.

Disappointed,
Josh
 
auralox said:
this is the antitheses of a fully baked "Enterprise" solution

but this is the kind of stuff that makes me want to just stop running FreeBSD entirely.

Disappointed,
Josh
Click to expand...

Josh, have you ever resolve any of your issues or did you decide to drop FreeBSD as a platform after all?
 
You must log in or register to reply here.
Back
Top