What is proper method to create a local mirror of pkg package repositories now that pkg.freebsd.org returns 401 when attempting to sync

A

auralox

Hi Freebsd Forum community,

We run a large installation of Freebsd 11 and 12 servers that have historically utilized a local FreeBSD package repository mirror on-premise that we sync periodically. The method we have been using for years -- since the official repositories were poisoned a few years ago -- no longer works due to the HTTP policy change on the pkg.freebsd.org servers. We have been trying to understand the new policy but I've not been able to find much. The best quote of the "non-answer" I've found of the about the changed policy is below (source):

Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the pkg(8) process. Alternatively specific packages and their dependencies can be fetched by running something like the following:
Click to expand...

We have a pretty diverse environment of FreeBSD hosts and generations and the current recommendation to manually download and sync all the packages we use like this with the `pkg` tool is not a scalable approach. [In fact the only way I can conceive of to do this would be to analyze our configuration management system and build a list of packages from there?) Although we have a working poudriere environment I'd prefer not to build the whole ports tree myself. The method I see in the current manual as the recommended -- untenable -- approach included with the anwer quoted above is to do the following for every package to get it and its dependencies:

Code: 
% pkg fetch -d -o /usr/local/mirror vim

Does anyone have a solution for us other than either running the above command for every package we need to install or building the whole ports tree with Poudreire? Or am I missing something? I'm open to ideas and would really love to hear any other solutions folks have working.

Thanks in advance,
Josh
 
auralox said:
Does anyone have a solution for us other than either running the above command for every package we need to install
Click to expand...

There is no need to run the command for every package separately, you can compile a list (prime-list) of packages as you mentioned it earlier and let pkg(8) read that list, e.g.: pkg fetch -y -d -o /usr/local/mirror `cat prime-list`.

The list can be expanded by running pkg prime-list > prime-list on machines having packages installed. You need also two separate systems, one for 11 and 12. The packages are downloaded according the systems ABI.

auralox said:
or building the whole ports tree with Poudreire?
Click to expand...
Do you need the whole ports tree, categories like arabic, astro, chinese, games, etc.? Other solution except those already mentioned are none with the official repositories.
 
Thanks chrbr and T-Daemon, I appreciate the feedback. However, neither of your answer's are particularly encouraging.

my current solution using www/squid as a caching proxy
Click to expand...

Although a SQUID proxy would be helpful if we were concerned about bandwidth, the technical requirement here is to have a static package repository not to use local caching or bandwidth limitations. In fact, the reason we don't use the public repositories to begin with is that we don't want hosts that were built or patched at different times to be running different minor versions of software across our fleet. I don't see how we can even "fake" having static package versions with this approach, although it would get us a local copy of the packages at least.

There is no need to run the command for every package separately, you can compile a list (prime-list) of packages as you mentioned it earlier and let pkg(8) read that list, e.g.: pkg fetch -y -d -o /usr/local/mirror `cat prime-list`.
Click to expand...

Yeah, I can see why this feature got added to pkg, but this is the antitheses of a fully baked "Enterprise" solution. It adds so many steps to installing a package in a well managed environment! Let me walk you through this: If we want to add packages to a host via configuration management we'd first have to sync the package manually to the package server AND re-generate the metadata before we even try to install it? I'm flummoxed how anyone thought this was a sustainable solution for enterprise clients.

I mean I guess we have to move to a caching proxy, but this is the kind of stuff that makes me want to just stop running FreeBSD entirely.

Disappointed,
Josh
 
auralox said:
this is the antitheses of a fully baked "Enterprise" solution

but this is the kind of stuff that makes me want to just stop running FreeBSD entirely.

Disappointed,
Josh
Click to expand...

Josh, have you ever resolve any of your issues or did you decide to drop FreeBSD as a platform after all?
 
T-Daemon said:
There is no need to run the command for every package separately, you can compile a list (prime-list) of packages as you mentioned it earlier and let pkg(8) read that list, e.g.: pkg fetch -y -d -o /usr/local/mirror `cat prime-list`.

The list can be expanded by running pkg prime-list > prime-list on machines having packages installed. You need also two separate systems, one for 11 and 12. The packages are downloaded according the systems ABI.


Do you need the whole ports tree, categories like arabic, astro, chinese, games, etc.? Other solution except those already mentioned are none with the official repositories.
Click to expand...
Just came across this post as I'm hoping to set up local repository on my LAN.

Assuming I run pkg fetch -y -d -o /usr/local/mirror `cat prime-list` on my server, what do I need to do on my workstations to use that repository? I guess I need to create a/usr/local/etc/pkg/repos/FreeBSD.conf but what should I put in that file to access the local repository?
 
If you want to create an actual repository from a directory with a bunch of packages, you're going to need to run pkg-repo(8) too. That will create the necessary meta data that makes it a repository. Without this meta data it's just a directory filled with packages.
 
balanga said:
Assuming I run pkg fetch -y -d -o /usr/local/mirror `cat prime-list` on my server, what do I need to do on my workstations to use that repository? I guess I need to create a/usr/local/etc/pkg/repos/FreeBSD.conf but what should I put in that file to access the local repository?
Click to expand...

This is the method I use at the moment to create and update a local package repository, with packages fetched from the official servers.

I do this for two reasons: The main reason is I connect to the internet by mobile data, and my data plan is limited. Having multiple systems as consumer, downloading the same package files multiple times would deplete the data quota unnecessarily.

The second reason is, fetching files from a local repository is much faster than from the internet, almost instantaneously.

I NFS export the package repository to distribute packages on a home LAN (bare metal and virtual machines). This can be done with a web server as well, but NFS is already in the base system, no 3rd party package needed, and easy to configure.

The package repository configuration file could be like as follows (no signature is used on the private LAN. If you want signature, see pkg-repo(8) EXAMPLES):
Code: 
FreeBSD: { enabled: no }

14: {
  url: "file:///usr/local/packages14/${ABI}/latest",
  enabled: yes
}

The url: "file:/// can be a directory on a local file system or a mount point of a NFS. /usr/local/packages14 can be NFS exported and mounted on the /usr/local/packages14 mount point of the remote system.

That's the easy part. A package repository directory hierarchy which pkg(8) expects must be created next.
Code: 
 # mkdir -p /usr/local/packages14/FreeBSD:14:amd64/latest/

Next step is to fetch repository files from the official package server. I use a script to fetch them every time the repository is updated.

/usr/local/packages14/FreeBSD:14:amd64/latest/fetch-repo
Code: 
#!/bin/sh
# fetch repository files
fetch https://pkg.freebsd.org/FreeBSD:14:amd64/latest/data.pkg
fetch https://pkg.freebsd.org/FreeBSD:14:amd64/latest/meta
fetch https://pkg.freebsd.org/FreeBSD:14:amd64/latest/meta.conf
fetch https://pkg.freebsd.org/FreeBSD:14:amd64/latest/packagesite.pkg

Next, create symlinks (this step is done only once):
Code: 
 # pwd
/usr/local/packages14/FreeBSD:14:amd64/latest/
 # ln -s data.pkg   data.txz
 # ln -s packagesite.pkg   packagesite.txz

pkg-fetch(8) the packages you want to install in that directory, including ports-mgmt/pkg package. A "All" directory is created automatically to drop the fetched packages inside.

Create "Latest" directory for ports-mgmt/pkg package. That's the directory base system /usr/sbin/pkg bootstraps ports-mgmt/pkg from:
Code: 
 # mkdir Latest
 # cp All/pkg-1.21.3.pkg   Latest/pkg.pkg
 # ln -s Latest/pkg.pkg   Latest/pkg.txz
In case pkg-1.21.3.pkg is upgraded, don't forget to upgrade Latest/pkg.pkg as well.

I don't use pkg-repo(8), the method I use makes upgrading packages easier.


How to upgrade the package repository?

Run fetch-repo-files script
Code: 
 # pwd
/usr/local/packages14/FreeBSD:14:amd64/latest
 # sh fetch-repo

Create a package upgrade list:
Code: 
 # pkg upgrade -n > /tmp/pkgup

Manually clean up list (I've shorted the list in case you wounder about the package numbers).
To clean up I use editors/neovim global delete functions.

/tmp/pkgup
Code: 
Updating local repository catalogue...
Fetching data.pkg: .......... done
Processing entries: .......... done
local repository update completed. 37696 packages processed.
All repositories are up to date.
Checking for upgrades (64 candidates): .......... done
Processing candidates (64 candidates): .......... done
The following 81 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    kf6-kwindowsystem: 6.3.0
    libfm-qt6: 2.0.2

Installed packages to be UPGRADED:
    aom: 3.9.0 -> 3.9.1
    curl: 8.7.1 -> 8.8.0

Installed packages to be REINSTALLED:
    llvm15-15.0.7_10 (direct dependency changed: python311)

Number of packages to be installed: 21
Number of packages to be upgraded: 58
Number of packages to be reinstalled: 2

The process will require 423 MiB more space.
391 MiB to be downloaded.

Cleaned up package list:
Code: 
kf6-kwindowsystem
libfm-qt6
aom
curl
llvm15

Fetch upgraded packages in repository:
Code: 
 # pkg fetch -o /usr/local/packages14/FreeBSD:14:amd64/latest/ `cat /tmp/pkgup`

The same procedure can be applied in installing packages not present in repository:
Code: 
 # pkg install <package> -n > /tmp/pkginstall
Clean up file, pkg fetch -o /usr/local/packages14/FreeBSD:14:amd64/latest `cat /tmp/pkginstall`

In time the repository fills up with duplicate packages with different versions. The duplicate lesser versions can be deleted manually with sysutils/ncdu or scripted ( see https://forums.freebsd.org/threads/guide-building-a-package-repository-with-portmaster.68179/ )

This upgrade method, involving manually editing fetch list is not particularly efficient, if someone has a better idea, I'd be happy to hear about it.
 
T-Daemon said:
Code: 
mkdir -p /usr/local/packages14/FreeBSD:14:amd64/latest/
Click to expand...
Structure should look be this:
Code: 
dice@chibacity:/usr/local/poudriere/data/packages/141-release-server % ls -g
total 5
lrwxr-xr-x  1 wheel 11 Jun  8 19:21 All -> .latest/All
lrwxr-xr-x  1 wheel 14 Jun  8 19:21 Latest -> .latest/Latest
lrwxr-xr-x  1 wheel 16 Jun  8 19:21 data.pkg -> .latest/data.pkg
lrwxr-xr-x  1 wheel 16 Jun  8 19:21 data.txz -> .latest/data.txz
lrwxr-xr-x  1 wheel 12 Jun  8 19:21 logs -> .latest/logs
lrwxr-xr-x  1 wheel 12 Jun  8 19:21 meta -> .latest/meta
lrwxr-xr-x  1 wheel 17 Jun  8 19:21 meta.conf -> .latest/meta.conf
lrwxr-xr-x  1 wheel 23 Jun  8 19:21 packagesite.pkg -> .latest/packagesite.pkg
lrwxr-xr-x  1 wheel 23 Jun  8 19:21 packagesite.txz -> .latest/packagesite.txz
The symlinks to .latest are due to the way poudriere saves previously built repositories. The All/ directory has all the packages. The Latest directory contains symlinks to packages in the All directory but have their version numbers removed from the package name.
Code: 
dice@chibacity:/usr/local/poudriere/data/packages/141-release-server % ls -al All/pkg-1.21.3.pkg
-rw-r--r--  1 root wheel 12323026 Jun  8 15:13 All/pkg-1.21.3.pkg
dice@chibacity:/usr/local/poudriere/data/packages/141-release-server % ls -al Latest/pkg.pkg
lrwxr-xr-x  1 root wheel 21 Jun  8 15:13 Latest/pkg.pkg -> ../All/pkg-1.21.3.pkg

This All and Latest construct still stems from the old pkg_* tools. If you only have a subset of packages then I recommend using pkg-repo(8). If you copy the meta data from the official servers a pkg search will tell you a certain package exists but may not actually be there. The pkg-repo(8) command will generate the proper catalog files for the packages that are actually in this repository.
 
SirDice said:
The Latest directory contains symlinks to packages in the All directory but have their version numbers removed from the package name.
Click to expand...
Not on the packages repository servers:

Index of /FreeBSD:14:amd64/latest/Latest/
Code: 
Index of /FreeBSD:14:amd64/latest/Latest/

File Name  ↓                              File Size  ↓                        Date  ↓
Parent directory/    -    -
pkg-devel.pkg                             21805428                            2024-May-21 21:43
pkg.pkg                                   12345800                            2024-May-23 01:06
pkg.pkg.sig                               727                                 2024-May-24 03:04
What you mention could be a feature of poudriere.

It looks like this step isn't necessary anymore:
T-Daemon said:
Code: 
# ln -s Latest/pkg.pkg    Latest/pkg.txz
Click to expand...
and there is a pkg-devel.pkg, didn't know that.
 
T-Daemon said:
Not on the packages repository servers:
Click to expand...
T-Daemon said:
What you mention could be a feature of poudriere.
Click to expand...
Well, the official repositories are created using poudriere. But yeah, I should have looked a bit further:
Code: 
dice@chibacity:/usr/local/poudriere/data/packages/141-release-server % ll Latest/
total 1
lrwxr-xr-x  1 root wheel 21 Jun  8 15:13 pkg.pkg@ -> ../All/pkg-1.21.3.pkg

That Latest directory is a leftover from the old pkg_tools. You could do pkg_add -r somepackage without having to specify the exact version. For today's PKGNG repositories I suspect it's only being done for pkg(8) to make it easier for pkg(7) to bootstrap. So it doesn't have to check what's available and can simply fetch ${PACKAGE_SITE}/Latest/pkg.pkg.
 
SirDice said:
If you want to create an actual repository from a directory with a bunch of packages, you're going to need to run pkg-repo(8) too. That will create the necessary meta data that makes it a repository. Without this meta data it's just a directory filled with packages.
Click to expand...
Can you give me an example of using pkg repo to create a local repo in /usr/local/etc/pkg/repos?
 
Code: 
myrepo: {
  enabled: yes
  url: "http://my-package-server.example.com/packages/some-directory
}

You can also use file:// URLs if you have the repository locally. I use this on the poudriere server itself for example:
Code: 
myrepo: {
  enabled: yes
  url: "file:///some/directory/with/pkg/repo"
}
 
Nit:

SirDice said:
url: "http://my-package-server.example.com/packages/some-directory
Click to expand...

– there should be a trailing " (pkg is not forgiving of some things, but not this).

Hint: use file(1) to tell whether content is JSON text data.

Example:

Code: 
% file /etc/pkg/FreeBSD-ports.conf
/etc/pkg/FreeBSD-ports.conf: symbolic link to /etc/pkg/FreeBSD.conf
% file /etc/pkg/FreeBSD.conf
/etc/pkg/FreeBSD.conf: JSON text data
% file /usr/local/etc/pkg/repos/FreeBSD-base.conf
/usr/local/etc/pkg/repos/FreeBSD-base.conf: JSON text data
% file /usr/local/etc/pkg/repos/FreeBSD-ports.conf
/usr/local/etc/pkg/repos/FreeBSD-ports.conf: ASCII text
% file /usr/local/etc/pkg/repos/poudriere.conf
/usr/local/etc/pkg/repos/poudriere.conf: JSON text data
%


If I omit the " from /usr/local/etc/pkg/repos/FreeBSD-base.conf for a single repo, there's a detrimental effect on multiple repos:

Code: 
root@mowa219-gjp4-zbook-freebsd:~ # pkg update -r FreeBSD-base && pkg update -r FreeBSD-ports ; date
pkg: Error parsing: '/usr/local/etc/pkg/repos//FreeBSD-base.conf': error while parsing <unknown>: line: 3, column: 57 - 'unexpected newline', character: '0x0a'
No repositories are enabled.
Mon Jun 24 15:49:30 BST 2024
root@mowa219-gjp4-zbook-freebsd:~ #

With the one " back in place:

Code: 
root@mowa219-gjp4-zbook-freebsd:~ # pkg update -r FreeBSD-base && pkg update -r FreeBSD-ports ; date
Updating FreeBSD-base repository catalogue...
FreeBSD-base repository is up to date.
All repositories are up to date.
Updating FreeBSD-ports repository catalogue...
FreeBSD-ports repository is up to date.
All repositories are up to date.
Mon Jun 24 15:49:46 BST 2024
root@mowa219-gjp4-zbook-freebsd:~ #
 
You must log in or register to reply here.
Back
Top