Solved updating pkg 1.19 -> 1.20 breaks pkg in loopback jails (partially solved: problem is curl and ssl, not loopback jals)

[sko]

I just came around updating a host that I have to admit I'm neglecting a bit...

The host runs on 12.4-RELEASE-p6 (was on p5 when I started debugging this), the jails are connected to a dedicated loopback interface (lo1) and PF handles all forwards/nat etc. (This host was first set up before vnet was "production-safe")
This has worked for several years without any issues.

Now when I update pkg in one of the jails it will no longer update from my local repository:

root@postfix:~ # pkg -v
1.19.2
root@postfix:~ # pkg update -f
Updating local repository catalogue...
[postfix.mail2.mydomain.tld] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
[postfix.mail2.mydomain.tld] Fetching packagesite.pkg: 100%  149 KiB 152.3kB/s    00:01    
Processing entries: 100%
local repository update completed. 584 packages processed.
All repositories are up to date.
root@postfix:~ # pkg upgrade -y pkg
Updating local repository catalogue...
local repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.19.2 -> 1.20.8

Number of packages to be upgraded: 1

The process will require 1 MiB more space.
[postfix.mail2.mydomain.tld] [1/1] Upgrading pkg from 1.19.2 to 1.20.8...
[postfix.mail2.mydomain.tld] [1/1] Extracting pkg-1.20.8: 100%
Updating local repository catalogue...
pkg: An error occured while fetching package
pkg: An error occured while fetching package
repository local has no meta file, using default settings
pkg: An error occured while fetching package
pkg: An error occured while fetching package
Unable to update repository local
Error updating repositories!
root@postfix:~ # pkg update
Updating local repository catalogue...
pkg: An error occured while fetching package
pkg: An error occured while fetching package
repository local has no meta file, using default settings
pkg: An error occured while fetching package
pkg: An error occured while fetching package
Unable to update repository local
Error updating repositories!

The host itself is fine:

root@mail2:~ # pkg -v
1.20.8
root@mail2:~ # pkg update -f
Updating local repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
Fetching packagesite.pkg: 100%  149 KiB 152.3kB/s    00:01    
Processing entries: 100%
local repository update completed. 584 packages processed.
All repositories are up to date.

nginx on the pkg host doesn't log any errors or 404s, but also no access when the jail tries to pkg update. I *can* see traffic on the hosts external interface to and from the pkg host, I can also successfully fetch the meta files from the jail and openssl s_client also establishes a successful connection to the pkg host. So only pkg is showing this behaviour.

The repository is configured with an https url; pkg+https as well as the srv mirror_type have been removed a while ago. I also tried forcing pkg to use IPv4 (which it did anyways, but just to be sure...)
All my other jailhosts have already been migrated to vnet and they don't show this behaviour, so it seems the http(s) handler of pkg has had some changes that break it in "loopback-jail" configurations + https (pkg+https works).


Other things I tried:
- using pkg-1.20.7 from the official repositories (updating from the official repos works with all pkg versions!)
- reducing the mtu on the jails loopback interface (I've once seen some weird behaviour with pkg and jumbo frames, so it was worth a try)
- switching to http


This host is bound to be replaced/rebuilt with either 13.2-RELEASE or even 14.0-RELEASE as soon as the 12 branch is nearing EOL, so this isn't super-critical as I can just lock pkg to the last 1.19 version for now. But if someone has any other ideas how to fix this I'd be grateful for any help.

 

[VladiBG]

There's change in repos config. Check if your repo reflect the new change.

example

Synth: {
url: "http://*********/pkg/",
mirror_type: "none",
signature_type: "none",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}

 

[sko]

There's change in repos config. Check if your repo reflect the new change.

example

pkg+https as well as the srv mirror_type have been removed a while ago.

I knew about this change as this broke my pkg setup on several other hosts, so the repo configs have been updated on all hosts and jails a while ago...


edit:

here's my current repository configuration (on all hosts and jails):

local: {
  url: "https://pkg.mydomain.tld/packages/${ABI}-quarterly-servers",
  signature_type: "pubkey",
  pubkey: "/usr/local/etc/pkg/keys/pkg.mydomain.key",
  priority: 5,
  enabled: yes
}

 

[VladiBG]

Maybe the debug output will help you to identify the issue
pkg -d update

 

[sko]

Maybe the debug output will help you to identify the issue
pkg -d update

THANKS

I don't know why I didn't try that myself...

So the culprit is pkg switching from fetch to CURL, which uses a wrong CA path:

[...]
* Connected to pkg.mydomain.tld (<myip>) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
[...]

The correct path should be /usr/share/certs/ or /usr/local/share/certs (or simply /etc/ssl/cert.pem, which is linked to /usr/local/share/certs/ca-root-nss.crt)
The host *has* that /etc/ssl/certs path with its contents linked to /usr/share/certs/trusted/, but the jails don't.
I don't get why pkg even switched to bloated curl if fetch is BSD-native and much more lightweight...


I dug a bit deeper and found that the second remaining 12.4-RELEASE host simply wasn't switched yet to the new quarterly branch - i.e. it still used pkg version 1.19.2. After upgrading one of its jails to 2023Q4 and pkg 1.20.7, updates also fail for all jails - as the host also has the symlink for /etc/ssl/certs in place.

So in fact it seems to boil down to simply being this problem (and thankfully not some weird loopback edge-case):

curl unable to connect via TLS after change to default trust store?

After the recent change to ftp/curl where security/ca_root_nss was removed as a dependency in favor of using the base system's default trust store, I can't connect via TLS after removing security/ca_root_nss from my system. $ curl -I https://google.com curl: (60) SSL certificate problem: unable...

forums.freebsd.org

 

[VladiBG]

But the default path which should hold all symlinks is /etc/ssl/certs/ and this is the default path which is picked up by <curl/curl.h> at the build time of curl. When you install third party CA those certs should be linked into /etc/ssl/certs/

here's the trustpath during the search of rehash command in certctl(8) which should rebuild and put all certs symlinks under /etc/ssl/certs/

TRUSTPATH         List of paths to search for trusted certificates.
                       Default: <DESTDIR>/usr/share/certs/trusted
                       <DESTDIR>/usr/local/share/certs
                       <DESTDIR>/usr/local/etc/ssl/certs

Maybe the issue here is that the jail doesn't have those symlinks. What will happen if you try to rebuild them with certctl rehash inside the jail?

 

[sko]

What will happen if you try to rebuild them with certctl rehash inside the jail?

That will solve it:

root@nginx:~ # ll /etc/ssl
total 9
lrwxr-xr-x  1 root  wheel     43 Oct  9 08:07 cert.pem@ -> ../../usr/local/share/certs/ca-root-nss.crt
-rw-r--r--  1 root  wheel  11000 Dec  7  2022 openssl.cnf
root@nginx:~ # certctl rehash
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
root@nginx:~ # ll /etc/ssl
total 26
drwxr-xr-x  2 root  wheel     40 Oct 13 17:51 blacklisted/
lrwxr-xr-x  1 root  wheel     43 Oct  9 08:07 cert.pem@ -> ../../usr/local/share/certs/ca-root-nss.crt
drwxr-xr-x  2 root  wheel    130 Oct 13 17:51 certs/
-rw-r--r--  1 root  wheel  11000 Dec  7  2022 openssl.cnf

Thanks! I never really dug deep into the whole topic on ssl certstores because it always 'just worked'™

The weird thing however, the release-dataset from which basejails are created and (re-)cloned contains the certs directory; the jails don't even after forcing an upgrade (i.e. re-cloning and merging of /etc). It *might* be the merging of /etc that does the wrong thing here...
I don't think I will debug this any further - those two hosts are to be replaced soon and certctl rehash is a viable solution to the problem. Thanks again for the help!