pkg update stalls on hosts with high network usage

Hello everyone,

I am hosting a few Tor relays on FreeBSD VMs and have recently ran into issues with installing updates via pkg on hosts with high network usage.
I have tried forcing pkg to use IPv4 or IPv6, but neither changed anything.

The hosts currently still run FreeBSD 14.3-RELEASE-p15, but will be upgraded to 15.1-RELEASE as soon as it's available.
They all have 2 GB of RAM and 4 GB of Swap configured and the network allows up to 1.000 Mbit/s of traffic.
The firewall used is pf, but the rule pass out quick keep state allows all outgoing traffic.
The host where it happens basically always has about 200 Mbit/s of traffic, with ~30.000 packets per second and about 15.000 tcp states in pf.
DNS is working also, even while under load:
Code:
% host pkg.freebsd.org
pkg.freebsd.org is an alias for pkgmir.geo.freebsd.org.
pkgmir.geo.freebsd.org has address 151.101.193.241
pkgmir.geo.freebsd.org has address 151.101.1.241
pkgmir.geo.freebsd.org has address 151.101.129.241
pkgmir.geo.freebsd.org has address 151.101.65.241
pkgmir.geo.freebsd.org has IPv6 address 2a04:4e42:400::497
pkgmir.geo.freebsd.org has IPv6 address 2a04:4e42:200::497
pkgmir.geo.freebsd.org has IPv6 address 2a04:4e42::497
pkgmir.geo.freebsd.org has IPv6 address 2a04:4e42:600::497
pkgmir.geo.freebsd.org mail is handled by 0 .

The top command reports the following load:
Code:
CPU: 21.9% user,  0.0% nice, 12.5% system,  6.3% interrupt, 59.4% idle
Mem: 681M Active, 812M Inact, 41M Laundry, 384M Wired, 154M Buf, 45M Free
Swap: 4096M Total, 2828K Used, 4093M Free

The vmstat command reports the following:
Code:
 procs    memory    page                      disks  faults       cpu
 r  b  w  avm  fre  flt  re  pi  po   fr   sr vtb0   in   sy   cs us sy id
 0  0  0 2.4G  44M   78  27   0   0   95  340    0 6.9k 268k  23k 17 16 66
 2  0  0 2.4G  44M    1   0   0   0    0  280    0 6.2k 449k  21k 16 19 63
 0  0  0 2.4G  44M    7   0   0   0    0  280    0 4.3k 410k  16k 19 16 64
 1  0  0 2.4G  44M    6   0   0   0    8  280    7 5.7k 381k  19k 20 17 61
 0  0  0 2.4G  44M    1   0   0   0    0  280    0 5.0k 431k  18k 19 21 58
 1  0  0 2.4G  44M    0   0   0   0    0  280    2 5.5k 441k  18k 19 25 54
 1  0  0 2.4G  44M    7   0   0   0    0  280    0 6.7k 425k  22k 19 25 55
 0  0  0 2.4G  44M   14   0   0   0    0  280    0 6.7k 390k  23k 18 16 65
 0  0  0 2.4G  44M    8   0   0   0    0  308    0 6.7k 409k  23k 17 14 67
 1  0  0 2.4G  44M    5   0   0   0    0  280    0 6.6k 411k  22k 20 19 59
 1  0  0 2.4G  44M   12   0   0   0    0  280    0 6.8k 426k  23k 22 22 55

When running pkg update, I get the following output:
Code:
Updating FreeBSD repository catalogue...
Fetching data:   0%    16 KiB  16.0 kB/s 05:59:27 ETA
pkg: An error occurred while fetching package
pkg: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/data.tzst: Not Found
Fetching packagesite:   0%    64 KiB  65.1 kB/s 01:27:35 ETA
pkg: An error occurred while fetching package
pkg: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/packagesite.tzst: Not Found
Unable to update repository FreeBSD
Updating FreeBSD-kmods repository catalogue...
FreeBSD-kmods repository is up to date.
Error updating repositories!
Checking for upgrades (1 candidates): 100%       1 B   0.0 kB/s    00:01
Processing candidates (1 candidates): 100%       1 B   0.0 kB/s    00:01
Checking integrity... done (0 conflicting)
Your packages are up to date.

Weirdly enough, connections over SSH and SFTP are not affected.
When uploading or downloading files, I get round 25 MB/s or 200 Mbit/s, so it's not a bandwidth problem.

As soon as I stop the Tor process, pkg will work normally again.
This however is not a viable solution, as pkg is supposed to be automatically updating itself via Cron.

Is there something I could check to find out why pkg is stalling?
 
I suspect it tries to fetch the packages through tor. Probably not a good idea.
Neither the host nor the tor jail pass any traffic through the Tor network, they use the regular internet.

This is going to bite you at some point too.
Well it has been working fine for the last 4 years, the only pkg on the jail is tor and bastille on the host. If something would break, I have monitoring set up that will alert me.
Also it's paramount that the latest patches are installed, so I'd rather choose a failed relay, ratger that a insecure one. :)

check if your computer has correct date/time then run
The date is correct, NTP is working. :)
The command pkg -d update shows the following:
Code:
root@tor-relay:~ # pkg -d update
Updating FreeBSD repository catalogue...
DBG(1)[10284]> PkgRepo: verifying update for FreeBSD
DBG(1)[10284]> Pkgrepo, begin update of '/var/db/pkg/repos/FreeBSD/db'
DBG(1)[10284]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/meta.conf
DBG(1)[10284]> (fetch) libfetch> connecting
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/meta.conf with opts "i"
DBG(1)[10284]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/data.pkg
DBG(1)[10284]> (fetch) libfetch> connecting
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/data.pkg with opts "i"
DBG(1)[10284]> (fetch) Fetch: fetcher used: pkg+https
[tor-relay] Fetching data:   0%
DBG(1)[10284]> (fetch) Read status: 8192 over 11468230
DBG(1)[10284]> (fetch) Read status: 16384 over 11468230
[tor-relay] Fetching data:   0%    16 KiB  16.4 kB/s    11:38 ETA
DBG(1)[10284]> (fetch) Read status: 24576 over 11468230
DBG(1)[10284]> (fetch) Read status: 32768 over 11468230
DBG(1)[10284]> (fetch) Read status: 40960 over 11468230
DBG(1)[10284]> (fetch) Read status: 49152 over 11468230
DBG(1)[10284]> (fetch) Read status: 57344 over 11468230
DBG(1)[10284]> (fetch) Read status: 65536 over 11468230
DBG(1)[10284]> (fetch) Read status: 73728 over 11468230
DBG(1)[10284]> (fetch) Read status: 81920 over 11468230
DBG(1)[10284]> (fetch) Read status: 90112 over 11468230
DBG(1)[10284]> (fetch) Read status: 98304 over 11468230
DBG(1)[10284]> (fetch) Read status: 106496 over 11468230
DBG(1)[10284]> (fetch) Read status: 114688 over 11468230
DBG(1)[10284]> (fetch) Read status: 122880 over 11468230
DBG(1)[10284]> (fetch) Read status: 131072 over 11468230
DBG(1)[10284]> (fetch) Read status: 139264 over 11468230
DBG(1)[10284]> (fetch) Read status: 147456 over 11468230
DBG(1)[10284]> (fetch) Read status: 155648 over 11468230
DBG(1)[10284]> (fetch) Read status: 163840 over 11468230
DBG(1)[10284]> (fetch) Read status: 172032 over 11468230
DBG(1)[10284]> (fetch) Read status: 180224 over 11468230
DBG(1)[10284]> (fetch) Read status: 188416 over 11468230
DBG(1)[10284]> (fetch) Read status: 196608 over 11468230
DBG(1)[10284]> (fetch) Read status: 204800 over 11468230
DBG(1)[10284]> (fetch) Read status: 212992 over 11468230
DBG(1)[10284]> (fetch) Read status: 221184 over 11468230
DBG(1)[10284]> (fetch) Read status: 229376 over 11468230
DBG(1)[10284]> (fetch) Read status: 237568 over 11468230
DBG(1)[10284]> (fetch) Read status: 245760 over 11468230
DBG(1)[10284]> (fetch) Read status: 253952 over 11468230
DBG(1)[10284]> (fetch) Read status: 262144 over 11468230
DBG(1)[10284]> (fetch) Read status: 270336 over 11468230
DBG(1)[10284]> (fetch) Read status: 278528 over 11468230
[tor-relay] Fetching data:   2%   272 KiB 262.1 kB/s    01:08 ETA
DBG(1)[10284]> (fetch) Read status: 286720 over 11468230
DBG(1)[10284]> (fetch) Read status: 294912 over 11468230
DBG(1)[10284]> (fetch) Read status: 303104 over 11468230
DBG(1)[10284]> (fetch) Read status: 311296 over 11468230
DBG(1)[10284]> (fetch) Read status: 319488 over 11468230
DBG(1)[10284]> (fetch) Read status: 327680 over 11468230
DBG(1)[10284]> (fetch) Read status: 335872 over 11468230
DBG(1)[10284]> (fetch) Read status: 344064 over 11468230
DBG(1)[10284]> (fetch) Read status: 352256 over 11468230
DBG(1)[10284]> (fetch) Read status: 360448 over 11468230
DBG(1)[10284]> (fetch) Read status: 368640 over 11468230
DBG(1)[10284]> (fetch) Read status: 376832 over 11468230
DBG(1)[10284]> (fetch) Read status: 385024 over 11468230
DBG(1)[10284]> (fetch) Read status: 393216 over 11468230
DBG(1)[10284]> (fetch) Read status: 401408 over 11468230
DBG(1)[10284]> (fetch) Read status: 409600 over 11468230
DBG(1)[10284]> (fetch) Read status: 417792 over 11468230
DBG(1)[10284]> (fetch) Read status: 425556 over 11468230
[tor-relay] Fetching data:   3%   416 KiB 147.0 kB/s    01:14 ETA
pkg: An error occurred while fetching package
DBG(1)[10284]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/data.tzst
DBG(1)[10284]> (fetch) libfetch> connecting
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/data.tzst with opts "i"
pkg: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/data.tzst: Not FoundDBG(1)[10284]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/packagesite.pkg
DBG(1)[10284]> (fetch) libfetch> connecting
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/packagesite.pkg with opts "i"
DBG(1)[10284]> (fetch) Fetch: fetcher used: pkg+https
[tor-relay] Fetching packagesite:   0%
DBG(1)[10284]> (fetch) Read status: 8192 over 11468590
DBG(1)[10284]> (fetch) Read status: 16384 over 11468590
DBG(1)[10284]> (fetch) Read status: 24576 over 11468590
DBG(1)[10284]> (fetch) Read status: 32768 over 11468590
DBG(1)[10284]> (fetch) Read status: 40960 over 11468590
DBG(1)[10284]> (fetch) Read status: 49152 over 11468590
[tor-relay] Fetching packagesite:   0%    48 KiB  49.2 kB/s    03:52 ETA
DBG(1)[10284]> (fetch) Read status: 57344 over 11468590
DBG(1)[10284]> (fetch) Read status: 65536 over 11468590
DBG(1)[10284]> (fetch) Read status: 73728 over 11468590
DBG(1)[10284]> (fetch) Read status: 81920 over 11468590
DBG(1)[10284]> (fetch) Read status: 90112 over 11468590
DBG(1)[10284]> (fetch) Read status: 98304 over 11468590
DBG(1)[10284]> (fetch) Read status: 106496 over 11468590
DBG(1)[10284]> (fetch) Read status: 114688 over 11468590
DBG(1)[10284]> (fetch) Read status: 122880 over 11468590
DBG(1)[10284]> (fetch) Read status: 131072 over 11468590
DBG(1)[10284]> (fetch) Read status: 139264 over 11468590
DBG(1)[10284]> (fetch) Read status: 147456 over 11468590
DBG(1)[10284]> (fetch) Read status: 155648 over 11468590
DBG(1)[10284]> (fetch) Read status: 163412 over 11468590
[tor-relay] Fetching packagesite:   1%   160 KiB 114.3 kB/s    04:13 ETA
pkg: An error occurred while fetching package
DBG(1)[10284]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/packagesite.tzst
DBG(1)[10284]> (fetch) libfetch> connecting
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/packagesite.tzst with opts "i"
pkg: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/latest/packagesite.tzst: Not Found
Unable to update repository FreeBSD
Updating FreeBSD-kmods repository catalogue...
DBG(1)[10284]> PkgRepo: verifying update for FreeBSD-kmods
DBG(1)[10284]> Pkgrepo, begin update of '/var/db/pkg/repos/FreeBSD-kmods/db'
DBG(1)[10284]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/kmods_latest_3/meta.conf
DBG(1)[10284]> (fetch) libfetch> connecting
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/kmods_latest_3/meta.conf with opts "i"
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/kmods_latest_3/meta.conf with opts "i"
DBG(1)[10284]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/kmods_latest_3/data.pkg
DBG(1)[10284]> (fetch) libfetch> connecting
DBG(1)[10284]> (fetch) libfetch> fetching from: https://pkgmir.geo.freebsd.org/FreeBSD:14:amd64/kmods_latest_3/data.pkg with opts "i"
FreeBSD-kmods repository is up to date.
Error updating repositories!

Also post your pkg repo config. /usr/local/etc/pkg/repos/
Here you go:
Code:
root@tor-relay:~ # pkg repos
FreeBSD: {
    url             : "pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
FreeBSD-kmods: {
    url             : "pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/kmods_latest_3",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
 
Back
Top