Solved pkg update running out of swap

T

trev

Running /usr/sbin/pkg update -f overnight caused this:

Code: 
Jul 24 03:21:00 centurion kernel: swap_pager: out of swap space
Jul 24 03:21:00 centurion kernel: swp_pager_getswapspace(10): failed
Jul 24 03:21:04 centurion kernel: pid 78070 (pkg), jid 0, uid 0, was killed: failed to reclaim memory
Jul 24 03:21:08 centurion kernel: pid 26453 (milter-relay), jid 0, uid 26, was killed: failed to reclaim memory
Jul 24 03:21:08 centurion kernel: swap_pager: out of swap space
Jul 24 03:21:08 centurion kernel: swp_pager_getswapspace(18): failed

The system has 1G of memory and a 1G swap partition. This is the typical memory usage:

Code: 
last pid: 87219;  load averages:  0.55,  0.58,  0.52             up 20+22:43:14  08:57:31
39 processes:  1 running, 38 sleeping
CPU:  0.8% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.0% idle
Mem: 25M Active, 296M Inact, 164K Laundry, 283M Wired, 90M Buf, 858M Free
Swap: 1024M Total, 26M Used, 997M Free, 2% Inuse

This has never happened before - the same command has been running via cron ever since pkg appeared.

Any ideas?
 
It has been updated.
Code: 
$ w3m -dump http://pkg.freebsd.org/FreeBSD:15:amd64/latest|grep pkg
data.pkg          12683817       2026-Jul-22 03:22
files.pkg         43439619       2026-Jul-21 17:11
packagesite.pkg   12684831       2026-Jul-22 03:22
 
Why do you run pkg update -f each day?

The force flag downloads the repos catalogues even if they are the same than the local ones.
When you want to install or upgrade something, pkg update is automatically issued.
Hence my question.
 
What happens if you run it again? Or did it complete anyway? it looks strrange. Why does updating binary files need swap space? Did it happen on a large one like lang/rust?
If the swap error is still going on, I would create a swapdrive of a large file as md device to see if it starts using it and really needs virtual memory. This is inefficient and a bit risky but informative.
 
Turns out cron runs /usr/sbin/pkg update -f monthly not daily. The system actually has 1.5G memory (not 1G) and 1G swap... and it still fails if I run the command manually (see below). Another system with 512MB of memory and 1G of swap has no issues running the command.

Code: 
root@centurion:~ # /usr/sbin/pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%     179 B   0.2 kB/s    00:01    
Fetching data: 100%    11 MiB   1.4 MB/s    00:08    
Processing entries: 100%
FreeBSD repository update completed. 37835 packages processed.
Fetching filesite: 100%    68 MiB   4.4 MB/s    00:16    
Processing file entries:   0%Killed

It sits at the "Processing file entries: 0%" for a while and then "Killed" appears and the log shows:

Code: 
Jul 26 09:31:53 centurion kernel: swap_pager: out of swap space
Jul 26 09:31:53 centurion kernel: swp_pager_getswapspace(8): failed
Jul 26 09:31:58 centurion kernel: pid 11755 (pkg), jid 0, uid 0, was killed: failed to reclaim memory
Jul 26 09:31:59 centurion kernel: pid 26416 (milter-greylist), jid 0, uid 26, was killed: failed to reclaim memory
Jul 26 09:31:59 centurion kernel: swap_pager: out of swap space
Jul 26 09:31:59 centurion kernel: swp_pager_getswapspace(24): failed
 
MG said:
What happens if you run it again? Or did it complete anyway? it looks strrange. Why does updating binary files need swap space? Did it happen on a large one like lang/rust?
If the swap error is still going on, I would create a swapdrive of a large file as md device to see if it starts using it and really needs virtual memory. This is inefficient and a bit risky but informative.
Click to expand...
Something odd is going on. I have three systems with the same crontab updating on different days once a month and only this one system fails. Running it multiple times from the command line always exhibits the same "out of swap" issue (and it really is according to top).

I don't think creating more swap is the answer as the system with 1/3rd the memory does not have the issue and completes the command very quickly as opposed to taking much longer (using up all the memory and swap space) before dying.
 
Emrion said:
Why do you run pkg update -f each day?
Click to expand...
Monthly actually (my mistake reading the crontab). As to why I use the -f flag, it is lost in the mists of time. It does however work on the other two systems and used to work on this system until a few days ago.

Emrion said:
The force flag downloads the repos catalogues even if they are the same than the local ones.
Click to expand...
Curiously, the man page for pkg does not list the -f flag in this context, only for the bootstrap command.

Edit: Ahhh, it is listed in the pkg-update man page.
 
Running pkg --debug update -f doesn't help identify the issue:

Code: 
Fetching filesite: 100%    68 MiB   5.1 MB/s    00:14    
DBG(1)[65710]> PkgRepo: extracting filesite.yaml of repo FreeBSD
DBG(1)[65713]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[65710]> Fingerprint 'b0170035af3acc5f3f3ae1859dc717101b4e6c1d0a794ad554928ca0cbb2f438' is trusted
DBG(1)[65710]> Signature verified with trusted fingerprint
Processing file entries:   0%Killed
 
trev said:
I have three systems with the same crontab updating on different days once a month and only this one system fails.
Click to expand...
Have you verified that the configuration is the same across all three systems?
pkg config pointing at the same repos (all quarterly or all latest)?
same version of pkg on all three systems?

It would also be useful if you tell us the freebsd-version -kru output on each of the three systems.
 
Is this against ports quarterly or latest?
Emrion I'm looking at quarterly, pkg --version says 2.7.5 and my db is about 70M.
quarterly seems to not have pkg 2.8.1, so I'm wondering if the problem is related more to latest repo than newer version of pkg.
 
I'm on latest, but pkg 2.7.5 was working better with the same repos. It's just people that have quarterly repos can't upgrade to 2.8.1, so no problem for them.
Note that I have pkg 2.8.0 on another machine (latest repos too) and no trouble there.

For me, it's clearly a bug (or a new design choice) of pkg.
 
Emrion said:
I'm on latest, but pkg 2.7.5 was working better with the same repos. It's just people that have quarterly repos can't upgrade to 2.8.1, so no problem for them. Note that I have pkg 2.8.0 on another machine (latest repos too) and no problem there.
Click to expand...
Thanks. So bisecting sounds like 2.8.0 to 2.8.1 introduced something, I would start with "how does /var/db/pkg/repos/FreeBSD-ports/db get created? Is that something pkg is creating locally based on things downloaded? Could it be related to the recent ports freeze?
I'm just speculating here, trying to see if there is missing data from the OP. Which is why I asked about comparing the 3 systems. OP states 2 work one doesn't so based on what you've said, I immediately go to "is the failing system looking at latest the 2 working looking at quarterly"?

Anyway, thanks for the data point.
 
I cannot confirm that 2.8.0 is OK. On my systems it also produces a 1.5GB db file.
The pkg repo looks strange to me, so no idea if git bisect is useful.

472c575e1ff9296761fc6d9e22ddcb0a3bb89a2f is bad
563486eed8f7fc7da5af1a91e6cc156311a40f42 does not compile
563486eed8f7fc7da5af1a91e6cc156311a40f42 + cherry-pick 9dc5efd65a4b6de7af23d9938d8e0554314a0579 (build fix) is OK
 
mer said:
Have you verified that the configuration is the same across all three systems?
pkg config pointing at the same repos (all quarterly or all latest)?
same version of pkg on all three systems?

It would also be useful if you tell us the freebsd-version -kru output on each of the three systems.
Click to expand...

Same config file for all - pkg.conf is identical to the pkg.conf.sample file

shadow and centurion point to quarterly point to quarterly; sentinel points to latest repo.

pkg 2.8.1 on all systems

shadow (32G/10G swap): 14.4 STABLE
centurion (1.5G/1G swap): 14.4-RELEASE-p7. (runs out of swap)
sentinel (512M/1G swap): 14.4-RELEASE-p7
 
You must log in or register to reply here.
Back
Top