Solved FreeBSD 12.1 - processes killed with "out of swap space" message

This is a FreeBSD 12.1 and it seems that the problem started after we upgraded from 11.3.

The server has 32GB of ram and it's a samba file server and also running several unison file sync processes.

The problem is that the server starts killing processes even thought it has 32G of swap and then added 32G more out of zroot but swap is mostly unused.


Device 1M-blocks Used Avail Capacity
/dev/zvol/zroot/swap0 32768 0 32768 0%
/dev/ada0p3 16384 0 16384 0%
/dev/ada1p3 16384 0 16384 0%
Total 65536 0 65536 0%


The screenshot of one of the daily crashes:
2020-12-07_08-08.png


I found this post that says that FreeBSD won't swap out active processes and someone recommends I can increase sysctl vm.pageout_oom_seq

another relevant post I read:

my ZFS allocations in /boot/loader.conf
vfs.zfs.arc_min="4G"
vfs.zfs.arc_max="8G"


Note that I've had to increase kern.maxvnodes to 6000000 after watching vfs.freevnodes and going up incrementally

2020-12-08_10-54.png


Any advice?
 
I would suggest finding out what keeps eating away your memory. I doubt it's caused by Samba, that looks like it's an innocent bystander.
 
I ended up turning off the daily ZFS snapshots in periodic.conf and the server stopped going out of memory and killing processes.

It's a 32GB machine and I have zfs arc set to:
>vfs.zfs.arc_min="4G"
>vfs.zfs.arc_max="8G"
 
I get this when using lsof... which consumes memory forever when enumerating all open files. Some recent change in the kernel (from around June?) is the cause for me and I noticed it through rkhunter which uses lsof.

Long story short, I just don't use rkhunter/lsof anymore... for now (problem is still there with lsof... consumes memory forever when running 'lsof').
 
How were things running on 12.1 before tweaking kern.maxvnodes? what about before the other tweaks?
before tweaking kern.maxvnodes things where ok, just slower when hitting the maxvnodes limit.

The reason of the system going out of swap space was that I enabled ZFS snapshots for all pools and as non-swappabled processes they would take up all the machine memory.
 
Back
Top