swap_pager_getswapspace(24): failed

[abishai]

Hello,
I have a strange issue with server, running bhyve VMs.

Mem: 8827M Active, 8035M Inact, 7908M Laundry, 34G Wired, 4110M Free
ARC: 21G Total, 1117M MFU, 18G MRU, 17M Anon, 268M Header, 1642M Other
     17G Compressed, 18G Uncompressed, 1.05:1 Ratio
Swap: 32G Total, 28G Used, 4380M Free, 86% Inuse

How can I prevent this issue?
I have vfs.zfs.arc_max="30G" and 64GB of memory.
5 VMs are running at the moment, 18 GB total and 17 jails (no idea how to measure jails). I think, such swapiness is unusual : we can see that there are some free memory, OS could decrease swap usage. I see hundreds getswapspace messages per day, but nothing is killed.

 

[SirDice]

we can see that there are some free memory, OS could decrease swap usage.

You have some free memory because some things are swapped out. FreeBSD likes to preemptively swap unused things out to disk to allow running processes to have more free memory immediately accessible (so the process doesn't have to wait for the system to swap stuff out in order to satisfy the request).

I see hundreds getswapspace messages per day, but nothing is killed.

I would start by lowering arc_max even further. Then try to find a "sweet spot", somewhere between a bit (<1%) and half (50%) of swap in use.

What version of FreeBSD are you using? Because I have noticed a difference in swap usage (or I should say, its eagerness to swap stuff out) between the various versions.

 

[abishai]

FreeBSD likes to preemptively swap unused things out to disk to allow running processes to have more free memory immediately accessible

So, getswapspace failures are harmless? I have this condition for 3 days and nothing gets killed. Was it 'preemptively swapping' that failed? uptime is 27 days though.

What version of FreeBSD are you using?

12.1-RELEASE.

 

[SirDice]

So, getswapspace failures are harmless?

Not exactly. It does mean it tried to swap more out but failed. So it isn't optimal, not a situation you want to keep. You could simply try to add more swap but I think 32GB for a 64GB machine should be enough.

I have one machine with 96GB RAM and 16GB swap. There are a number of VMs running on it, for a total of ~80GB. ARC is capped at 40GB and I have 0 swap in use at the moment. This machine runs a recent 12-STABLE and swap usage has been fluctuating a lot with previous revisions. It seems to have settled quite nicely for me at the moment.

 

[abishai]

I capped ARC slightly, swap looks stable at ~10GB for 2 weeks.