Hello!
Machine started logging kernel: swap_pager_getswapspace(12): failed around 24 hours ago, which is a surprise, considering it has 64GB physical RAM, and it's used for one main thing (MySQL) which only consumes about 10% of that RAM. A small amount of the remaining RAM is used for transient processes, but it's mostly consumed by ZFS ARC.
I'm wondering why, with 5861MB free, and 42GB of ARC, swap is at 94% (now) and has at several times apparently been at 100%? Shouldn't ZFS be releasing some of that ARC when free RAM gets low?
I tried swapoff -a && swapon -a to at least (temporarily) get past the high swap, but it complains it cannot allocate memory. Now THAT is really confusing: 5.8GB free RAM, plus many GB of expendable ARC... why is a release of 2GB of swap failing?
The only thing I can think of is that some process is suddenly requesting a large allocation, not giving ZFS enough time to shed some ARC and increase free memory... but I cannot find anything which would do this.
The server has been running in this configuration for several months, and top typically displays swap at around 15% or less. This is the first time the logs show the swap error.
Any ideas?
FreeBSD 12.1-RELEASE
Xeon E5-2630L
64GB ECC RAM
Uptime: 64 days
Thanks.
Code:
CPU: 14.1% user, 0.0% nice, 0.9% system, 0.1% interrupt, 84.8% idle
Mem: 2611M Active, 978M Inact, 1096M Laundry, 52G Wired, 109M Buf, 5861M Free
ARC: 42G Total, 11G MFU, 27G MRU, 1808M Anon, 322M Header, 1137M Other
36G Compressed, 78G Uncompressed, 2.16:1 Ratio
Swap: 2048M Total, 1941M Used, 107M Free, 94% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1736 mysql 77 20 0 5177M 4683M select 3 2462.4 177.25% mysqld
....
Machine started logging kernel: swap_pager_getswapspace(12): failed around 24 hours ago, which is a surprise, considering it has 64GB physical RAM, and it's used for one main thing (MySQL) which only consumes about 10% of that RAM. A small amount of the remaining RAM is used for transient processes, but it's mostly consumed by ZFS ARC.
I'm wondering why, with 5861MB free, and 42GB of ARC, swap is at 94% (now) and has at several times apparently been at 100%? Shouldn't ZFS be releasing some of that ARC when free RAM gets low?
I tried swapoff -a && swapon -a to at least (temporarily) get past the high swap, but it complains it cannot allocate memory. Now THAT is really confusing: 5.8GB free RAM, plus many GB of expendable ARC... why is a release of 2GB of swap failing?
The only thing I can think of is that some process is suddenly requesting a large allocation, not giving ZFS enough time to shed some ARC and increase free memory... but I cannot find anything which would do this.
The server has been running in this configuration for several months, and top typically displays swap at around 15% or less. This is the first time the logs show the swap error.
Any ideas?
FreeBSD 12.1-RELEASE
Xeon E5-2630L
64GB ECC RAM
Uptime: 64 days
Thanks.