Server keeps running out of RAM and uses swap even if inactive memory is very high

B

Bergamin

Hello guys,
I'm running a FreeBSD 12 server and I'm experiencing this strange issue. I've looked around in this forum for people with the same problem, but these topics where a bit too old and I think that the proposed solution might not apply to FreeBSD 12. Many where suggesting using
sysctl vm.defer_swapspace_pageouts=1
but on FreeBSD 12 is not possible (?).

My server has a total of 4GB of ram and 2GB of swap. My processes use about 3.6GB of ram. Right now, after a fresh restart, my server has about 900MB of free ram.
The problem is that after a while (let's say 1-2 days), inactive memory gets really high (up to 2GB) and swap usage goes really high too.
Why won't the server recycle the inactive memory?

I'm running a mysql+minecraft server on my machine. My guess (but I'm a newbie) is a mal configured mysql server. I left it with default settings, maybe I should try running mysqltuner.
A part from that, it seems to me very strange.

Has anyone run in a similar situation? Is there anything that I can to do force my server to recycle inactive memory?

Thanks in advance!
 
I've run into a similar situation where compiling a huge application would push my desktop machine into swap and left parts of the memory swapped out even long after the compilation was finished and there being multiple GB of available RAM again resulting in quite a bit of lag (i have a pretty slow harddisk) when applications tried to access it and it had to be paged in again. I've not investigated this much but the people i spoke to told me that there wasn't much (in the sense of kernel settings) i could do about it. I haven't checked the accuracy of this information yet but if there really isn't i might try to patch in some knobs myself. That would happen somewhere in the future though as right now i sadly don't have the time for it.
 
For the Minecraft server, you might want to limit the RAM it is allowed to use with Java parms -Xms (initial RAM pool) and -Xmx (maximum RAM).. example: -Xms256M -Xmx2G, this will help in low memory environments, at least this was the fix a few years ago when I ran one. You also probably do need to tune your MySQL server to do the same thing, by limiting the RAM utilization for cached queries, etc.
 
Bergamin said:
Why won't the server recycle the inactive memory?
Click to expand...

Quite simply, why should it?
Inactive memory means it's... inactive. It relates to sleeping processes. It's highly likely that it's tagged to be swapped out should you require the memory.
If you want to remove inactive processes, start killing off stuff you're not using.

If your swap reaches its limit, then increase swap. See the handbook about how to do this (as a file): 11.12.2. Creating a Swap File

It might be a good idea to see if your mysql settings are overly generous.

I know nothing about your set up, but lots of things impact the memory consumption, from bloatware like web browsers to java, to certain desktops (looking at you KDE) and so on.

So, in summary, I'm saying it's too vague a question you're asking. However, for example, a headless server, 4GB RAM and 1/2 of that for swap is more than adequate. So, what's your setup and why is it exhausting virtual memory?
 
ekvz said:
I've run into a similar situation where compiling a huge application would push my desktop machine into swap and left parts of the memory swapped out even long after the compilation was finished
Click to expand...
That’s normal behavior. Why should the kernel waste time and I/O bandwidth swapping things in that are not (yet) used? It’s preferable to keep them in swap, in case something else needs the memory.

Older versions of FreeBSD even swapped out unused processes proactively after some time, even if there was sufficient free memory. Typical “victims” were the unused getty(8) process on the virtual syscons terminals; these always ended up in swap after some time. However, FreeBSD 12 doesn’t seem to do that anymore. I guess that feature was removed because today the size of RAM is typically much larger than the size of swap space. 25 years ago it was rather the opposite: swap space was usually larger than RAM.

Today, the usual advice for people who have problems with things being swapped: Buy more RAM. 4 GB is nothing by today’s standards, especially if your run Java + database stuff, and even more so if you put ZFS into the mix.
 
olli@ said:
That’s normal behavior.
Click to expand...

Oh, i didn't want to imply that this was an unreasonable default behavior in any way.

olli@ said:
Why should the kernel waste time and I/O bandwidth swapping things in that are not (yet) used?
Click to expand...

Because i hate having to wait ~30 seconds for my editors window to appear ;)

olli@ said:
Today, the usual advice for people who have problems with things being swapped: Buy more RAM. 4 GB is nothing by today’s standards, especially if your run Java + database stuff, and even more so if you put ZFS into the mix.
Click to expand...

I fully agree on this being the most pragmatic approach. Throwing money at the problem isn't always preferable though in which case having a bunch of settings to influence the swapping behavior would be pretty useful. Like i said before it's not like expect someone else to implement this for me and i'll probably just try to add some switches myself when i find the time. I guess i'll see how this works out when i get to it but i feel like it shouldn't be that hard (i already have some kind of logic in mind that should stop it from becoming to trigger happy) and a little more flexibility wouldn't hurt, would it?
 
sysctl vm.defer_swapspace_pageouts=1:I do not have this sysctl knob, do you mean sysctl -d vm.disable_swapspace_pageouts?
olli@ said:
Today, the usual advice for people who have problems with things being swapped: Buy more RAM. 4 GB is nothing by today’s standards, especially if your run Java + database stuff, and even more so if you put ZFS into the mix.
Click to expand...
4 GB RAM should be fairly enough for a small DB; but 4GB RAM vs. 2 GB swap is a suboptimal choice.
Bergamin, please have look into this thread. It boiled down to be an NFS issue.
 
I use FreeBSD 12.1 :

Code: 
sysctl -d vm.pageout_update_period
vm.pageout_update_period: Maximum active LRU update period

sysctl vm.pageout_update_period
vm.pageout_update_period: 600

Edit: I thought you were replying to me.
 
I'm working through something similar with MySQL 5.6 and mysqldump, but on a server with 32Gb of RAM.

A mysqldump script runs at 7.15 a.m., no swap is used.

Exactly the same script runs at 7.15 p.m., and mysqld uses swap and doesn't release it until the process is restarted.

So I think something happening during the 7.15 p.m. run (MySQL busier?) that makes it/FreeBSD/jemalloc want to swap a bit. I might have given InnoDB too much of the RAM. Time to try mysqltuner myself!

More RAM should help your case but MySQL may still need some tuning (as in my case.)
 
You must log in or register to reply here.
Back
Top