Solved inactive memory not reallocated, server goes swapping

S

sixday

I've got a server with 16G RAM, running 10.3 and a few internal systems (gitlab, jenkins, mysql, etc.)
For some unknown reasons after a while all the memory will be Inactive (at the moment I have 12G of the 16G as Inactive), and then it start swapping, and when there is no more swap processes starts dying.
I think the Inactive memory should be re-allocated but it's not happening!
Is there any way to cleanup/force reallocate it?

I've tested to stop all running services/processes except sshd and there is still 6G of memory is inactive (the rest is reallocated to free when I stopped the processes).
The only way to get rid of it is to reboot the server, but of course this isn't a proper solution.

I know this 6G of memory probably dirty due to some processes memory leak, but there must be a way to clean it up.

Anyone any idea?
Thanks.
 
Hei.
Have you looked through all relevant log files?
Maybe you could find a clue of some application having problems/errors with freeing memory.
A clean start with all the services you suspect disabled in /etc/rc.conf,run top and then start services one by one from the console in forgound-mode if possible and with more verbosity or debug output whatever the services offer you.

Chances are high mysql is the problem so I would start investigating there.

Is the server on ZFS?
You should at least have 1/2 the memory as swap.
 
Thanks ASX, this looks like a working solution, I've deployed it 2 weeks before and no swap usage since.
 
Hi k.jacker

I'm running multiple mysql servers and none of it behaving like this. Personally I suspect java (jenkins/gitlab).
But this all doesn't matter because as I wrote in the opening post my real problem is that after stopping everything on the server (except sshd) there is still 6GB of dirty inactive ram there, which in my opinion should be already gone (the only way to clean it up is to reboot the server which of course isn't a real solution).

Looks like ASX's solution is working so I think this topic is a success.
 
sixday said:
Thanks ASX, this looks like a working solution, I've deployed it 2 weeks before and no swap usage since.
Click to expand...

I'm glad it was of some help, yet ... I wonder if it "help" to hide some unknown issue ... because I agree with your initial assessment:
sixday said:
Inactive), and then it start swapping, and when there is no more swap processes starts dying.
I think the Inactive memory should be re-allocated but it's not happening!
Click to expand...
 
This happens too on my FreeBSD machine. It has 4GB memory and about 75% is "inactive", and this never is released before using swapspace. I also have a NetBSD laptop with 2GB memory and there the inactive memory is given back before using any swap. This laptop rarely uses swap. So I'm going to try this sysctl.
 
ASX said:
Try:
Code: 
sysctl vm.defer_swapspace_pageouts=1
Click to expand...

Is there a risk, perhaps in more loaded machines that performance will suffer because now the machine is trying to flush out inactive (cached) memory instead of trying to swap out some lesser used stuff? I realize swapping is slower, but can't help wonder if this simply defers performance to when its needed most? Maybe this would be just an issue for loaded down machines to which this configuration would not be applied.

Is there a one time command someone can execute, maybe from the root account, that instructs the machine to purge Inactive memory?
 
PacketMan said:
Is there a risk, perhaps in more loaded machines that performance will suffer because now the machine is trying to flush out inactive (cached) memory instead of trying to swap out some lesser used stuff? I realize swapping is slower, but can't help wonder if this simply defers performance to when its needed most? Maybe this would be just an issue for loaded down machines to which this configuration would not be applied.

Is there a one time command someone can execute, maybe from the root account, that instructs the machine to purge Inactive memory?
Click to expand...

Does anyone have any idea about this?
 
CyberCr33p said:
Does anyone have any idea about this?
Click to expand...
This comment on Reddit goes into some depth about the different vm tunables.
https://www.reddit.com/r/freebsd/comments/2k6hq0/swap_virtual_machines/clj2ic1/

It looks as though the vm daemon is proactively paging extremely-idle processes to disk, so that in the event of a future allocation of lots of RAM, the daemon doesn't have to pause the new process in order to page out stuff and make room. I don't think it is being forced out by Inactive Memory being held. These appear to be two separate things, in my mind.
 
You must log in or register to reply here.
Back
Top