64 GB RAM yet applications keeps crashing - what to do?

[Tracker]

Hi,

I have a machine with 64 gb memory. The swap space on it is 8 gb.

Now when I put the machine on sleep/hibernate - the swap fills up. It doesn't clear when the machine is turned back on and htop shows it to be running almost full.

As a result of this the applications I am running keep crashing randomly.

What is the solution to fixing this issue? Is there some way to recycle the swap to memory (to prevent crashing applications)?

PS: there is plenty of RAM but it's not being used somehow

 

[Alain De Vos]

First and faremost you need to know which applications are eating this memory as this is not normal.
Run htop .

 

[Tracker]

First and faremost you need to know which applications are eating this memory as this is not normal.
Run htop .

I still have free memory (that's what htop shows) - most of it is the clogged up swap space (which is full and not recycling back to memory after hibernating/sleep) . To be clear - there is free memory - swap is full - perhaps causing the crash(es).

Do you mean check what's utilizing the swap space? If yes - how?

 

[VladiBG]

How to find out what is using swap on FreeBSD?

I'm trying to pinpoint what process(es) is eating swap on my FreeBSD server. I can only see how much is being used (top) but not what process is swapping its guts out. I tried searching Google for this but didn't find anything. Is there a way to display list of swapped data by process...

forums.freebsd.org

 

[Tracker]

How to find out what is using swap on FreeBSD?

I'm trying to pinpoint what process(es) is eating swap on my FreeBSD server. I can only see how much is being used (top) but not what process is swapping its guts out. I tried searching Google for this but didn't find anything. Is there a way to display list of swapped data by process...

forums.freebsd.org

Here is an image of the processes - seems like swap usage with top -woswap method isn't
reflecting the actual usage?

 

[VladiBG]

The swap partition/file doesn't clear up instantly when it's not used. You can use swapinfo or top to check the usage and swapon(8) / swapoff to turn it on/off which will cleanup it but if there's any current usage of the swap you need to have the same amount of free memory before turning off the swap.

 

[Tracker]

The swap partition/file doesn't clear up instantly when it's not used. You can use swapinfo or top to check the usage and swapon(8) / swapoff to turn it on/off which will cleanup it but if there's any current usage of the swap you need to have the same amount of free memory before turning off the swap.

I tried the swapoff/swapon method but it doesn't seem to work :
sudo swapoff -a
Password:
swapoff: /dev/ada0p4.eli: Cannot allocate memory

PS: I literally have 30 gb + memory free

Here is output of swapinfo for reference :
swapinfo
Device 512-blocks Used Avail Capacity
/dev/ada0p4.eli 16244736 16244720 16 100%


top output is in the image above - but it doesn't seem to sum up to 8gb of swap usage

What can I possibly do?

 

[yuripv79]

44G laundry.

What are you running on that system?

 

[Tracker]

44G laundry.

What are you running on that system?

? I'm just taking revenge on my system for complaining about less memory - but it seems swap issues won't let me have my peace ?

I'm actually not running much - except for a heavy instance of a browser for now ... so wondering how to fix things ?

 

[yuripv79]

No, seriously, what is running there? See e.g. the following posts:

Re: huge amount laundry memory not being cleaned up

Re: huge amount laundry memory not being cleaned up

I also see chrome in the process list, does quitting chrome and/or X reduce that laundry number?

 

[Tracker]

No, seriously, what is running there? See e.g. the following posts:

Re: huge amount laundry memory not being cleaned up

Re: huge amount laundry memory not being cleaned up

I also see chrome in the process list, does quitting chrome and/or X reduce that laundry number?

It's actually just an instance of chrome (that has quite a few tabs) - and another browser with jupyter running in the background. It's not a server - so X is required. Trying not to close Chrome since I am in the middle of some work - is there any other way to diagnose the issue?

Btw this is a 13.2 system as the post you linked to also mentioned - is there some issue with that release? Or something else?

The swap always remains full once I cause it to sleep/S3 lid action - and doesn't empty despite memory available somehow

 

[Tracker]

Is there anything to do to recycle swap? ?‍

 

[Alain De Vos]

No this is automatic when a process frees memory when it terminates.
What' the output of

ps -m -o rss,command

 

[mer]

The swap always remains full once I cause it to sleep/S3 lid action - and doesn't empty despite memory available somehow

This is an important fact.

Is there anything to do to recycle swap?

The only way I know of is stopping the processes that are using swap or the swapoff/swapon.

 

[Tracker]

No this is automatic when a process frees memory when it terminates.

So it's a dead-end when something goes wrong like in my case? :|

What' the output of

ps -m -o rss,command

ps -m -o rss,command
   RSS COMMAND
  5848 vim ch2.hs
 84332 /usr/local/lib/ghc-9.4.5/bin/./ghc-9.4.5 -B/usr/local/lib/ghc-9.4.5/lib --interactive
100960 /usr/local/lib/ghc-9.4.5/bin/./ghc-9.4.5 -B/usr/local/lib/ghc-9.4.5/lib --interactive ch2.hs
  3528 top -w
  2896 bash
  1296 bash
     8 bash
     8 bash
     8 bash
   812 bash
     8 bash
     8 bash
     8 bash
     8 bash
     8 bash
     8 bash
   432 sctd --latitude 25.204849 --longitude 55.270782
  1884 less ch4/2-thirdElement.hs
  3020 ps -m -o rss,command
114912 /usr/local/bin/python3.9 /usr/local/bin/jupyter-lab --no-browser
  4088 python3.9

The only way I know of is stopping the processes that are using swap or the swapoff/swapon.

Swapoff/swapon didn't work for me as stated in my earlier reply above - do you think it's possibly because of GELI encryption that it didn't work?

 

[Alucn]

我也遇到了这个问题。为什么不回收交换空间的使用？交换空间的使用率一直在增加？

 

[gpw928]

The FreeBSD Manual warns that Suspend/Resume has some significant limitations.

As yuripv79 suggested, that 44GB Laundry is quite a worry. My concern is that "suspend" is somehow clearing memory to swap, and there's only 8G of swap...

Are you using S3 (Sleep mode), or S4 (Hibernation)? The manual says that Hibernation is not supported.

The manual also says "But currently the Suspend and Resume functions only work properly on some specific laptops".

 

[Alain De Vos]

Increase swap space to 64GB , non-encrypted , and try at new... ?
To view memory usage meaybe try ,

GitHub - jupyter-server/jupyter-resource-usage: Jupyter Notebook Extension for monitoring your own Resource Usage

Jupyter Notebook Extension for monitoring your own Resource Usage - jupyter-server/jupyter-resource-usage

github.com

 

[Alain De Vos]

我也遇到了这个问题。为什么不回收交换空间的使用？交换空间的使用率一直在增加？

Please use English.

 

[smithi]

The FreeBSD Manual warns that Suspend/Resume has some significant limitations.

Do we even know what make & model Tracker's machine is? It might help ...

Generally if resume after suspend worked, it works - but there may be a lot of variables in a case like this.

That section also covers adding swap, but I noticed it suggests adding swapfiles with ZFS is a bad idea, despite being the easist thing to test with UFS.

As yuripv79 suggested, that 44GB Laundry is quite a worry. My concern is that "suspend" is somehow clearing memory to swap, and there's only 8G of swap...

That isn't something suspend normally does, to my understanding (which is less than perfect).

Adequate stats recorded before and after suspending should indicate any such issues.

8G swap for 64G memory seems way too little. The old advice of 'twice memory' may be overkill, but 1/8 of memory smells like trouble.

Alain's advice of 64G swap seems worth testing; 32G anyway.

Are you using S3 (Sleep mode), or S4 (Hibernation)? The manual says that Hibernation is not supported.

Only on some really ancient laptops with BIOS support for it: 's4bios'. If resuming then it's S3 suspend.

The manual also says "But currently the Suspend and Resume functions only work properly on some specific laptops".

Most Thinkpads anyway, but likely more others more recently. Most up to date seems to be:
https://wiki.freebsd.org/Laptops

With more historical info at:
https://wiki.freebsd.org/SuspendResume

HTH

 

[Tracker]

The FreeBSD Manual warns that Suspend/Resume has some significant limitations.

It seems to work - like the machine goes to sleep and resumes fine. It's just the memory that's an issue.

That isn't something suspend normally does, to my understanding (which is less than perfect).

This swap-not-emptying problem for me has existed since ages - and to touch upon your model of laptop direction - it's been the same issue with two different models of laptops. Maybe I'm just unlucky or maybe this is a genuine issue - I suspect the latter.

Are you using S3 (Sleep mode), or S4 (Hibernation)? The manual says that Hibernation is not supported.

Only on some really ancient laptops with BIOS support for it: 's4bios'. If resuming then it's S3 suspend.

This is what I use to sleep : sudo acpiconf -s 3
Also - this is what is supported on the machine :

sysctl hw.acpi.supported_sleep_state
hw.acpi.supported_sleep_state: S3 S4 S5

As yuripv79 suggested, that 44GB Laundry is quite a worry. My concern is that "suspend" is somehow clearing memory to swap, and there's only 8G of swap...

I suspect the same - suspend (via `acpiconf -s 3` is somehow dumping it to swap and that doesn't clear upon resumption.

Adequate stats recorded before and after suspending should indicate any such issues.

If there are any specific commands I could try easily, please let me know.

8G swap for 64G memory seems way too little. The old advice of 'twice memory' may be overkill, but 1/8 of memory smells like trouble.

Alain's advice of 64G swap seems worth testing; 32G anyway.

Seems like an overkill for a 512 G machine to allocate 32/64G to swap - also I'm barely using ~20 G RAM at most peak times. It's the **NOT** _clearing_ of swap that, to my untrained mind, is probably the root cause.

To view memory usage meaybe try ,

GitHub - jupyter-server/jupyter-resource-usage: Jupyter Notebook Extension for monitoring your own Resource Usage

Jupyter Notebook Extension for monitoring your own Resource Usage - jupyter-server/jupyter-resource-usage

github.com

Thanks - I installed this as well to rule out excessive memory usage by jupyter - seems like it's using 1.5'ish GB max.

Current htop snapshot shows 13.7 GB used/ 64 GB ..... BUT swap is fully utilized.

Is this a FreeBSD suspend/resume issue? ?

 

[Alain De Vos]

My PC has 32GB memory & 64GB non-encrypted swap partition.
It's not an overkill , as 64GB disk space is cheap.

 

[mer]

Is this a FreeBSD suspend/resume issue?

Perhaps.
How often do you suspend?
What happens to swap use if you do a complete power off, restart, then use the system as you normally do?
Does the swap use go up?
I don't know if suspend writes anything to swap, but since "suspend may save some state" I can imagine it may.
Toss in the encrypted swap: I think that is done with a one time key; is that key actually preserved across suspend? If not, that may exacerbate the problem.
A way to test that theory would be to modify /etc/fstab so swap is not encrypted. Yes it sounds like to get to that point you need to reboot, but it would at least give a datapoint.

Tracker
Have you tried adding another swap device, bigger than the current one?
I think if you add another swap device (even if you use a temporary external USB one), do a swapon to the new device, then do a swapoff on the original one, the kernel should move whatever needs to be in swap over to the new device. It may take a bit, but if that worked, the system may recognize that the stuff it just moved to the new swap device isn't really needed and will let you swapoff the new one and then swapon the original one.

 

[Alain De Vos]

The following is a memory script i've written. [pkg install zsh gawk]
You could have a look to the output,

#!/usr/local/bin/zsh
W=`sysctl vm.stats.vm.v_wire_count    | gawk '{$2=$2*4096/1024/1024;printf("%04.0f\n",$2);}'`
A=`sysctl vm.stats.vm.v_active_count  | gawk '{$2=$2*4096/1024/1024;printf("%04.0f\n",$2);}'`
L=`sysctl vm.stats.vm.v_laundry_count | gawk '{$2=$2*4096/1024/1024;printf("%04.0f\n",$2);}'`
I=`sysctl vm.stats.vm.v_inactive_count| gawk '{$2=$2*4096/1024/1024;printf("%04.0f\n",$2);}'`
C=`sysctl vm.stats.vm.v_cache_count   | gawk '{$2=$2*4096/1024/1024;printf("%04.0f\n",$2);}'`
F=`sysctl vm.stats.vm.v_free_count    | gawk '{$2=$2*4096/1024/1024;printf("%04.0f\n",$2);}'`
T=`sysctl vm.stats.vm.v_page_count    | gawk '{$2=$2*4096/1024/1024;printf("%04.0f\n",$2);}'`
T2="$(( $W + $A + $L + $I + $C + $F))"
G=`echo "$(( $T -$T2))"               | gawk '{$1=$1*4096/1024/1024;printf("%04.0f\n",$1);}'`
echo "   Wired:"$W"M"
echo "  Active:"$A"M"
echo " Laundry:"$L"M"
echo "Inactive:"$I"M"
echo "   Cache:"$C"M"
echo "    Free:"$F"M"
echo "     Gap:"$G"M"
echo "--------------"
echo "   Total:"$T"M"

 

[astyle]

My PC has 32GB memory & 64GB non-encrypted swap partition.
It's not an overkill , as 64GB disk space is cheap.

Sorry, but 64 GB of swap is overkill, even if disk space is cheap. 32 GB RAM allows me to run Poudriere without issues, even if I have ZFS and just 2 GB swap... ?‍