64 GB RAM yet applications keeps crashing - what to do?

[mer]

Sorry, but 64 GB of swap is overkill, even if disk space is cheap. 32 GB RAM allows me to run Poudriere without issues, even if I have ZFS and just 2 GB swap... ?‍

In general I agree, but in this case of trying to recover, especially with the high laundry usage, I'd toss an external device of that size (if one has it handy) and enable as swap temporarily. A quick check of wally world shows 64G USB3 thumb drive is roughly $15 US.

But that's all my opinion

 

[smithi]

This swap-not-emptying problem for me has existed since ages - and to touch upon your model of laptop direction - it's been the same issue with two different models of laptops.

Di you have a specific reason not to reveal make/model of this laptop? It's always been question #1 with laptops, as so many have make and/or model issues; see the wiki.

Maybe I'm just unlucky or maybe this is a genuine issue - I suspect the latter.

Nothing to do with luck, but which issue is the issue ... did you peruse that recent thread on freebsd-current that yuripv79 posted in #11?

Also, try answering Mer's most recent questions, and try Alain's script?

From earlier:

Trying not to close Chrome since I am in the middle of some work - is there any other way to diagnose the issue?

If chrome is set to save and restore its tabs, it wouldn't take 5 minutes to try closing and reopening it, recording state of swap, laundry etc before and after - given hours for swap to settle.

Also, in that screenshot in post #5, why do the chrome entries show Size 1115G (1.1TB!)? I know it's not much used, but does the system think it could be? Again, restarting chrome should be indicative.

This is what I use to sleep : sudo acpiconf -s 3
Also - this is what is supported on the machine :

sysctl hw.acpi.supported_sleep_state
hw.acpi.supported_sleep_state: S3 S4 S5

Standard.

I suspect the same - suspend (via `acpiconf -s 3` is somehow dumping it to swap and that doesn't clear upon resumption.

Which "it" though? Try NOT suspending after clearing swap by methods above, including reboot if needed, but using it as heavily for a while. I.e. does swap ONLY grow if you've suspended?

If there are any specific commands I could try easily, please let me know.

Ok. In /boot/loader.conf:
boot_verbose="YES"
verbose_loading="YES"
In /etc/sysctl.conf:
hw.acpi.verbose=1

After booting, you'll have much more system info than you usually want, perhaps with useful clues re swap, in /var/log/messages.

After suspend / resume you will see there lots of info, perhaps including any problems related to this.

Seems like an overkill for a 512 G machine to allocate 32/64G to swap - also I'm barely using ~20 G RAM at most peak times. It's the **NOT** _clearing_ of swap that, to my untrained mind, is probably the root cause.

So seek out trained minds - that's what I'd do anyway ..

One more thing you might try:

sysctl -ad | grep swap > somefile
sysctl -a | grep swap >> somefile

Is this a FreeBSD suspend/resume issue? ?

I doubt it, but you'll have to be prepared to try things to find out

 

[bob2112]

The swap always remains full once I cause it to sleep/S3 lid action - and doesn't empty despite memory available somehow

Swap is supposed to remain in use if it's still caching valid data. People that try to clear it with swapon/swapoff are just making the kernel run less efficiently. It's the filling of swap that's the odd thing.

 

[Tracker]

Sorry for getting back a bit late - got preoccupied with a few things.

How often do you suspend?

Quite often actually. My uptime has been 15 days - so whenever I am taking a break and/or going to sleep/eat I simply put the machine to go to sleep, often 3-4 times a day at least.

What happens to swap use if you do a complete power off, restart, then use the system as you normally do?
Does the swap use go up?

Swap starts from 0 upon machine restart. Usually one of the first signs of swap usage going up (from what I recollect) is the machine being put to sleep and then restarted (thus presumably causing swap space to come into play/being full - and after which it usually stays full).

I don't know if suspend writes anything to swap, but since "suspend may save some state" I can imagine it may.

I'd imagine the same

Toss in the encrypted swap: I think that is done with a one time key; is that key actually preserved across suspend? If not, that may exacerbate the problem.

The hard drive is encrypted.

A way to test that theory would be to modify /etc/fstab so swap is not encrypted. Yes it sounds like to get to that point you need to reboot, but it would at least give a datapoint.

So since Chrome was one of the biggest users of RAM/swap - I did a work around

pkill -P $chromepid

- this *did kill* chrome (sub-)processes but it **didn't** cause swap to decrease!

Also, in that screenshot in post #5, why do the chrome entries show Size 1115G (1.1TB!)? I know it's not much used, but does the system think it could be? Again, restarting chrome should be indicative.

Not sure how it can show 115G - my system has nowhere near that capacity. Hope the pkill -P is enough? Didn't seem to have much of an effect.

Di you have a specific reason not to reveal make/model of this laptop? It's always been question #1 with laptops, as so many have make and/or model issues; see the wiki.

I just think it's not a good idea to lay down all specs of a system - however if you must know this particular machine is a thinkpad laptop.

Have you tried adding another swap device, bigger than the current one?

Not yet - I suspect it would cause swap usage to simply toss around between the current swap and new swap, until it probably fills both?

Swap is supposed to remain in use if it's still caching valid data. People that try to clear it with swapon/swapoff are just making the kernel run less efficiently. It's the filling of swap that's the odd thing.

Shouldn't it be clearing out the swap to memory when there is enough of RAM available? Instead it's crashing applications - seems problematic if that's not the intended behaviour.

Alain De Vos I did try your zsh/gawk script - here is the output :

./swaptest.sh
   Wired:5230M
  Active:4344M
 Laundry:30425M
Inactive:3342M
   Cache:0000M
    Free:18872M
     Gap:0005M
--------------
   Total:63522M

One more thing you might try:

sysctl -ad | grep swap > somefile
sysctl -a | grep swap >> somefile

Here is the ouput

sysctl -ad | grep swap
kern.maxswzone: Maximum memory for swap metadata
kern.nswbuf: Number of swap buffers
vm.swap_enabled: Enable entire process swapout
vm.domain.0.stats.unswppdpgs: Unswappable pages scanned by the page daemon
vm.domain.0.stats.unswappable: Unswappable pages
vm.swap_idle_threshold2: Time before a process will be swapped out
vm.swap_idle_threshold1: Guaranteed swapped in time for a process
vm.swap_idle_enabled: Allow swapout on idle criteria
vm.disable_swapspace_pageouts: Disallow swapout of dirty pages
vm.swap_objects: List of swap VM objects
vm.stats.vm.v_swappgsout: Swap pages swapped out
vm.stats.vm.v_swappgsin: Swap pages swapped in
vm.stats.vm.v_swapout: Swap pager pageouts
vm.stats.vm.v_swapin: Swap pager pageins
vm.stats.swap: VM swap stats
vm.stats.swap.free_completed: Number of deferred frees completed
vm.stats.swap.free_deferred: Number of pages that deferred freeing swap space
vm.swap_info: Swap statistics by device
vm.nswapdev: Number of swap devices
vm.dmmax: Maximum size of a swap block in pages
vm.swap_fragmentation: Swap Fragmentation Info
vm.swap_async_max: Maximum running async swap ops
vm.swap_maxpages: Maximum amount of swap supported
vm.swzone: Actual size of swap metadata zone
vm.swap_total: Total amount of available swap storage.
vm.swap_reserved: Amount of swap storage needed to back all allocated anonymous memory.
vfs.tmpfs.memory_reserved: Amount of available memory and swap below which tmpfs growth stops

sysctl -a | grep swap
1 PART ada0p4 8317304832 512 i 4 o 273678336 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
z0xfffff80005e11800 [shape=box,label="SWAP\nswap\nr#4"];
        <type>freebsd-swap</type>
        <label>swap0</label>
      <name>swap</name>
vm.swap_enabled: 1
vm.domain.0.stats.unswappable: 319738
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.swap_idle_enabled: 0
vm.disable_swapspace_pageouts: 0
vm.stats.vm.v_swappgsout: 6015612
vm.stats.vm.v_swappgsin: 4249876
vm.stats.vm.v_swapout: 885071
vm.stats.vm.v_swapin: 761629
vm.stats.swap.free_completed: 1669204
vm.stats.swap.free_deferred: 1952473
vm.nswapdev: 1
vm.swap_fragmentation:
vm.swap_async_max: 4
vm.swap_maxpages: 130094000
vm.swap_total: 8317304832
vm.swap_reserved: 349365651542016

Let me know how what else I can do to diagnose the issue.

 

[mer]

Not yet - I suspect it would cause swap usage to simply toss around between the current swap and new swap, until it probably fills both

But my whole question was:
Add a swap device bigger than the current one, bigger than what is wired in laundry and then do a swapoff of the original swap partition.

If you do this the system should migrate swap that is still in use to the new device and eventually the original swap device will be "unmounted".

I was theorizing that by doing this the system may wind up recognized "swap that's not actually in use anymore" and you would decrease the overall usage of swap.

Shouldn't it be clearing out the swap to memory when there is enough of RAM available?

My understanding is that swap is not cleared until the process that "owns" the swap is exited.

 

[smithi]

One more thing you might try:

Here is the ouput

Yeah, that was really an afterthought, for you and especially others who may spot a clue there. I lack experience in having swap problems.

Let me know how what else I can do to diagnose the issue.

My primary suggestion was to crank up verbose logging to eliminate guesswork, viz:

Ok. In /boot/loader.conf:
boot_verbose="YES"
verbose_loading="YES"
In /etc/sysctl.conf:
hw.acpi.verbose=1

After booting, you'll have much more system info than you usually want, perhaps with useful clues re swap, in /var/log/messages.

After suspend / resume you will see there lots of info, perhaps including any problems related to this.

And do add more swap as advised, but a little learning about the grittier details is unlikely to hurt much