Out of memory

JohnnySorocil · Feb 23, 2019

Do you have problems with out of memory problems under FreeBSD?
I am having problems for many years with unresponsive machines when they get without memory and/or swap. As far I can tell, when swap is enabled it will just prolong agony before some application is killed.

Tonight around 5AM I was using Firefox and then everything interactive stopped... GUI panel clock stopped changing, it wasn't possible to switch virtual desktop, nor start xkill or xterm. Try to go to console (Ctrl-Alt-F1) was also unsuccessful.
Daemons (lie hostapd, dhcpd, pf) continue to work, but I was not able to ssh to machine.

Code:

other_machine% ssh machine1
Last login: Thu Feb 21 05:11:25 2019 from unix:0.0
FreeBSD 11.2-RELEASE-p8 (GENERIC) #0: Tue Jan  8 21:35:12 UTC 2019

And then... nothing, no shell, nothing.
The machine was leaved as is and after 12h state was same: black monitor, no reaction to keyboard or mouse. Ssh login was also in same state.
There is nothing in logs (last log was at 2:50) until the computer is rebooted.
After reboot I can use machine normally, but nothing in logs:

Code:

# less /var/log/messages
Feb 21 01:22:54 innovator sshd[64784]: error: maximum authentication attempts exceeded for invalid user admin from 94.190.122.195 port 34315 ssh2 [preauth]
Feb 21 02:26:04 innovator in.tftpd[5747]: RRQ from 184.105.139.98 filename a.pdf
Feb 21 02:40:13 innovator sshd[81165]: error: Received disconnect from 185.254.120.6 port 14618:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:40:15 innovator sshd[81901]: error: Received disconnect from 185.254.120.6 port 15936:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:40:32 innovator sshd[82338]: error: Received disconnect from 185.254.120.6 port 16246:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:40:35 innovator sshd[83660]: error: Received disconnect from 185.254.120.6 port 18850:3: com.jcraft.jsch.JSchException: Auth fail [preauth]

...

Feb 21 02:50:00 innovator sshd[34824]: error: Received disconnect from 185.254.120.6 port 41940:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:50:11 innovator sshd[34963]: error: Received disconnect from 185.254.120.6 port 42573:3: java.net.SocketTimeoutException: Read timed out [preauth]
Feb 21 16:48:22 innovator syslogd: kernel boot file is /boot/kernel/kernel
Feb 21 16:48:22 innovator kernel: Copyright (c) 1992-2018 The FreeBSD Project.

Same problem is observed on two machines, one 12.0, other 11.2-RELEASE, with 16 and 8 GB of RAM.
Both machines are my dev machines with ZFS and usually firefox, music player, GUI filemanager and many xterms with tmux and vim.
It is not nice when server grade OS stops reacting to anything expect power cycling...
How to prevent that from happening (expect maybe rebooting every day)? Is there a way to limit resource consumption by program (limits(1)/rcctl(1)/...something)?

Code:

machine1% uname -sr
FreeBSD 11.2-RELEASE-p8

machine1% limits
Resource limits (current):
  cputime              infinity secs
  filesize             infinity kB
  datasize              2097152 kB
  stacksize              524288 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked               64 kB
  maxprocesses            12085
  openfiles              231435
  sbsize               infinity bytes
  vmemoryuse           infinity kB
  pseudo-terminals     infinity
  swapuse              infinity kB
  kqueues              infinity
  umtxp                infinity

machine2% uname -sr
FreeBSD 12.0-RELEASE-p2

machine2% limits
Resource limits (current):
  cputime              infinity secs
  filesize             infinity kB
  datasize              4194304 kB
  stacksize              524288 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked               64 kB
  maxprocesses            19486
  openfiles              468477
  sbsize               infinity bytes
  vmemoryuse           infinity kB
  pseudo-terminals     infinity
  swapuse              infinity kB
  kqueues              infinity
  umtxp                infinity

Sometimes there are logs (and machine will become responsive after 10 or more minutes).

Code:

# grep -i swap /var/log/messages
/var/log/messages:Jan 30 01:14:28 innovator kernel: GEOM_ELI: Device gpt/swap.eli destroyed.
/var/log/messages:Jan 30 01:14:28 innovator kernel: GEOM_ELI: Detached gpt/swap.eli on last close.
/var/log/messages:Jan 31 10:06:01 innovator kernel: pid 18922 (deadbeef), uid 1001, was killed: out of swap space
/var/log/messages:Feb  9 07:52:39 innovator kernel: pid 41854 (waterfox), uid 1001, was killed: out of swap space
/var/log/messages:Feb 14 12:52:35 innovator kernel: pid 36817 (deadbeef), uid 1001, was killed: out of swap space
/var/log/messages:Feb 16 21:07:42 innovator kernel: pid 89879 (waterfox), uid 1001, was killed: out of swap space
/var/log/messages:Feb 21 16:48:23 innovator kernel: GEOM_ELI: Device gpt/swap.eli created.
/var/log/messages:Feb 21 16:48:59 innovator kernel: GEOM_ELI: Device gpt/swap.eli destroyed.
/var/log/messages:Feb 21 16:48:59 innovator kernel: GEOM_ELI: Detached gpt/swap.eli on last close.

Sometimes there are logs like this (but currently I do not have them in /var/log/messages):

Code:

swap_pager_getswapspace(x): failed

I am not too much worried about killing leaky web browser or music player, the machine unresponsivibilty is what concerns me.
I can recompile and install debug kernel or something, it's not a problem.

Bobi B. · Feb 23, 2019

Usually when machine starts swapping performance falls down to zero. How much RAM do you have? Can you describe your hardware configuration? Do you use this machine as a server or as a desktop? How long until you run out of free RAM? You should pinpoint program responsible for out-of-RAM condition. Better disable swap temporarily. Also, since you're using ZFS, limit ARC size to 1/3 or 1/2 of RAM, depending on what programs you run.

There are built-in utilities to monitor resources use.

PMc · Feb 23, 2019

JohnnySorocil said:
Do you have problems with out of memory problems under FreeBSD?

No. Not at all. Thats what swap is for: machine gets slow, but is recoverable and the issue can be analyzed.

It is not nice when server grade OS stops reacting to anything expect power cycling...

There may be a little misunderstanding here: a server grade machine is not simply "better". It is more reliable IF it is handled in a reliable way: a server setup needs proper planning. A consumer machine may forgive sloppy operation, a server machine usually does not.

The other problem is that you don't have an imminent console. If you switch from graphics to console, that operation needs an ugly lot of ressources - so if you are already in a ressource exhaustion situation, at that point you're usually lost.

I am having problems for many years with unresponsive machines when they get without memory and/or swap. As far I can tell, when swap is enabled it will just prolong agony before some application is killed.

Well, in that case the most common approach is to have a look upon what is eating how much of the memory, and how much is currently left - before it gets exhausted. There is a tool called top that continuousely shows that.

It could be interesting to wait at that point until it maybe spits out an error message - which it may or may not do after a couple of minutes. This may give a clue about what is actually wrong.

I somehow doubt that this is out-of-memory. I suppose it is some other ressource exhaustion (buffers, process slots, whatever) or a defect.

How to prevent that from happening (expect maybe rebooting every day)? Is there a way to limit resource consumption by program (limits(1)/rcctl(1)/...something)?

Sure, lots of such. But we need to know what ressource the problem is, before adjusting it. No therapy without diagnosis.

Code:

# grep -i swap /var/log/messages
/var/log/messages:Jan 30 01:14:28 innovator kernel: GEOM_ELI: Device gpt/swap.eli destroyed.
/var/log/messages:Jan 30 01:14:28 innovator kernel: GEOM_ELI: Detached gpt/swap.eli on last close.
/var/log/messages:Jan 31 10:06:01 innovator kernel: pid 18922 (deadbeef), uid 1001, was killed: out of swap space
[code]
Sometimes there are logs like this (but currently I do not have them in /var/log/messages):
[code]
swap_pager_getswapspace(x): failed

Now, this does not look like you're out of swapspace, it looks like somehow you loose your swap device. And then, Your swap is not just a plain disk device - You have encryption on it, and that whole thing is probably stacked above ZFS. And if such a construct doesn't work properly, then You get quite likely the picture you describe.

What I would do, at first I would check if that swap does function at all: grab awk and let it bloat an array until it walks into swap. Watch with top how the swap usage grows, and kill the awk before it fills. Something like this should do:
awk 'END { for(i=0; i< 9999999; i++) a[I]=[/I]"abcdefghij";}' < /dev/null

Second step: Switch off the gpt/eli/whatever swap, and configure a plain unencrypted swap partition outside of ZFS, e.g. on an usb stick. That will be slow, but it should work, and the machine should not stall. See if the failure repeats or is now avoided.

Third step: figure out the ressouce usage. In most cases that is imminent visible with top, otherwise stronger measures are needed. With 8 Gig RAM Your desktop stuff should almost run without swap, and should certainly not stall the machine - unless there is some flaw in the layout.

JohnnySorocil · Feb 26, 2019

Bobi B. said:
Usually when machine starts swapping performance falls down to zero.

Agree. That is the reason why I (usually) disable swap on my machines.

Bobi B. said:
How much RAM do you have? Can you describe your hardware configuration? Do you use this machine as a server or as a desktop? How long until you run out of free RAM? You should pinpoint program responsible for out-of-RAM condition. Better disable swap temporarily. Also, since you're using ZFS, limit ARC size to 1/3 or 1/2 of RAM, depending on what programs you run.

8 GB in one machine, 16 GB in 2nd.
Both machines are used as a desktop (X11 WMs, xterm, text editor, firefox, music player, occasionally some compiling and video playing but nothing fancy).
After few days (and few firefox/waterfox instances later) problem occurs (sometimes it is enough to leave a few web browsers and media player instances and wait).

Code:

                                                                                                                [~][pts/10][19.02.26. 17:47:05]
% zfs-stats -AE

------------------------------------------------------------------------
ZFS Subsystem Report                            Tue Feb 26 17:33:51 2019
------------------------------------------------------------------------

ARC Summary: (THROTTLED)
        Memory Throttle Count:                  32


ARC Misc:
        Deleted:                                2.10m
        Recycle Misses:                         0
        Mutex Misses:                           38.73k
        Evict Skips:                            15.75m

ARC Size:                               47.99%  1.79    GiB
        Target Size: (Adaptive)         50.42%  1.88    GiB
        Min Size (Hard Limit):          22.27%  849.52  MiB
        Max Size (High Water):          4:1     3.73    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       64.62%  1.21    GiB
        Frequently Used Cache Size:     35.38%  680.55  MiB

ARC Hash Breakdown:
        Elements Max:                           511.70k
        Elements Current:               9.95%   50.93k
        Collisions:                             680.71k
        Chain Max:                              7
        Chains:                                 1.20k

------------------------------------------------------------------------

ARC Efficiency:                                 551.47m
        Cache Hit Ratio:                95.25%  525.25m
        Cache Miss Ratio:               4.75%   26.22m
        Actual Hit Ratio:               94.46%  520.91m

        Data Demand Efficiency:         85.73%  106.92m
        Data Prefetch Efficiency:       41.65%  159.30k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.61%   3.22m
          Most Recently Used:           10.05%  52.79m
          Most Frequently Used:         89.12%  468.12m
          Most Recently Used Ghost:     0.06%   330.26k
          Most Frequently Used Ghost:   0.15%   793.74k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  17.45%  91.67m
          Prefetch Data:                0.01%   66.35k
          Demand Metadata:              81.39%  427.47m
          Prefetch Metadata:            1.15%   6.04m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  58.18%  15.26m
          Prefetch Data:                0.35%   92.95k
          Demand Metadata:              37.34%  9.79m
          Prefetch Metadata:            4.12%   1.08m

------------------------------------------------------------------------

Bobi B. said:
There are built-in utilities to monitor resources use.

Can you recommend something which will be useful in my situation?

Yesterday around 16:50 the machine with 8 GB of RAM freezes (same problem as descried - taskbar clock stops, music stops after currently playing song, screen won't go off after few hours, cannot login over ssh, no response to Ctrl-Alt-F1,...).
Problem started when I had to run another Firefox instance. Didn't have memory meter other than the one in Conky, but it was around 99% (or something like that, definitely close to max).
But this time I waited.
I wasn't at home until around 17:00 today, but machine was responsive (music continued, physical console (Ctrl-Alt-F1) was on screen)
logs:

Code:

Feb 23 19:25:46 innovator sshd[7909]: error: maximum authentication attempts exceeded for root from 37.110.104.187 port 45227 ssh2 [preauth]
Feb 23 21:11:33 innovator sshd[78410]: error: maximum authentication attempts exceeded for invalid user admin from 174.103.99.159 port 47898 ssh2 [preauth]
Feb 24 03:50:09 innovator sshd[28129]: error: maximum authentication attempts exceeded for root from 2.179.183.88 port 52200 ssh2 [preauth]
Feb 24 04:47:47 innovator sshd[38586]: error: maximum authentication attempts exceeded for root from 91.137.127.104 port 62504 ssh2 [preauth]
Feb 24 06:23:18 innovator sshd[53559]: error: maximum authentication attempts exceeded for root from 5.224.43.97 port 52978 ssh2 [preauth]
Feb 24 12:43:07 innovator sshd[99155]: error: maximum authentication attempts exceeded for invalid user admin from 116.3.101.243 port 33375 ssh2 [preauth]
Feb 24 16:47:37 innovator sshd[17678]: error: maximum authentication attempts exceeded for root from 182.45.53.77 port 38863 ssh2 [preauth]
Feb 25 10:41:01 innovator sshd[3877]: error: maximum authentication attempts exceeded for root from 222.220.84.95 port 40538 ssh2 [preauth]
Feb 25 12:09:43 innovator sshd[82956]: error: maximum authentication attempts exceeded for invalid user admin from 222.113.145.41 port 37131 ssh2 [preauth]
Feb 25 13:42:03 innovator sshd[80577]: error: maximum authentication attempts exceeded for root from 58.236.98.240 port 41500 ssh2 [preauth]
Feb 25 16:19:46 innovator sshd[29762]: error: maximum authentication attempts exceeded for root from 42.115.173.65 port 49458 ssh2 [preauth]
Feb 25 17:18:34 innovator sshd[11175]: error: maximum authentication attempts exceeded for root from 115.55.152.216 port 27027 ssh2 [preauth]
Feb 25 18:43:40 innovator ppp[11659]: tun0: Warning: ff02::/: Change route failed: errno: Network is unreachable
Feb 25 18:43:50 innovator ppp[11659]: tun0: Warning: ff02::/: Change route failed: errno: Network is unreachable
Feb 25 18:49:04 innovator sshd[68665]: error: maximum authentication attempts exceeded for invalid user admin from 93.138.63.121 port 52693 ssh2 [preauth]
Feb 25 23:04:23 innovator sshd[77023]: error: maximum authentication attempts exceeded for root from 119.130.105.246 port 49886 ssh2 [preauth]
Feb 26 08:15:03 innovator kernel: ath0: stuck beacon; resetting (bmiss count 4)
Feb 26 08:52:32 innovator kernel: ath0: device timeout
Feb 26 11:45:34 innovator kernel: pid 4188 (waterfox), uid 1001, was killed: out of swap space
Feb 26 14:31:47 innovator sshd[28062]: error: maximum authentication attempts exceeded for root from 119.130.105.246 port 43634 ssh2 [preauth]
Feb 26 16:27:17 innovator sshd[51415]: error: maximum authentication attempts exceeded for root from 123.24.67.128 port 47838 ssh2 [preauth]
Feb 26 17:08:42 innovator sshd[75261]: error: maximum authentication attempts exceeded for root from 81.83.212.173 port 40886 ssh2 [preauth]

So, after more than half a day machine comes to life with logs which aren't that descriptive...

Sevendogsbsd · Feb 26, 2019

There was a 10,000 (kidding) post thread on Firefox causing issues like this here, can't remember the thread title - search for it. May have been fixed. I have no issues like this but I don't use Firefox and my machine only stays up around 8 hours max - I shut down after using.

I "think" this is it: Thread 67657 but seems shorter than I remember.

JohnnySorocil · Feb 26, 2019

PMc said:
The other problem is that you don't have an imminent console. If you switch from graphics to console, that operation needs an ugly lot of ressources - so if you are already in a ressource exhaustion situation, at that point you're usually lost.

Interesting, didn't know what. I'll try to reproduce it while on console and see how it goes

PMc said:
Well, in that case the most common approach is to have a look upon what is eating how much of the memory, and how much is currently left - before it gets exhausted. There is a tool called top that continuousely shows that.

Usually the memory heavy programs are ran before exhaustion. But I expected that then guilty process will be killed, not whole machine to basically halt.

PMc said:
It could be interesting to wait at that point until it maybe spits out an error message - which it may or may not do after a couple of minutes. This may give a clue about what is actually wrong.

Well, I have waited for a day, and there was only one useful line in log (freeze occurred approximately Feb 25 16:50):

Code:

Feb 26 11:45:34 innovator kernel: pid 4188 (waterfox), uid 1001, was killed: out of swap space

PMc said:
I somehow doubt that this is out-of-memory. I suppose it is some other ressource exhaustion (buffers, process slots, whatever) or a defect.

Something like this?

Code:

sysctl -h vfs | \grep buffer

PMc said:
Now, this does not look like you're out of swapspace, it looks like somehow you loose your swap device. And then, Your swap is not just a plain disk device - You have encryption on it, and that whole thing is probably stacked above ZFS. And if such a construct doesn't work properly, then You get quite likely the picture you describe.

Sorry, that was part of the log when I have disabled swap device after boot. Swap partition (when used) is plain GPT partition:

Code:

# swapinfo
Device          1K-blocks     Used    Avail Capacity
# gpart show -p ada0
=>       40  468862048    ada0  GPT  (224G)
         40       2008          - free -  (1.0M)
       2048     204800  ada0p1  efi  (100M)
     206848       1024  ada0p2  freebsd-boot  (512K)
     207872       1024          - free -  (512K)
     208896  461373440  ada0p3  freebsd-zfs  (220G)
  461582336    2097152  ada0p4  freebsd-swap  (1.0G)
  463679488    5182600          - free -  (2.5G)

PMc said:
What I would do, at first I would check if that swap does function at all: grab awk and let it bloat an array until it walks into swap. Watch with top how the swap usage grows, and kill the awk before it fills. Something like this should do:
awk 'END { for(i=0; i< 9999999; i++) a[I]=[/I]"abcdefghij";}' < /dev/null

Second step: Switch off the gpt/eli/whatever swap, and configure a plain unencrypted swap partition outside of ZFS, e.g. on an usb stick. That will be slow, but it should work, and the machine should not stall. See if the failure repeats or is now avoided.

Third step: figure out the ressouce usage. In most cases that is imminent visible with top, otherwise stronger measures are needed. With 8 Gig RAM Your desktop stuff should almost run without swap, and should certainly not stall the machine - unless there is some flaw in the layout.

Thanks for the suggestions, I'll try to reproduce the same problem on machine with 16 GB RAM (and limited ARC).

JohnnySorocil · Feb 26, 2019

Sevendogsbsd said:
There was a 10,000 (kidding) post thread on Firefox causing issues like this here, can't remember the thread title - search for it. May have been fixed. I have no issues like this but I don't use Firefox and my machine only stays up around 8 hours max - I shut down after using.

I "think" this is it: Thread 67657 but seems shorter than I remember.

Thanks, I'll look into it! Firefox is definitely a memory pig

But, can I prevent some program (Firefox or not, shouldn't matter) from doing this to machine? I would prefer that OS kills memory intensive program before it stalls the whole machine for hours.

SirDice · Feb 26, 2019

JohnnySorocil said:
Agree. That is the reason why I (usually) disable swap on my machines.

Contrary to popular belief, swapping in and of itself is NOT bad. It's excessive swapping that will cause performance issues. And even then it may not be bad, if the choice is between a system that completely hangs itself up due to memory starvation or a system that works, slowly, but still works, then the choice should be simple.

In short, always add swap.

shkhln · Feb 26, 2019

JohnnySorocil said:
I would prefer that OS kills memory intensive program before it stalls the whole machine for hours.

FreeBSD has OOM killer, but it's not very reliable as you can see.

JohnnySorocil · Feb 26, 2019

shkhln said:
FreeBSD has OOM killer, but it's not very reliable as you can see.

Maybe something in sys/swap_pager.c and/or sys/vm_pageout.c can be tuned?
In worst case, I would be OK if somehow bad process name can be hardcoded somewhere (so at least, X11 and good processes(tm) survive).

Or, can somehow 32-128-NNN MB of RAM be reserved for responsiveness of OOM killer (so it has enough to kill bad processes)?

PMc · Feb 26, 2019

JohnnySorocil said:
8 GB in one machine, 16 GB in 2nd.
Both machines are used as a desktop (X11 WMs, xterm, text editor, firefox, music player, occasionally some compiling and video playing but nothing fancy).
After few days (and few firefox/waterfox instances later) problem occurs (sometimes it is enough to leave a few web browsers and media player instances and wait).

Code:

ARC Summary: (THROTTLED) Memory Throttle Count: 32

Something is very strange here.
My desktop was running with 4G memory until last month. With firefox, 1-2 vlc, occasionally 1-2 additional browsers, OpenOffice, occasionally recompiling OS and all ports - but I do not recall ever having seen a count on this memory throttle metric. I had no problems whatsoever, except that it gets very slow when paging, and firefox producing coredumps at termination.

Yesterday around 16:50 the machine with 8 GB of RAM freezes (same problem as descried - taskbar clock stops, music stops after currently playing song, screen won't go off after few hours, cannot login over ssh, no response to Ctrl-Alt-F1,...).

This is not a typical swapping behaviour. Swapping may reduce speed to about 1/100 (which is much too slow for most people), but it does not freeze things (for longer than occasionally a couple of seconds).

Problem started when I had to run another Firefox instance.

That may indeed hurt. I found the newer firefox (with multiple content processes) to use an extreme amount of memory: about 2-3 Gig per installed CPU! So, depending on how many CPUs there are, a second one of that ilk may lead into troubles.
-> Go to "Preferences", uncheck the "use recommended performance settings" box, see what it says for "content process limit" (should be the number of your CPUs), and set it to a sensible value (2 or 3 should do). Set it to 1 for all secondary instances.

Bobi B. · Feb 26, 2019

You can try with setting resource limits, but the way you do it depends from the shell you use. With tcsh(1) the command you need is limit, whereas with sh(1) and bash(1) it is probably ulimit.

To monitor resource use you can try top(1) ( top -S -o size), of course, systat(1) ( systat -vmstat), vmstat(8) ( vmstat 1). To monitor process' resource usage try procstat(1) ( procstat -l <pid>, procstat -r <pid>). top(1) can also show you processes doing most I/O with top -S -m io -o total and gstat(8) will display per-disk/partition utilization.

You might leave some of those running in a X11 terminal so you can look for culprit when you notice the machine is going down.

PMc · Feb 26, 2019

Sorry, that was part of the log when I have disabled swap device after boot. Swap partition (when used) is plain GPT partition:

Code:

# swapinfo
Device          1K-blocks     Used    Avail Capacity
# gpart show -p ada0
=>       40  468862048    ada0  GPT  (224G)
         40       2008          - free -  (1.0M)
       2048     204800  ada0p1  efi  (100M)
     206848       1024  ada0p2  freebsd-boot  (512K)
     207872       1024          - free -  (512K)
     208896  461373440  ada0p3  freebsd-zfs  (220G)
  461582336    2097152  ada0p4  freebsd-swap  (1.0G)
  463679488    5182600          - free -  (2.5G)

No that won't work. Give the beast some swap it can use. 10 Gig for starters. Or at least give it the 3.5 that are there.

PMc · Feb 26, 2019

JohnnySorocil said:
Thanks, I'll look into it! Firefox is definitely a memory pig

But, can I prevent some program (Firefox or not, shouldn't matter) from doing this to machine? I would prefer that OS kills memory intensive program before it stalls the whole machine for hours.

Well, but I would NOT prefer such behaviour.

I want it to try and struggle its way through, if that at any means might work out. It's a server OS - and if somebody sends a bloated query to the database, we do NOT want it to continue running with one of the database processes killed (and that way the whole thing becoming somehow half-operative).

racct should allow to do that, it can limit memory usage. But I didn't try that (as I don't want killed processes). What I did try was to use it to limit core memory usage and force the process to page out early and leave space for other processes (that obviousely requires ample swap space). That one works, but it puts much useless load on the vmdaemon, so the result didn't look very feasible to me.

shkhln · Feb 26, 2019

PMc said:
This is not a typical swapping behaviour. Swapping may reduce speed to about 1/100 (which is much too slow for most people), but it does not freeze things (for longer than occasionally a couple of seconds).

It's not a swapping behavior, it's a memory exhaustion behavior. Once you are out of physical and swap memory anything trying to call malloc (or mmap) is going to stall indefinitely.

PMc said:
Something is very strange here.

Nothing is strange here. I have a similar configuration (that is, zfs with swap disabled) and I occasionally get similar lockups. I don't believe it's a consequence of the disabled swap, presumably they are just easier to trigger that way.

PMc · Feb 26, 2019

shkhln said:
It's not a swapping behavior, it's a memory exhaustion behavior. Once you are out of physical and swap memory anything trying to call malloc (or mmap) is going to stall indefinitely.

Yepp.

Nothing is strange here. I have a similar configuration (that is, zfs with swap disabled) and I occasionally get similar lock ups. I don't believe it's a consequence of a disabled swap, presumably they are just easier to trigger that way.

Ah. I never disable swap. And it is indeed not really strange when considering that there is only 1 Gig swap - and that does not help with 8 Gig memory. So if there are maybe 4 processors, firefox in default configuration might likely expand to 8 Gig for itself, will sqeeze the zfs arc to uselessness, will at the same time walk out to swap, will soon hit the end of the swap, and there the show stops. From that viewpoint it figures.

Deleted member 30996 · Feb 27, 2019

Here's the business end of my X61 .mp3 player running FreeBSD 11.1-RELEASE-p10 at 292 days uptime. sysutils/screenfetch isn't showing the CPU for some reason but it's an Intel Core 2 Duo T7300 @ 2.00 GHz with 4GB RAM and Scorpio Black 200GB HDD:

It shows a total of 3947MB Swap allocated with 82MB in use, or 3947MB with 2% in use as you prefer.

I never turn sysutils/gkrellm2 or multimedia/xmms off and leave what songs I have loaded playing so I can just pick up my headphones and hear music. I never take it online and it couldn't be running better so I don't see the need to update ATM, but is a full FreeBSD build and every bit as capable as the T61 running FreeBSD 12.0-RELEASE-p3 with same specs I use as a desktop.

The T61 currently shows 3979 MB Swap allocated with 5608K in use at 6 days uptime. The T400 with Intel Core2 Duo P8600 @ 2.40GHz, 8GB RAM and Scorpio Black 200GB HDD running FreeBSD 11.2-RELEASE-p9 I'm on now is showing Swap at 3979MB allocated and 3979MB free at 3 days uptime.

I've never had a problem with Swap on any of my machines and they're never short on RAM no matter what I use them for.

Deleted member 48958 · Feb 27, 2019

JohnnySorocil said:
Do you have problems with out of memory problems under FreeBSD?

Yes. I started to have some problems with memory usage after update to 11.2, as far as I remember, IMO something happened after that release. I always use 2 machines at the same time, I use first machine to watch some content-listen to the music, while I work on my laptop, I use ssh and vnc to manipulate first machine. First machine is running FreeBSD 12 now, second one - Devuan 9. And I've almost never saw full RAM memory on Linux machine, while after the upgrade, FreeBSD machine RAM is full almost always... I just need to start to use some memory intensive apps, and soon RAM will be full, and it doesn't fully frees up even when I close all applications, about 1.5-2 GB of RAM is always full, as well as some amount of SWAP. Maybe it's some kind of glitch, which appeared after freebsd-update, and I just need to reinstall this OS, but unfortunately, I don't have much time for now, to make full reinstallation. BTW, I had much bigger memory problems with 11.2 release, RAM and SWAP were almost always full (I've downgraded to 10.4 release at that time, which used to work well for me, including memory usage), 12.0 works a little bit better, but anyway, 11.0 used to use memory much more reasonably for me.

twllnbrck · Feb 27, 2019

ILUXA said:
I just need to start to use some memory intensive apps, and soon RAM will be full, and it doesn't fully frees up even when I close all applications, about 1.5-2 GB of RAM is always full, as well as some amount of SWAP.

I recognized similar issues since 11.2-R upgrade. The largest memory junk was firefox what finally has caused me to switch to www/chromium. It is still a heavy browser and a memory hog but 1 site open on both firefox is consuming half more RAM.
But the problem with not fully freed up memory remains even on 12.0-p3.

PacketMan · Mar 19, 2019

I'm running 11.2-RELEASE on a headless home server, with 16GB ram, with a fair few programs running, and as far as I am concerned there is no memory issues there. I could let that thing run for a year and it would keep on going. And although I have not written down any numbers, I'm pretty sure any program I shut down, its memory is 'released' and 'free for future use'.

usdmatt · Mar 19, 2019

Much as I like ZFS, I've had memory issues with it since day 1, and always limit ARC manually. I tried a new server a few weeks ago without a manual limit to see if it was any better, but it started killing processes and the NFS service (which is what it was being used for) went down within 24 hours. I actually posted on the mailing list about it as I thought they'd been improvements made over the years but only got a few "me too" responses.

Not trying to knock FreeBSD too much, I've used it since 3.x and much prefer it to everything else; It's just frustrating that after more than 10 years, ZFS still seems, in my experience at least, to need draconian memory limits to stop the entire system starving itself to death.

twllnbrck · Mar 19, 2019

usdmatt said:
Much as I like ZFS, I've had memory issues with it since day 1, and always limit ARC manually

Im also playing around with ZFS tuning on my desktop PC (8GB RAM). What tunables do you change besides vfs.zfs.arc_max and are there any rule of thumbs to get some improvements?

D-FENS · Mar 19, 2019

usdmatt said:
Much as I like ZFS, I've had memory issues with it since day 1, and always limit ARC manually. I tried a new server a few weeks ago without a manual limit to see if it was any better, but it started killing processes and the NFS service (which is what it was being used for) went down within 24 hours. I actually posted on the mailing list about it as I thought they'd been improvements made over the years but only got a few "me too" responses.

Not trying to knock FreeBSD too much, I've used it since 3.x and much prefer it to everything else; It's just frustrating that after more than 10 years, ZFS still seems, in my experience at least, to need draconian memory limits to stop the entire system starving itself to death.

How much RAM does your system have? And how much disk space?

Ancient · Mar 19, 2019

Look...I know that FreeBSD installations consume 2GB of minimum. But you mentioned that you have a few programs...look: only Xorg consumes 1GB. I know it because I used it.

usdmatt · Mar 19, 2019

How much RAM does your system have? And how much disk space?

The system I was running the other day had 12GB RAM and 2TB of storage (4 1TB disks in mirrors). I currently have ARC limited to 10GB and it's been fine since. Of course it shouldn't really matter that much. Filling a system with 10TB disks shouldn't mean you suddenly need 128GB of RAM to stop it from falling over. ARC should always leave some memory free and release some if the system starts to run out.

Out of memory

JohnnySorocil

Bobi B.

PMc

JohnnySorocil

Sevendogsbsd

JohnnySorocil

JohnnySorocil

SirDice

Administrator

shkhln

JohnnySorocil

PMc

Bobi B.

PMc

PMc

shkhln

PMc

Deleted member 30996

Guest

Deleted member 48958

Guest

twllnbrck

PacketMan

usdmatt

twllnbrck

D-FENS

Ancient

usdmatt