Do you have problems with out of memory problems under FreeBSD?
I am having problems for many years with unresponsive machines when they get without memory and/or swap. As far I can tell, when swap is enabled it will just prolong agony before some application is killed.
Tonight around 5AM I was using Firefox and then everything interactive stopped... GUI panel clock stopped changing, it wasn't possible to switch virtual desktop, nor start xkill or xterm. Try to go to console (Ctrl-Alt-F1) was also unsuccessful.
Daemons (lie hostapd, dhcpd, pf) continue to work, but I was not able to ssh to machine.
And then... nothing, no shell, nothing.
The machine was leaved as is and after 12h state was same: black monitor, no reaction to keyboard or mouse. Ssh login was also in same state.
There is nothing in logs (last log was at 2:50) until the computer is rebooted.
After reboot I can use machine normally, but nothing in logs:
Same problem is observed on two machines, one 12.0, other 11.2-RELEASE, with 16 and 8 GB of RAM.
Both machines are my dev machines with ZFS and usually firefox, music player, GUI filemanager and many xterms with tmux and vim.
It is not nice when server grade OS stops reacting to anything expect power cycling...
How to prevent that from happening (expect maybe rebooting every day)? Is there a way to limit resource consumption by program (limits(1)/rcctl(1)/...something)?
Sometimes there are logs (and machine will become responsive after 10 or more minutes).
Sometimes there are logs like this (but currently I do not have them in /var/log/messages):
I am not too much worried about killing leaky web browser or music player, the machine unresponsivibilty is what concerns me.
I can recompile and install debug kernel or something, it's not a problem.
I am having problems for many years with unresponsive machines when they get without memory and/or swap. As far I can tell, when swap is enabled it will just prolong agony before some application is killed.
Tonight around 5AM I was using Firefox and then everything interactive stopped... GUI panel clock stopped changing, it wasn't possible to switch virtual desktop, nor start xkill or xterm. Try to go to console (Ctrl-Alt-F1) was also unsuccessful.
Daemons (lie hostapd, dhcpd, pf) continue to work, but I was not able to ssh to machine.
Code:
other_machine% ssh machine1
Last login: Thu Feb 21 05:11:25 2019 from unix:0.0
FreeBSD 11.2-RELEASE-p8 (GENERIC) #0: Tue Jan 8 21:35:12 UTC 2019
The machine was leaved as is and after 12h state was same: black monitor, no reaction to keyboard or mouse. Ssh login was also in same state.
There is nothing in logs (last log was at 2:50) until the computer is rebooted.
After reboot I can use machine normally, but nothing in logs:
Code:
# less /var/log/messages
Feb 21 01:22:54 innovator sshd[64784]: error: maximum authentication attempts exceeded for invalid user admin from 94.190.122.195 port 34315 ssh2 [preauth]
Feb 21 02:26:04 innovator in.tftpd[5747]: RRQ from 184.105.139.98 filename a.pdf
Feb 21 02:40:13 innovator sshd[81165]: error: Received disconnect from 185.254.120.6 port 14618:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:40:15 innovator sshd[81901]: error: Received disconnect from 185.254.120.6 port 15936:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:40:32 innovator sshd[82338]: error: Received disconnect from 185.254.120.6 port 16246:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:40:35 innovator sshd[83660]: error: Received disconnect from 185.254.120.6 port 18850:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
...
Feb 21 02:50:00 innovator sshd[34824]: error: Received disconnect from 185.254.120.6 port 41940:3: com.jcraft.jsch.JSchException: Auth fail [preauth]
Feb 21 02:50:11 innovator sshd[34963]: error: Received disconnect from 185.254.120.6 port 42573:3: java.net.SocketTimeoutException: Read timed out [preauth]
Feb 21 16:48:22 innovator syslogd: kernel boot file is /boot/kernel/kernel
Feb 21 16:48:22 innovator kernel: Copyright (c) 1992-2018 The FreeBSD Project.
Same problem is observed on two machines, one 12.0, other 11.2-RELEASE, with 16 and 8 GB of RAM.
Both machines are my dev machines with ZFS and usually firefox, music player, GUI filemanager and many xterms with tmux and vim.
It is not nice when server grade OS stops reacting to anything expect power cycling...
How to prevent that from happening (expect maybe rebooting every day)? Is there a way to limit resource consumption by program (limits(1)/rcctl(1)/...something)?
Code:
machine1% uname -sr
FreeBSD 11.2-RELEASE-p8
machine1% limits
Resource limits (current):
cputime infinity secs
filesize infinity kB
datasize 2097152 kB
stacksize 524288 kB
coredumpsize infinity kB
memoryuse infinity kB
memorylocked 64 kB
maxprocesses 12085
openfiles 231435
sbsize infinity bytes
vmemoryuse infinity kB
pseudo-terminals infinity
swapuse infinity kB
kqueues infinity
umtxp infinity
machine2% uname -sr
FreeBSD 12.0-RELEASE-p2
machine2% limits
Resource limits (current):
cputime infinity secs
filesize infinity kB
datasize 4194304 kB
stacksize 524288 kB
coredumpsize infinity kB
memoryuse infinity kB
memorylocked 64 kB
maxprocesses 19486
openfiles 468477
sbsize infinity bytes
vmemoryuse infinity kB
pseudo-terminals infinity
swapuse infinity kB
kqueues infinity
umtxp infinity
Sometimes there are logs (and machine will become responsive after 10 or more minutes).
Code:
# grep -i swap /var/log/messages
/var/log/messages:Jan 30 01:14:28 innovator kernel: GEOM_ELI: Device gpt/swap.eli destroyed.
/var/log/messages:Jan 30 01:14:28 innovator kernel: GEOM_ELI: Detached gpt/swap.eli on last close.
/var/log/messages:Jan 31 10:06:01 innovator kernel: pid 18922 (deadbeef), uid 1001, was killed: out of swap space
/var/log/messages:Feb 9 07:52:39 innovator kernel: pid 41854 (waterfox), uid 1001, was killed: out of swap space
/var/log/messages:Feb 14 12:52:35 innovator kernel: pid 36817 (deadbeef), uid 1001, was killed: out of swap space
/var/log/messages:Feb 16 21:07:42 innovator kernel: pid 89879 (waterfox), uid 1001, was killed: out of swap space
/var/log/messages:Feb 21 16:48:23 innovator kernel: GEOM_ELI: Device gpt/swap.eli created.
/var/log/messages:Feb 21 16:48:59 innovator kernel: GEOM_ELI: Device gpt/swap.eli destroyed.
/var/log/messages:Feb 21 16:48:59 innovator kernel: GEOM_ELI: Detached gpt/swap.eli on last close.
Sometimes there are logs like this (but currently I do not have them in /var/log/messages):
Code:
swap_pager_getswapspace(x): failed
I am not too much worried about killing leaky web browser or music player, the machine unresponsivibilty is what concerns me.
I can recompile and install debug kernel or something, it's not a problem.