Help understand what FreeBSD froze

fred974

fred974

Hi,

My server went offline in the night and I managed to fix it by rebooting the system.
Before rebooting, I was unable to ssh into the server or any jails.
When directly connected to the keyboard, keystroke was not recorded

I run a FreeBSD host based on 12.0-RELEASE and I have several jails running..
13x webjail
1x DBjail
1x webprox
1x mailjail
1x bhyve vm

When I looked at the /var/log/messages I have a lot of the following entries which reconcile with the monitoring tool showing the sites been down.
Code: 
Apr 25 01:45:34 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (2 occurrences)
Apr 25 01:46:45 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (19 occurrences)
Apr 25 01:47:45 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (15 occurrences)
Apr 25 01:48:56 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (14 occurrences)
Apr 25 01:49:57 FreeBSD-node1 syslogd: last message repeated 1 times
Apr 25 01:50:58 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (20 occurrences)
Apr 25 01:51:58 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (22 occurrences)
Apr 25 01:52:59 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (10 occurrences)
Apr 25 01:53:59 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (11 occurrences)
Apr 25 01:55:00 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (19 occurrences)
Apr 25 01:56:08 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (20 occurrences)
Apr 25 01:57:15 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8064a254000: Listen queue overflow: 193 already in queue awaiting acceptance (12 occurrences)
Apr 25 01:58:19 FreeBSD-node1 syslogd: last message repeated 1 times
I looked at dovecot log on the email jail and found noting out of the ordinary
Could anyone please help me undestand what happened?
 
I also found on google that this issue can be link to network card..
I have a QLogic NetXtreme II BCM5709. I have the following network setup
Code: 
ifconfig_bce0="up"
ifconfig_bce1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto failover laggport bce0 laggport bce1 194.xxx.13.xxx/29"
defaultrouter="194.xxx.13.xxx"
 
Listen queue overflows are often a symptom, not a cause. They can happen when the listening service can't handle connections fast enough and so it starts queuing connections. Find out why the service is slow. Running out of disk space or memory are often the underlying issues. Also look at your I/O load, if it's constantly more than 80% busy it will also cause slow downs for everything that requires disk access.

If you have a LOT of connections coming in you can also try to raise kern.ipc.soacceptqueue, the default value (128) is fairly conservative.
 
Look at the output of netstat -LAan. You'll find similar hex addresses, try to match that with the one from your logs. But I often find the logs refer to a process that has since stopped so you can't find it in the netstat(1) output. Then it's usually an educated guess which process might be the culprit.
 
You must log in or register to reply here.
Back
Top