Listen queue overflow

dvl@

dvl@

Developer
This post is:
  • part documentation
  • part looking for suggestions
I see three pages of search results for 'Listen queue overflow'. Over the past year or two I've tried several times to learn the source of these messages on my systems.

I see it happens from time to time on all of my servers at home, but I have been unable to locate the cause. I search the forum, try the suggestions found there, but no solution/cause found.

All of these servers have varying workloads and different uses.

Looking in the logs, this, happens at various times during the day,

Code: 
Feb 11 01:59:13 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 11 04:49:14 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 11 12:18:24 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 11 12:33:23 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (2 occurrences)
Feb 11 17:54:20 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (2 occurrences)
Feb 11 17:57:49 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 11 19:03:29 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (4 occurrences)
Feb 13 03:04:42 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 13 03:33:27 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (4 occurrences)
Feb 13 19:04:55 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 14 22:15:08 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (4 occurrences)
Feb 16 23:33:40 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 17 01:48:42 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 17 13:03:31 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (4 occurrences)
Feb 17 16:04:27 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 17 16:33:34 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (5 occurrences)
Feb 18 03:14:39 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 18 08:18:40 r720-01 kernel: sonewconn: pcb 0xfffff801459821e8: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)

Of note:
  • is it always 8 already in queue awaiting
  • it is not always the same pcb number e.g. 0xfffff801459821e8
Based on that 8, I thought I should look for a queue size less than 8, i.e. maxqlen in the output of netstat -aL.

r720-01 is a Dell R720 running FreeBSD 12.1 - it is primarily a jail server (more details in this blog post) with 384GB RAM.

Let's look at netstat -aL from 5 hours after the last such message.

The entries with maxqlen <= 8 are:

Code: 
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address         
tcp4  0/0/5                            pg03.5666             
tcp4  0/0/5                            bacula-sd-02.5666     
tcp4  0/0/5                            pg02.5666             
tcp4  0/0/5                            mqtt01.5666            
tcp4  0/0/5                            r720-01.5666           
tcp4  0/0/5                            *.3551                 
tcp4  0/0/5                            r720-01.snmp           
unix  0/0/5                            /var/agentx/master
unix  0/0/5                            /var/agentx/master
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe

I think these services are the most likely candidates for further investigation. Do you agree?

Most of these services are nrpe listening on port 5666. Checking in /usr/local/etc/nrpe.cfg in the pg03 jail, I found:

Code: 
#listen_queue_size=5

I uncommented that line and doubled the queue size to 10 and restarted nrpe in that jail.

Rerunning the netstat -aL command on the host, we see the maxqlen has doubled.

Code: 
[dan@r720-01:~] $ netstat -aL | grep pg03
tcp4  0/0/10                           pg03.5666             
tcp4  0/0/244                          pg03.postgresql       
tcp4  0/0/100                          pg03.smtp             
tcp4  0/0/128                          pg03.ssh               
[dan@r720-01:~] $

I repeated the process for nrpe in all jails on this host.

Now we wait.

The original netstat -aL appears below.

Code: 
[dan@r720-01:~] $ netstat -aL
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address         
tcp4  0/0/244                          pg02.postgresql       
tcp4  0/0/244                          pg03.postgresql       
tcp4  0/0/5                            pg03.5666             
tcp4  0/0/100                          pg03.smtp             
tcp4  0/0/128                          pg03.ssh               
tcp4  0/0/128                          bacula-sd-02.ssh       
tcp4  0/0/50                           bacula-sd-02.bacula-sd
tcp4  0/0/512                          bacula-sd-02.https     
tcp4  0/0/512                          bacula-sd-02.http     
tcp4  0/0/5                            bacula-sd-02.5666     
tcp4  0/0/100                          bacula-sd-02.smtp     
tcp4  0/0/512                          mqtt01.https           
tcp4  0/0/512                          mqtt01.http           
tcp4  0/0/50                           r720-01.bacula-fd     
tcp4  0/0/5                            pg02.5666             
tcp4  0/0/100                          pg02.smtp             
tcp4  0/0/128                          pg02.ssh               
tcp4  0/0/128                          *.ssh                 
tcp4  0/0/128                          127.1.0.201.8083       
tcp4  0/0/5                            mqtt01.5666           
tcp4  0/0/100                          mqtt01.smtp           
tcp4  0/0/100                          mqtt01.8883           
tcp4  0/0/128                          r720-01.ssh           
tcp4  0/0/5                            r720-01.5666           
tcp4  0/0/5                            *.3551                 
tcp4  0/0/100                          r720-01.smtp           
tcp4  0/0/5                            r720-01.snmp           
unix  0/0/128                          /tmp/ssh-j2svqs0ys7/agent.36076
unix  0/0/244                          /tmp/.s.PGSQL.5432
unix  0/0/244                          /tmp/.s.PGSQL.5432
unix  0/0/100                          private/scache
unix  0/0/100                          private/anvil
unix  0/0/100                          private/lmtp
unix  0/0/100                          private/virtual
unix  0/0/100                          private/local
unix  0/0/100                          private/discard
unix  0/0/100                          private/retry
unix  0/0/100                          private/error
unix  0/0/100                          public/showq
unix  0/0/100                          private/relay
unix  0/0/100                          private/smtp
unix  0/0/100                          private/proxywrite
unix  0/0/100                          private/proxymap
unix  0/0/100                          public/flush
unix  0/0/100                          private/verify
unix  0/0/100                          private/trace
unix  0/0/100                          private/defer
unix  0/0/100                          private/bounce
unix  0/0/100                          private/rewrite
unix  0/0/100                          private/tlsmgr
unix  0/0/100                          public/qmgr
unix  0/0/100                          public/cleanup
unix  0/0/100                          public/pickup
unix  0/0/5                            /var/agentx/master
unix  0/0/128                          /tmp/tmux-1003/default
unix  0/0/512                          /var/run/php-fpm.sock
unix  0/0/100                          private/scache
unix  0/0/100                          private/anvil
unix  0/0/100                          private/lmtp
unix  0/0/100                          private/virtual
unix  0/0/100                          private/local
unix  0/0/100                          private/discard
unix  0/0/100                          private/retry
unix  0/0/100                          private/error
unix  0/0/100                          public/showq
unix  0/0/100                          private/relay
unix  0/0/100                          private/smtp
unix  0/0/100                          private/proxywrite
unix  0/0/100                          private/proxymap
unix  0/0/100                          public/flush
unix  0/0/100                          private/verify
unix  0/0/100                          private/trace
unix  0/0/100                          private/defer
unix  0/0/100                          private/bounce
unix  0/0/100                          private/rewrite
unix  0/0/100                          private/tlsmgr
unix  0/0/100                          public/qmgr
unix  0/0/100                          public/cleanup
unix  0/0/100                          public/pickup
unix  0/0/100                          private/scache
unix  0/0/100                          private/anvil
unix  0/0/100                          private/lmtp
unix  0/0/100                          private/virtual
unix  0/0/100                          private/local
unix  0/0/100                          private/discard
unix  0/0/100                          private/retry
unix  0/0/100                          private/error
unix  0/0/100                          public/showq
unix  0/0/100                          private/relay
unix  0/0/100                          private/smtp
unix  0/0/100                          private/proxywrite
unix  0/0/100                          private/proxymap
unix  0/0/100                          public/flush
unix  0/0/100                          private/verify
unix  0/0/100                          private/trace
unix  0/0/100                          private/defer
unix  0/0/100                          private/bounce
unix  0/0/100                          private/rewrite
unix  0/0/100                          private/tlsmgr
unix  0/0/100                          public/qmgr
unix  0/0/100                          public/cleanup
unix  0/0/100                          public/pickup
unix  0/0/5                            /var/agentx/master
unix  0/0/100                          private/scache
unix  0/0/100                          private/anvil
unix  0/0/100                          private/lmtp
unix  0/0/100                          private/virtual
unix  0/0/100                          private/local
unix  0/0/100                          private/discard
unix  0/0/100                          private/retry
unix  0/0/100                          private/error
unix  0/0/100                          public/showq
unix  0/0/100                          private/relay
unix  0/0/100                          private/smtp
unix  0/0/100                          private/proxywrite
unix  0/0/100                          private/proxymap
unix  0/0/100                          public/flush
unix  0/0/100                          private/verify
unix  0/0/100                          private/trace
unix  0/0/100                          private/defer
unix  0/0/100                          private/bounce
unix  0/0/100                          private/rewrite
unix  0/0/100                          private/tlsmgr
unix  0/0/100                          public/qmgr
unix  0/0/100                          public/cleanup
unix  0/0/100                          public/pickup
unix  0/0/512                          /var/run/php-fpm.sock
unix  0/0/512                          /var/run/supervisor/supervisor.sock.6980
unix  0/0/100                          private/scache
unix  0/0/100                          private/anvil
unix  0/0/100                          private/lmtp
unix  0/0/100                          private/virtual
unix  0/0/100                          private/local
unix  0/0/100                          private/discard
unix  0/0/100                          private/retry
unix  0/0/100                          private/error
unix  0/0/100                          public/showq
unix  0/0/100                          private/relay
unix  0/0/100                          private/smtp
unix  0/0/100                          private/proxywrite
unix  0/0/100                          private/proxymap
unix  0/0/100                          public/flush
unix  0/0/100                          private/verify
unix  0/0/100                          private/trace
unix  0/0/100                          private/defer
unix  0/0/100                          private/bounce
unix  0/0/100                          private/rewrite
unix  0/0/100                          private/tlsmgr
unix  0/0/100                          public/qmgr
unix  0/0/100                          public/cleanup
unix  0/0/100                          public/pickup
unix  0/0/100                          private/scache
unix  0/0/100                          private/anvil
unix  0/0/100                          private/lmtp
unix  0/0/100                          private/virtual
unix  0/0/100                          private/local
unix  0/0/100                          private/discard
unix  0/0/100                          private/retry
unix  0/0/100                          private/error
unix  0/0/100                          public/showq
unix  0/0/100                          private/relay
unix  0/0/100                          private/smtp
unix  0/0/100                          private/proxywrite
unix  0/0/100                          private/proxymap
unix  0/0/100                          public/flush
unix  0/0/100                          private/verify
unix  0/0/100                          private/trace
unix  0/0/100                          private/defer
unix  0/0/100                          private/bounce
unix  0/0/100                          private/rewrite
unix  0/0/100                          private/tlsmgr
unix  0/0/100                          public/qmgr
unix  0/0/100                          public/cleanup
unix  0/0/100                          public/pickup
unix  0/0/512                          /var/run/supervisor/supervisor.sock.4062
unix  0/0/100                          private/scache
unix  0/0/100                          private/anvil
unix  0/0/100                          private/lmtp
unix  0/0/100                          private/virtual
unix  0/0/100                          private/local
unix  0/0/100                          private/discard
unix  0/0/100                          private/retry
unix  0/0/100                          private/error
unix  0/0/100                          public/showq
unix  0/0/100                          private/relay
unix  0/0/100                          private/smtp
unix  0/0/100                          private/proxywrite
unix  0/0/100                          private/proxymap
unix  0/0/100                          public/flush
unix  0/0/100                          private/verify
unix  0/0/100                          private/trace
unix  0/0/100                          private/defer
unix  0/0/100                          private/bounce
unix  0/0/100                          private/rewrite
unix  0/0/100                          private/tlsmgr
unix  0/0/100                          public/qmgr
unix  0/0/100                          public/cleanup
unix  0/0/100                          public/pickup
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe
[dan@r720-01:~] $
 
You can increase the queue by setting kern.ipc.soacceptqueue, the default is fairly conservative so for a high load service you may need to increase it. But note that applications like Apache or nginx can set their own queue limits. It depends on the service itself.

That said, the reason the queue is overflowing is typically because the receiving service isn't handling incoming connections fast enough. Which can be caused due to all sorts of underlying reasons (slow disk access for example).

There isn't a clear cut "one solution" to fix this issue, a lot of it depends on the services themselves and the overall load on those services. And as I said, it can also be a symptom of an underlying problem.
 
Current settings

Code: 
[dan@r720-01:~] $ sysctl kern.ipc.soacceptqueue
kern.ipc.soacceptqueue: 512
 
Try setting it to 1024 and see if that improves things. But if there are underlying issues there's a high chance that queue will also overflow.

Code: 
     The kern.ipc.somaxconn sysctl limits the size of the listen queue for
     accepting new TCP connections.  The default value of 128 is typically too
     low for robust handling of new connections in a heavily loaded web server
     environment.  For such environments, we recommend increasing this value
     to 1024 or higher.  The service daemon may itself limit the listen queue
     size (e.g., sendmail(8), apache) but will often have a directive in its
     configuration file to adjust the queue size up.  Larger listen queues
     also do a better job of fending off denial of service attacks.
This is from tuning(7). It's a bit old, it still refers to somaxconn, you should use kern.ipc.soacceptqueue instead. For compatibility reasons kern.ipc.somaxconn still works but is basically an alias for kern.ipc.soacceptqueue.
 
27 hours after the post, no further Listen queue overflow seen on this host.
 
I've tried using the PCB addresses to find the exact process in the past ( netstat -Aan). But it's typically a process that's already long gone by the time I get to investigate the error. So it's more often than not just an educated guess which service has its queue flooded.
 
You must log in or register to reply here.
Back
Top