Listen queue overflow

On a 9.2-RELEASE machine (amd64), I'm getting these messages in /var/log/messages a lot:

Code:
Dec  8 15:11:03 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec  8 15:11:03 srv02 last message repeated 345 times
Dec  8 15:33:12 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec  8 15:33:13 srv02 last message repeated 940 times
Dec  8 17:01:59 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec  8 17:02:00 srv02 last message repeated 987 times
Dec  8 17:02:48 srv02 last message repeated 1846 times
Dec  8 17:49:04 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec  8 17:49:04 srv02 last message repeated 626 times

How to trace this to which process it could be? I thought that was netstat -Lan, but nothing comes close to this 1537 number, and I see no Listen queue overflows at all:

Code:
[root@srv02 /var/log]# netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/128        *.199
tcp4  0/0/5          *.161
tcp4  0/0/2048       aaa.bbb.ccc.ddd.443
tcp4  0/0/2048       aaa.bbb.ccc.eee.443
tcp4  0/0/2048       *.80
tcp4  0/0/128        *.22
tcp6  0/0/128        *.22
tcp4  0/0/128        *.4949
tcp4  0/0/1024       127.0.0.1.9000
tcp4  0/0/128        *.10000
tcp4  0/0/10         *.587
tcp6  0/0/10         *.25
tcp4  0/0/10         *.25
tcp6  0/0/9          *.21
tcp4  0/0/9          *.21
Some tcp sockets may have been deleted.
unix  0/0/4          /var/run/devd.pipe

It's Nginx running on :80 and :443, not reporting any errors.

My /etc/sysct.conf:

Code:
#security.bsd.see_other_uids=0
net.inet.ip.fw.dyn_buckets=65536
net.inet.ip.fw.dyn_max=65536
net.inet.ip.fw.dyn_ack_lifetime=120
kern.ipc.maxsockets=51200
kern.ipc.somaxconn=2048
kern.ipc.nmbclusters=125600
kern.ipc.maxsockbuf=5000000
net.inet.tcp.finwait2_timeout=5000

This
Code:
net.inet.tcp.finwait2_timeout=5000
is an experiment (not related) - problems occur before and after.

Some other data:

/var/run/dmesg.boot:

Code:
[root@srv02 /var/log]# cat /var/run/dmesg.boot
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013
    root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU           L5420  @ 2.50GHz (2500.05-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x1067a  Family = 0x6  Model = 0x17  Stepping = 10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xc0ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,OSXSAVE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 2041835520 (1947 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <SMCI SMCISLP2> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci2: <ACPI PCI bus> on pcib2
3ware device driver for 9000 series storage controllers, version: 3.80.06.003
twa0: <3ware 9000 series Storage Controller> port 0x2000-0x20ff mem 0xf8000000-0xf9ffffff,0xfc100000-0xfc100fff irq 16 at device 0.0 on pci2
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-2LP, 2 ports, Firmware FE9X 4.10.00.027, BIOS BE9X 4.08.00.004
pcib3: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <Intel 82801I (ICH9) USB controller> port 0x1820-0x183f irq 16 at device 26.0 on pci0
usbus0 on uhci0
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfc600000-0xfc6003ff irq 18 at device 26.7 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x3000-0x301f mem 0xfc200000-0xfc21ffff irq 16 at device 0.0 on pci5
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:48:d4:19:9c
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x4000-0x401f mem 0xfc300000-0xfc31ffff irq 17 at device 0.0 on pci6
em1: Using an MSI interrupt
em1: Ethernet address: 00:30:48:d4:19:9d
uhci1: <Intel 82801I (ICH9) USB controller> port 0x1840-0x185f irq 16 at device 29.0 on pci0
usbus2 on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0x1860-0x187f irq 17 at device 29.1 on pci0
usbus3 on uhci2
uhci3: <Intel 82801I (ICH9) USB controller> port 0x1880-0x189f irq 18 at device 29.2 on pci0
usbus4 on uhci3
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfc600400-0xfc6007ff irq 16 at device 29.7 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
atapci0: <ITE IT8213F UDMA133 controller> port 0x5098-0x509f,0x5090-0x5093,0x5080-0x508f irq 17 at device 0.0 on pci7
ata2: <ATA channel> at channel 0 on atapci0
vgapci0: <VGA-compatible display> port 0x5000-0x507f mem 0xfa000000-0xfbffffff,0xfc000000-0xfc03ffff at device 1.0 on pci7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH9 SATA300 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x18d0-0x18df,0x18c0-0x18cf at device 31.2 on pci0
ata0: <ATA channel> at channel 0 on atapci1
ata1: <ATA channel> at channel 1 on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci2: <Intel ICH9 SATA300 controller> port 0x1c28-0x1c2f,0x1c1c-0x1c1f,0x1c20-0x1c27,0x1c18-0x1c1b,0x18f0-0x18ff,0x18e0-0x18ef irq 18 at device 31.5 on pci0
ata3: <ATA channel> at channel 0 on atapci2
ata4: <ATA channel> at channel 1 on atapci2
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est4 attach returned 6
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est5 attach returned 6
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est6 attach returned 6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est7 attach returned 6
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
da0 at twa0 bus 0 scbus0 target 0 lun 0
da0: <AMCC 9650SE-2LP DISK 4.10> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
SMP: AP CPU #7 Launched!
cd0 at ata0 bus 0 scbus2 target 0 lun 0
cd0: <TEAC DV-28E-V 1.AB> Removable CD-ROM SCSI-0 device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA2, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
SMP: AP CPU #6 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1250023687 Hz quality 1000
uhub5: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/da0p2 [rw]...


netstat -m:

Code:
[root@srv02 /var/log]# netstat -m
1416/4479/5895 mbufs in use (current/cache/total)
1355/3575/4930/125600 mbuf clusters in use (current/cache/total/max)
1355/2741 mbuf+clusters out of packet secondary zone in use (current/cache)
9/1367/1376/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
3100K/13737K/16837K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
4 requests for I/O initiated by sendfile
0 calls to protocol drain routines

vmstat -z

Code:
[root@srv02 /var/log]# vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     189,      15,     189,   0,   0
UMA Zones:             1408,      0,     189,       1,     189,   0,   0
UMA Slabs:              568,      0,   20853,    2331,  433142,   0,   0
UMA RCntSlabs:          568,      0,    3841,     254,  118830,   0,   0
UMA Hash:               256,      0,      79,      11,      83,   0,   0
16 Bucket:              152,      0,      12,     138,     216,   0,   0
32 Bucket:              280,      0,      66,     116,     389,   0,   0
64 Bucket:              536,      0,      65,      75,     783, 119,   0
128 Bucket:            1048,      0,    1670,       4,3073770131,45038,   0
VM OBJECT:              232,      0,   48300,   55732,104122833,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120,  87544,      97,    1856, 8615921,   0,   0
MAP ENTRY:              120,      0,    3942,    3498,289072021,   0,   0
fakepg:                 120,      0,       0,      62,       4,   0,   0
mt_zone:               4112,      0,     328,      58,     328,   0,   0
16:                      16,      0,    2262,    1602,8093258537,   0,   0
32:                      32,      0,    2476,    3483,1616864618,   0,   0
64:                      64,      0,   12027,   18605,5728177109,   0,   0
128:                    128,      0,   10670,   10036,4780327938,   0,   0
256:                    256,      0,    3054,   18531,69606267,   0,   0
512:                    512,      0,    1927,    2154, 6768983,   0,   0
1024:                  1024,      0,      66,    1382,21445529,   0,   0
2048:                  2048,      0,     327,    1607,27358134,   0,   0
4096:                  4096,      0,     466,    1393,4761624033,   0,   0
Files:                   80,      0,    4137,   31413,13302476952,   0,   0
rl_entry:                40,      0,    1390,     374,    1390,   0,   0
TURNSTILE:              136,      0,    1690,     110,    1702,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,      65,    1351, 4489629,   0,   0
THREAD:                1160,      0,    1489,     200,    2132,   0,   0
SLEEPQUEUE:              80,      0,    1690,     253,    1702,   0,   0
VMSPACE:                392,      0,      49,    1291, 4489611,   0,   0
cpuset:                  72,      0,      91,     109,      91,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,    1356,    2740,14447161684,   0,   0
mbuf:                   256,      0,      63,    1736,44688318104,   0,   0
mbuf_cluster:          2048, 125600,    4096,     834,  219264,   0,   0
mbuf_jumbo_page:       4096,  12800,       9,    1367,181669080,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,    1176,13300968,   0,   0
ttyinq:                 160,      0,     180,     516,    1530,   0,   0
ttyoutq:                256,      0,      95,     400,     795,   0,   0
g_bio:                  248,      0,       0,    2595,41021572,   0,   0
ata_request:            328,      0,       0,     888, 1101467,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
FPU_save_area:          576,      0,       0,       0,       0,   0,   0
VNODE:                  504,      0,   45738,   27926, 3765921,   0,   0
VNODEPOLL:              112,      0,       4,     227,      10,   0,   0
S VFS Cache:            108,      0,   49821,   33537, 3792607,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,     389,    6775,  556848,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,    1348,10028212387,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,      25,     819,   61854,   0,   0
pipe:                   728,      0,      21,    1274, 4376199,   0,   0
Mountpoints:            824,      0,       5,       7,       5,   0,   0
ksiginfo:               112,      0,    1409,    1759,  173224,   0,   0
itimer:                 344,      0,       0,      33,       2,   0,   0
KNOTE:                  128,      0,    4078,   10799,16653712635,   0,   0
socket:                 680,  51204,   18756,   25686,4977766739,   0,   0
ipq:                     56,   3969,       0,     882,    6542,   0,   0
udp_inpcb:              392,  51200,       4,    1796,19124476,   0,   0
udpcb:                   16,  51240,       4,    1844,19124476,   0,   0
tcp_inpcb:              392,  51200,   23672,   27528,4958487647,33449,   0
tcpcb:                  976,  51200,   13422,   30754,4958487647,   0,   0
tcptw:                   72,  10250,   10250,       0,17968516,3073650374,   0
syncache:               152,  15375,     274,    1651,3377447939,   0,   0
hostcache:              136,  15372,    3819,    7269, 2424896,   0,   0
tcpreass:                40,   7896,       7,    2009, 3880470,   0,   0
sackhole:                32,      0,       3,    1714,20878750,   0,   0
sctp_ep:               1384,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     288,       8,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400032,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  51200,       0,     330,   22522,   0,   0
unpcb:                  240,  51200,      30,     578,   98483,   0,   0
rtentry:                200,      0,      25,      70,      25,   0,   0
selfd:                   56,      0,    2306,    1411,718114578,   0,   0
SWAPMETA:               288, 251459,      91,     169,  144321,   0,   0
FFS inode:              168,      0,   45707,   40511, 3765385,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,   45707,   33568, 3765385,   0,   0
IPFW dynamic rule:      120,  65565,      16,    1503,12996201,   0,   0
taskq_zone:              48,      0,       0,       0,       0,   0,   0
space_seg_cache:         64,      0,       0,       0,       0,   0,   0
zio_cache:              944,      0,       0,       0,       0,   0,   0
zio_link_cache:          48,      0,       0,       0,       0,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,       0,       0,       0,   0,   0
dnode_t:                856,      0,       0,       0,       0,   0,   0
dmu_buf_impl_t:         224,      0,       0,       0,       0,   0,   0
arc_buf_hdr_t:          216,      0,       0,       0,       0,   0,   0
arc_buf_t:               72,      0,       0,       0,       0,   0,   0
zil_lwb_cache:          192,      0,       0,       0,       0,   0,   0
zfs_znode_cache:        368,      0,       0,       0,       0,   0,   0


vmstat -m:

Code:
[root@srv02 /var/log]# vmstat -m
         Type InUse MemUse HighUse Requests  Size(s)
      acpidev    34     3K       -       34  64
       USBdev    24     3K       -       24  64,128
       isadev     6     1K       -        6  128
          USB    42    26K       -       42  16,32,128,4096
         cdev     7     2K       -        7  256
      entropy  1024    64K       -     1024  64
        sigio     9     1K       -    10802  64
     filedesc   141   439K       -  4689529  16,32,64,128,256,512,1024,2048,4096
      kdtrace  1556   377K       -  4491768  64,256
         kenv    76    11K       -       82  16,32,64,128
       kqueue    18   113K       - 15836423  256,512,2048,4096
    proc-args    41     3K       -  3968447  16,32,64,128,256
        hhook     2     1K       -        2  256
       kbdmux     6    18K       -        6  16,512,1024,2048
      ithread    96    16K       -       96  32,128,256
          LED     4     1K       -        4  16,128
       KTRACE   100    13K       -      100  128
       linker   244   983K       -      439  16,32,64,128,256,512,1024,2048,4096
        lockf    20     3K       - 5684573744  64,128
   loginclass     3     1K       -    70295  64
       ip6ndp    12     1K       -       13  64,128
         temp    52     4K       - 4801923227  16,32,64,128,256,512,1024,2048,4096
       devbuf 19602 35542K       -    19664  16,32,64,128,256,512,1024,2048,4096
     pci_link    16     2K       -       16  32,64,128
       module   489    62K       -      489  128
     mtx_pool     2    16K       -        2
          osd     3     1K       -        5  16,32,64
     pmchooks     1     1K       -        1  128
      subproc  1483   973K       -  4491054  512,4096
         proc     2    16K       -        2
      session    27     4K       -   468194  128
         pgrp    30     4K       -   469565  128
         cred    98    16K       - 43342030  64,256
      uidinfo     5     3K       -   800607  128,2048
       plimit    29     8K       -  1145264  256
    sysctltmp     0     0K       -  2371637  16,32,64,128,4096
    sysctloid  4942   246K       -     5275  16,32,64,128
       sysctl     0     0K       -  3273436  16,32,64
      tidhash     1    16K       -        1
      callout     7  3584K       -        7
         umtx  3378   423K       -     3402  128
     p1003.1b     1     1K       -        1  16
         SWAP     2   549K       -        2  64
       feeder     7     1K       -        7  32
       bus-sc    95   284K       -     5016  16,32,128,256,512,1024,2048,4096
          bus  1221   106K       -  4624564  16,32,64,128,256,1024
      devstat     8    17K       -        8  32,4096
 eventhandler    90     8K       -       90  64,128
   CAM periph     6     2K       -       56  16,32,64,128,256
         kobj   338  1352K       -      658  4096
      Per-cpu     1     1K       -        1  32
       DEVFS1   101    51K       -      138  512
       DEVFS3   121    31K       -      192  256
         rman   256    30K       -      633  16,32,128
         sbuf     1     1K       -  3016550  16,32,64,128,256,512,1024,2048,4096
        DEVFS    16     1K       -       17  16,128
       DEVFSP     0     0K       -        5  64
        stack     0     0K       -    21366  256
    taskqueue    25     2K       -       25  16,32,64,128
       Unitno    15     1K       -  6638303  32,64
          iov     0     0K       - 4650603632  16,64,128,256,512,4096
       select  1500   188K       -     1500  128
     ioctlops     0     0K       - 1609191934  16,32,64,128,256,512,1024,2048
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1
          tty    20    20K       -    21416  1024,2048
          pts     1     1K       -       21  256
     mbuf_tag     0     0K       -      192  32,128
        shmfd     1     8K       -        1
          pcb   261   164K       - 1599382981  16,32,128,1024,2048,4096
       soname     3     1K       - 6492269335  16,32,128
          acl     0     0K       -    77185  4096
     vfscache     1  1024K       -        1
   cl_savebuf     0     0K       -  3753882  64
     vfs_hash     1   512K       -        1
       vnodes     2     1K       -        2  256
        mount    61     3K       -      131  16,32,64,128,256
  vnodemarker     0     0K       -   852777  512
      fadvise     0     0K       -       15  64
          BPF    10     2K       -       45  16,128,512,4096
  ether_multi    40     3K       -       46  16,32,64
       ifaddr    97    25K       -      117  32,64,128,256,512,4096
        ifnet    11    21K       -       11  128,2048
        clone     6    24K       -        6  4096
       arpcom     3     1K       -        3  16
      lltable    32    13K       -      754  256,512
     routetbl    44     6K       -  2215001  32,64,128,256,512
         igmp    10     3K       -       10  256
     in_multi     2     1K       -        2  256
    sctp_iter     0     0K       -        7  256
     sctp_ifn     2     1K       -        2  128
     sctp_ifa     9     2K       -        9  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -        7  16
    hostcache     1    28K       -        1
     syncache     1    96K       -        1
    in6_multi    25     3K       -       25  32,256
          mld    10     2K       -       10  128
      NFS FHA     1     2K       -        1  2048
          rpc     2     1K       -        2  256
audit_evclass   180     6K       -      219  32
      jblocks     8     2K       -        8  128,256
     savedino     0     0K       -  1143371  256
        sbdep     0     0K       -   218972  64
      jsegdep   417    27K       - 43235203  64
         jseg    43     6K       -  3237789  128
    jfreefrag     0     0K       -  9756313  128
      jnewblk     8     1K       - 32273077  128
       jmvref     0     0K       -     1345  128
      jremref     0     0K       -   431605  128
      jaddref     0     0K       -   774208  128
      freedep     0     0K       -   110538  64
     freework     1     1K       -  2683399  64,128
    newdirblk     0     0K       -    39950  64
       dirrem     0     0K       -   430013  128
        mkdir     0     0K       -    77792  128
       diradd     0     0K       -   696416  128
     freefile     0     0K       -   296163  64
     freeblks     0     0K       -   297949  256
     freefrag     0     0K       -  9756313  128
     indirdep     4     1K       -   340559  128
       newblk   374   222K       - 32273078  256
    bmsafemap     3     9K       -   772213  256
     inodedep    11   517K       -  1311989  512
      pagedep     1   128K       -   242262  256
  ufs_dirhash   203    95K       -     8608  16,32,64,128,256,512,1024,2048,4096
    ufs_quota     1   512K       -        1
    ufs_mount    12    58K       -       12  512,4096
    vm_pgdata     2   257K       -        2  128
      UMAHash     4   204K       -       26  512,1024,2048,4096
    pfs_nodes    21     6K       -       21  256
         GEOM    88    16K       -   108003  16,32,64,128,256,512,1024,2048
      memdesc     1     4K       -        1  4096
     atkbddev     2     1K       -        2  64
     acpiintr     1     1K       -        1  64
       acpica  2041   202K       - 20774773  16,32,64,128,256,512,1024,2048
     acpitask     1     8K       -        1
 twa_commands     2   124K       -    32020  16,32,64,128,256
CAM dev queue     7     1K       -        7  128
    raid_data     0     0K       -      180  32,128,256
         UART     6     4K       -        6  16,512,1024
md_nvidia_data     0     0K       -       29  512
      ata_pci     2     1K       -        2  64
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
      acpisem    20     3K       -       20  128
  md_sii_data     0     0K       -       29  512
     CAM path    11     1K       -      145  32
      CAM CCB   259   518K       -      313  2048
      io_apic     1     2K       -        1  2048
      scsi_cd     0     0K       -       10  16
          MCA     8     1K       -        8  128
          msi     2     1K       -        2  128
     nexusdev     3     1K       -        3  16
      CAM DEV     9    18K       -       86  2048
      CAM XPT    34     3K       -    42943  16,32,64,128,256,1024,2048
      scsi_da     0     0K       -      145  16
    CAM queue    26     9K       -      497  16,32,64,128,256,512,4096
      CAM SIM     7     2K       -        7  256
  IpFw/IpAcct   132   499K       -     1179  16,32,64,128,256,512,1024,2048
      solaris   104  4359K       -   106236  16,32,64,128,1024
   kstat_data     5     1K       -        5  64
 
Ideally queues shouldn't be used at all. When an application receives a connection it will start handling it. Only when it cannot handle the connection is the OS going to queue. It's more or less a small safety net. So it's possible nginx only temporarily gets a lot of connections and starts queueing. Eventually this queue runs out if the system isn't capable of handling all of them. But it's possible by the time you're looking all the queues are empty again.
 
Ok, it appeared to be not nginx, but php-fpm! After increasing kern.ipc.somaxconn I didn't restart this daemon, so it didn't show up with up to date values in netstat -Lan. Now it does, and my problems seem to be gone.

It would be nice though to have some sort of hint in the error message which PID or process name was causing the warning.
 
Sorry to resurrect an old thread, but this is the first hit for listen queue overflow errors, and I wanted to throw out a quick way you can pair up the error in dmesg with a process/bind.

lsof(8) lists the pcb, so you can just grep for this in the listening process output:

Code:
# lsof -iTCP -sTCP:LISTEN -P | grep 0xfffff802800f7810
named       887   bind   20u  IPv4 0xfffff802800f7810      0t0  TCP localhost:53 (LISTEN)
# netstat -Lan | grep '\.53'
tcp4  0/0/10         127.0.0.1.53

Also for future reference, the errors you saw about the overflow happen when the queue is at about 150% capacity, so 1527 cap means the queue size you'll see in netstat(1) is around 1000. I see port 9000 in your example is the only one at 1024, and it appears this is the php-fpm port.
 
Last edited by a moderator:
Back
Top