On a 9.2-RELEASE machine (amd64), I'm getting these messages in /var/log/messages a lot:
How to trace this to which process it could be? I thought that was
It's Nginx running on :80 and :443, not reporting any errors.
My /etc/sysct.conf:
This
is an experiment (not related) - problems occur before and after.
Some other data:
/var/run/dmesg.boot:
Code:
Dec 8 15:11:03 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec 8 15:11:03 srv02 last message repeated 345 times
Dec 8 15:33:12 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec 8 15:33:13 srv02 last message repeated 940 times
Dec 8 17:01:59 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec 8 17:02:00 srv02 last message repeated 987 times
Dec 8 17:02:48 srv02 last message repeated 1846 times
Dec 8 17:49:04 srv02 kernel: sonewconn: pcb 0xfffffe004df69ab8: Listen queue overflow: 1537 already in queue awaiting acceptance
Dec 8 17:49:04 srv02 last message repeated 626 timesHow to trace this to which process it could be? I thought that was
netstat -Lan, but nothing comes close to this 1537 number, and I see no Listen queue overflows at all:
Code:
[root@srv02 /var/log]# netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen Local Address
tcp4 0/0/128 *.199
tcp4 0/0/5 *.161
tcp4 0/0/2048 aaa.bbb.ccc.ddd.443
tcp4 0/0/2048 aaa.bbb.ccc.eee.443
tcp4 0/0/2048 *.80
tcp4 0/0/128 *.22
tcp6 0/0/128 *.22
tcp4 0/0/128 *.4949
tcp4 0/0/1024 127.0.0.1.9000
tcp4 0/0/128 *.10000
tcp4 0/0/10 *.587
tcp6 0/0/10 *.25
tcp4 0/0/10 *.25
tcp6 0/0/9 *.21
tcp4 0/0/9 *.21
Some tcp sockets may have been deleted.
unix 0/0/4 /var/run/devd.pipeIt's Nginx running on :80 and :443, not reporting any errors.
My /etc/sysct.conf:
Code:
#security.bsd.see_other_uids=0
net.inet.ip.fw.dyn_buckets=65536
net.inet.ip.fw.dyn_max=65536
net.inet.ip.fw.dyn_ack_lifetime=120
kern.ipc.maxsockets=51200
kern.ipc.somaxconn=2048
kern.ipc.nmbclusters=125600
kern.ipc.maxsockbuf=5000000
net.inet.tcp.finwait2_timeout=5000This
Code:
net.inet.tcp.finwait2_timeout=5000Some other data:
/var/run/dmesg.boot:
Code:
[root@srv02 /var/log]# cat /var/run/dmesg.boot
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013
root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz (2500.05-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x1067a Family = 0x6 Model = 0x17 Stepping = 10
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xc0ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,OSXSAVE>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 2147483648 (2048 MB)
avail memory = 2041835520 (1947 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <SMCI SMCISLP2> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci2: <ACPI PCI bus> on pcib2
3ware device driver for 9000 series storage controllers, version: 3.80.06.003
twa0: <3ware 9000 series Storage Controller> port 0x2000-0x20ff mem 0xf8000000-0xf9ffffff,0xfc100000-0xfc100fff irq 16 at device 0.0 on pci2
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-2LP, 2 ports, Firmware FE9X 4.10.00.027, BIOS BE9X 4.08.00.004
pcib3: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <Intel 82801I (ICH9) USB controller> port 0x1820-0x183f irq 16 at device 26.0 on pci0
usbus0 on uhci0
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfc600000-0xfc6003ff irq 18 at device 26.7 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x3000-0x301f mem 0xfc200000-0xfc21ffff irq 16 at device 0.0 on pci5
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:48:d4:19:9c
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x4000-0x401f mem 0xfc300000-0xfc31ffff irq 17 at device 0.0 on pci6
em1: Using an MSI interrupt
em1: Ethernet address: 00:30:48:d4:19:9d
uhci1: <Intel 82801I (ICH9) USB controller> port 0x1840-0x185f irq 16 at device 29.0 on pci0
usbus2 on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0x1860-0x187f irq 17 at device 29.1 on pci0
usbus3 on uhci2
uhci3: <Intel 82801I (ICH9) USB controller> port 0x1880-0x189f irq 18 at device 29.2 on pci0
usbus4 on uhci3
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfc600400-0xfc6007ff irq 16 at device 29.7 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
atapci0: <ITE IT8213F UDMA133 controller> port 0x5098-0x509f,0x5090-0x5093,0x5080-0x508f irq 17 at device 0.0 on pci7
ata2: <ATA channel> at channel 0 on atapci0
vgapci0: <VGA-compatible display> port 0x5000-0x507f mem 0xfa000000-0xfbffffff,0xfc000000-0xfc03ffff at device 1.0 on pci7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH9 SATA300 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x18d0-0x18df,0x18c0-0x18cf at device 31.2 on pci0
ata0: <ATA channel> at channel 0 on atapci1
ata1: <ATA channel> at channel 1 on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci2: <Intel ICH9 SATA300 controller> port 0x1c28-0x1c2f,0x1c1c-0x1c1f,0x1c20-0x1c27,0x1c18-0x1c1b,0x18f0-0x18ff,0x18e0-0x18ef irq 18 at device 31.5 on pci0
ata3: <ATA channel> at channel 0 on atapci2
ata4: <ATA channel> at channel 1 on atapci2
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est4 attach returned 6
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est5 attach returned 6
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est6 attach returned 6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 4719471906004719
device_attach: est7 attach returned 6
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
da0 at twa0 bus 0 scbus0 target 0 lun 0
da0: <AMCC 9650SE-2LP DISK 4.10> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 476827MB (976541696 512 byte sectors: 255H 63S/T 60786C)
SMP: AP CPU #7 Launched!
cd0 at ata0 bus 0 scbus2 target 0 lun 0
cd0: <TEAC DV-28E-V 1.AB> Removable CD-ROM SCSI-0 device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA2, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
SMP: AP CPU #6 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1250023687 Hz quality 1000
uhub5: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/da0p2 [rw]... netstat -m:
Code:
[root@srv02 /var/log]# netstat -m
1416/4479/5895 mbufs in use (current/cache/total)
1355/3575/4930/125600 mbuf clusters in use (current/cache/total/max)
1355/2741 mbuf+clusters out of packet secondary zone in use (current/cache)
9/1367/1376/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
3100K/13737K/16837K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
4 requests for I/O initiated by sendfile
0 calls to protocol drain routines vmstat -z
Code:
[root@srv02 /var/log]# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 189, 15, 189, 0, 0
UMA Zones: 1408, 0, 189, 1, 189, 0, 0
UMA Slabs: 568, 0, 20853, 2331, 433142, 0, 0
UMA RCntSlabs: 568, 0, 3841, 254, 118830, 0, 0
UMA Hash: 256, 0, 79, 11, 83, 0, 0
16 Bucket: 152, 0, 12, 138, 216, 0, 0
32 Bucket: 280, 0, 66, 116, 389, 0, 0
64 Bucket: 536, 0, 65, 75, 783, 119, 0
128 Bucket: 1048, 0, 1670, 4,3073770131,45038, 0
VM OBJECT: 232, 0, 48300, 55732,104122833, 0, 0
MAP: 232, 0, 8, 24, 8, 0, 0
KMAP ENTRY: 120, 87544, 97, 1856, 8615921, 0, 0
MAP ENTRY: 120, 0, 3942, 3498,289072021, 0, 0
fakepg: 120, 0, 0, 62, 4, 0, 0
mt_zone: 4112, 0, 328, 58, 328, 0, 0
16: 16, 0, 2262, 1602,8093258537, 0, 0
32: 32, 0, 2476, 3483,1616864618, 0, 0
64: 64, 0, 12027, 18605,5728177109, 0, 0
128: 128, 0, 10670, 10036,4780327938, 0, 0
256: 256, 0, 3054, 18531,69606267, 0, 0
512: 512, 0, 1927, 2154, 6768983, 0, 0
1024: 1024, 0, 66, 1382,21445529, 0, 0
2048: 2048, 0, 327, 1607,27358134, 0, 0
4096: 4096, 0, 466, 1393,4761624033, 0, 0
Files: 80, 0, 4137, 31413,13302476952, 0, 0
rl_entry: 40, 0, 1390, 374, 1390, 0, 0
TURNSTILE: 136, 0, 1690, 110, 1702, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1192, 0, 65, 1351, 4489629, 0, 0
THREAD: 1160, 0, 1489, 200, 2132, 0, 0
SLEEPQUEUE: 80, 0, 1690, 253, 1702, 0, 0
VMSPACE: 392, 0, 49, 1291, 4489611, 0, 0
cpuset: 72, 0, 91, 109, 91, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 1356, 2740,14447161684, 0, 0
mbuf: 256, 0, 63, 1736,44688318104, 0, 0
mbuf_cluster: 2048, 125600, 4096, 834, 219264, 0, 0
mbuf_jumbo_page: 4096, 12800, 9, 1367,181669080, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 1176,13300968, 0, 0
ttyinq: 160, 0, 180, 516, 1530, 0, 0
ttyoutq: 256, 0, 95, 400, 795, 0, 0
g_bio: 248, 0, 0, 2595,41021572, 0, 0
ata_request: 328, 0, 0, 888, 1101467, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0
FPU_save_area: 576, 0, 0, 0, 0, 0, 0
VNODE: 504, 0, 45738, 27926, 3765921, 0, 0
VNODEPOLL: 112, 0, 4, 227, 10, 0, 0
S VFS Cache: 108, 0, 49821, 33537, 3792607, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 389, 6775, 556848, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 1348,10028212387, 0, 0
NCLNODE: 568, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 25, 819, 61854, 0, 0
pipe: 728, 0, 21, 1274, 4376199, 0, 0
Mountpoints: 824, 0, 5, 7, 5, 0, 0
ksiginfo: 112, 0, 1409, 1759, 173224, 0, 0
itimer: 344, 0, 0, 33, 2, 0, 0
KNOTE: 128, 0, 4078, 10799,16653712635, 0, 0
socket: 680, 51204, 18756, 25686,4977766739, 0, 0
ipq: 56, 3969, 0, 882, 6542, 0, 0
udp_inpcb: 392, 51200, 4, 1796,19124476, 0, 0
udpcb: 16, 51240, 4, 1844,19124476, 0, 0
tcp_inpcb: 392, 51200, 23672, 27528,4958487647,33449, 0
tcpcb: 976, 51200, 13422, 30754,4958487647, 0, 0
tcptw: 72, 10250, 10250, 0,17968516,3073650374, 0
syncache: 152, 15375, 274, 1651,3377447939, 0, 0
hostcache: 136, 15372, 3819, 7269, 2424896, 0, 0
tcpreass: 40, 7896, 7, 2009, 3880470, 0, 0
sackhole: 32, 0, 3, 1714,20878750, 0, 0
sctp_ep: 1384, 25600, 0, 0, 0, 0, 0
sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 288, 8, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 104, 400032, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 51200, 0, 330, 22522, 0, 0
unpcb: 240, 51200, 30, 578, 98483, 0, 0
rtentry: 200, 0, 25, 70, 25, 0, 0
selfd: 56, 0, 2306, 1411,718114578, 0, 0
SWAPMETA: 288, 251459, 91, 169, 144321, 0, 0
FFS inode: 168, 0, 45707, 40511, 3765385, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 45707, 33568, 3765385, 0, 0
IPFW dynamic rule: 120, 65565, 16, 1503,12996201, 0, 0
taskq_zone: 48, 0, 0, 0, 0, 0, 0
space_seg_cache: 64, 0, 0, 0, 0, 0, 0
zio_cache: 944, 0, 0, 0, 0, 0, 0
zio_link_cache: 48, 0, 0, 0, 0, 0, 0
zio_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0
zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0
sa_cache: 80, 0, 0, 0, 0, 0, 0
dnode_t: 856, 0, 0, 0, 0, 0, 0
dmu_buf_impl_t: 224, 0, 0, 0, 0, 0, 0
arc_buf_hdr_t: 216, 0, 0, 0, 0, 0, 0
arc_buf_t: 72, 0, 0, 0, 0, 0, 0
zil_lwb_cache: 192, 0, 0, 0, 0, 0, 0
zfs_znode_cache: 368, 0, 0, 0, 0, 0, 0 vmstat -m:
Code:
[root@srv02 /var/log]# vmstat -m
Type InUse MemUse HighUse Requests Size(s)
acpidev 34 3K - 34 64
USBdev 24 3K - 24 64,128
isadev 6 1K - 6 128
USB 42 26K - 42 16,32,128,4096
cdev 7 2K - 7 256
entropy 1024 64K - 1024 64
sigio 9 1K - 10802 64
filedesc 141 439K - 4689529 16,32,64,128,256,512,1024,2048,4096
kdtrace 1556 377K - 4491768 64,256
kenv 76 11K - 82 16,32,64,128
kqueue 18 113K - 15836423 256,512,2048,4096
proc-args 41 3K - 3968447 16,32,64,128,256
hhook 2 1K - 2 256
kbdmux 6 18K - 6 16,512,1024,2048
ithread 96 16K - 96 32,128,256
LED 4 1K - 4 16,128
KTRACE 100 13K - 100 128
linker 244 983K - 439 16,32,64,128,256,512,1024,2048,4096
lockf 20 3K - 5684573744 64,128
loginclass 3 1K - 70295 64
ip6ndp 12 1K - 13 64,128
temp 52 4K - 4801923227 16,32,64,128,256,512,1024,2048,4096
devbuf 19602 35542K - 19664 16,32,64,128,256,512,1024,2048,4096
pci_link 16 2K - 16 32,64,128
module 489 62K - 489 128
mtx_pool 2 16K - 2
osd 3 1K - 5 16,32,64
pmchooks 1 1K - 1 128
subproc 1483 973K - 4491054 512,4096
proc 2 16K - 2
session 27 4K - 468194 128
pgrp 30 4K - 469565 128
cred 98 16K - 43342030 64,256
uidinfo 5 3K - 800607 128,2048
plimit 29 8K - 1145264 256
sysctltmp 0 0K - 2371637 16,32,64,128,4096
sysctloid 4942 246K - 5275 16,32,64,128
sysctl 0 0K - 3273436 16,32,64
tidhash 1 16K - 1
callout 7 3584K - 7
umtx 3378 423K - 3402 128
p1003.1b 1 1K - 1 16
SWAP 2 549K - 2 64
feeder 7 1K - 7 32
bus-sc 95 284K - 5016 16,32,128,256,512,1024,2048,4096
bus 1221 106K - 4624564 16,32,64,128,256,1024
devstat 8 17K - 8 32,4096
eventhandler 90 8K - 90 64,128
CAM periph 6 2K - 56 16,32,64,128,256
kobj 338 1352K - 658 4096
Per-cpu 1 1K - 1 32
DEVFS1 101 51K - 138 512
DEVFS3 121 31K - 192 256
rman 256 30K - 633 16,32,128
sbuf 1 1K - 3016550 16,32,64,128,256,512,1024,2048,4096
DEVFS 16 1K - 17 16,128
DEVFSP 0 0K - 5 64
stack 0 0K - 21366 256
taskqueue 25 2K - 25 16,32,64,128
Unitno 15 1K - 6638303 32,64
iov 0 0K - 4650603632 16,64,128,256,512,4096
select 1500 188K - 1500 128
ioctlops 0 0K - 1609191934 16,32,64,128,256,512,1024,2048
msg 4 30K - 4 2048,4096
sem 4 106K - 4 2048,4096
shm 1 20K - 1
tty 20 20K - 21416 1024,2048
pts 1 1K - 21 256
mbuf_tag 0 0K - 192 32,128
shmfd 1 8K - 1
pcb 261 164K - 1599382981 16,32,128,1024,2048,4096
soname 3 1K - 6492269335 16,32,128
acl 0 0K - 77185 4096
vfscache 1 1024K - 1
cl_savebuf 0 0K - 3753882 64
vfs_hash 1 512K - 1
vnodes 2 1K - 2 256
mount 61 3K - 131 16,32,64,128,256
vnodemarker 0 0K - 852777 512
fadvise 0 0K - 15 64
BPF 10 2K - 45 16,128,512,4096
ether_multi 40 3K - 46 16,32,64
ifaddr 97 25K - 117 32,64,128,256,512,4096
ifnet 11 21K - 11 128,2048
clone 6 24K - 6 4096
arpcom 3 1K - 3 16
lltable 32 13K - 754 256,512
routetbl 44 6K - 2215001 32,64,128,256,512
igmp 10 3K - 10 256
in_multi 2 1K - 2 256
sctp_iter 0 0K - 7 256
sctp_ifn 2 1K - 2 128
sctp_ifa 9 2K - 9 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 7 16
hostcache 1 28K - 1
syncache 1 96K - 1
in6_multi 25 3K - 25 32,256
mld 10 2K - 10 128
NFS FHA 1 2K - 1 2048
rpc 2 1K - 2 256
audit_evclass 180 6K - 219 32
jblocks 8 2K - 8 128,256
savedino 0 0K - 1143371 256
sbdep 0 0K - 218972 64
jsegdep 417 27K - 43235203 64
jseg 43 6K - 3237789 128
jfreefrag 0 0K - 9756313 128
jnewblk 8 1K - 32273077 128
jmvref 0 0K - 1345 128
jremref 0 0K - 431605 128
jaddref 0 0K - 774208 128
freedep 0 0K - 110538 64
freework 1 1K - 2683399 64,128
newdirblk 0 0K - 39950 64
dirrem 0 0K - 430013 128
mkdir 0 0K - 77792 128
diradd 0 0K - 696416 128
freefile 0 0K - 296163 64
freeblks 0 0K - 297949 256
freefrag 0 0K - 9756313 128
indirdep 4 1K - 340559 128
newblk 374 222K - 32273078 256
bmsafemap 3 9K - 772213 256
inodedep 11 517K - 1311989 512
pagedep 1 128K - 242262 256
ufs_dirhash 203 95K - 8608 16,32,64,128,256,512,1024,2048,4096
ufs_quota 1 512K - 1
ufs_mount 12 58K - 12 512,4096
vm_pgdata 2 257K - 2 128
UMAHash 4 204K - 26 512,1024,2048,4096
pfs_nodes 21 6K - 21 256
GEOM 88 16K - 108003 16,32,64,128,256,512,1024,2048
memdesc 1 4K - 1 4096
atkbddev 2 1K - 2 64
acpiintr 1 1K - 1 64
acpica 2041 202K - 20774773 16,32,64,128,256,512,1024,2048
acpitask 1 8K - 1
twa_commands 2 124K - 32020 16,32,64,128,256
CAM dev queue 7 1K - 7 128
raid_data 0 0K - 180 32,128,256
UART 6 4K - 6 16,512,1024
md_nvidia_data 0 0K - 29 512
ata_pci 2 1K - 2 64
apmdev 1 1K - 1 128
madt_table 0 0K - 1 4096
acpisem 20 3K - 20 128
md_sii_data 0 0K - 29 512
CAM path 11 1K - 145 32
CAM CCB 259 518K - 313 2048
io_apic 1 2K - 1 2048
scsi_cd 0 0K - 10 16
MCA 8 1K - 8 128
msi 2 1K - 2 128
nexusdev 3 1K - 3 16
CAM DEV 9 18K - 86 2048
CAM XPT 34 3K - 42943 16,32,64,128,256,1024,2048
scsi_da 0 0K - 145 16
CAM queue 26 9K - 497 16,32,64,128,256,512,4096
CAM SIM 7 2K - 7 256
IpFw/IpAcct 132 499K - 1179 16,32,64,128,256,512,1024,2048
solaris 104 4359K - 106236 16,32,64,128,1024
kstat_data 5 1K - 5 64