Hi there
I recently updated a FreeBSD VM (esxi 6.7) with MySQL and PF from 12 to 12.1. Since then, this server runs out of connectivity to the internet (vmx0) and the lan (vmx1) approximately 1 time a day, although it does not match heavy loads. When this happens, I have to restart the server, because if I do:
service netif restart && /etc/rc.d/routing restart && service local_unbound restart
The server has connectivity for a few seconds, and then it goes offline again, and I have to reboot. Before the update this never happened, this VM was months without restarting.
vmstat -m while the network is hanged, returned:
If I reload or disable PF while the network is hanged, it has no effect
I tried to remove and add again the 2 network interfaces, the issue continues.
Any ideas?
Thank you
I recently updated a FreeBSD VM (esxi 6.7) with MySQL and PF from 12 to 12.1. Since then, this server runs out of connectivity to the internet (vmx0) and the lan (vmx1) approximately 1 time a day, although it does not match heavy loads. When this happens, I have to restart the server, because if I do:
service netif restart && /etc/rc.d/routing restart && service local_unbound restart
The server has connectivity for a few seconds, and then it goes offline again, and I have to reboot. Before the update this never happened, this VM was months without restarting.
vmstat -m while the network is hanged, returned:
Code:
Type InUse MemUse HighUse Requests Size(s)
CAM DEV 39 78K - 132 2048
entropy 1 1K - 419 32,4096
CAM CCB 0 0K - 12832796 2048
CAM path 44 2K - 364 32
CAM periph 12 3K - 74 16,32,64,128,256
feeder 7 1K - 7 32
CAM I/O Scheduler 5 1K - 5 128
CAM queue 73 28K - 362 16,32,64,128,256,512,2048
CAM dev queue 34 3K - 34 64
scsi_da 0 0K - 375 32,64,256
ata_pci 1 1K - 1 64
vtbuf 24 1968K - 46 4096
vt 11 6K - 11 512
DEVFS3 114 29K - 156 256
DEVFS1 95 48K - 136 512
DEVFS_RULE 56 27K - 56 64,512
DEVFS 9 1K - 10 16,128
NFSD V4client 1 1K - 1 256
NFSD lckfile 1 1K - 1 256
NFSD session 1 1K - 1 1024
pfs_nodes 20 10K - 20 512
GEOM 201 37K - 2754 16,32,64,128,256,512,1024,2048,4096,8192,16384
raid_data 0 0K - 408 32,128,256
isadev 7 1K - 7 128
cdev 4 1K - 4 256
filedesc 25 513K - 1295 16,32,64,128,256,4096,8192,16384,32768,65536
sigio 0 0K - 1 64
filecaps 3 1K - 884 16,32,64
kdtrace 531 121K - 31652 64,256
kenv 101 12K - 105 16,32,64,128,8192
kqueue 472 1270K - 2891257 64,256,512,2048,4096,8192,16384
proc-args 80 4K - 6134 16,32,64,128,256
hhook 13 4K - 13 256
ithread 176 27K - 176 32,128,256
prison 4 1K - 4 32
KTRACE 100 13K - 100 128
linker 238 200K - 269 16,32,64,128,256,512,1024,2048,4096,8192,16384
lockf 3683 450K - 86103 64,128
loginclass 3 1K - 5 64
devbuf 18061 34293K - 18349 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
temp 26 18K - 1086806 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
evdev 4 4K - 4 1024
module 508 64K - 509 128
mtx_pool 2 72K - 2 8192,65536
osd 3 1K - 9 16,32,64,128,256
pmchooks 1 1K - 1 128
pmc 1 1K - 1 64
pgrp 19 3K - 676 128
session 17 3K - 588 128
proc 2 64K - 2 32768
subproc 191 320K - 9138 512,4096
cred 53 14K - 6019 256
acpiintr 1 1K - 1 64
plimit 23 6K - 8879 256
uidinfo 10 10K - 387 128,8192
acpica 8884 819K - 143654 16,32,64,128,256,512,1024,2048
sysctl 0 0K - 3 64
sysctloid 5787 292K - 5902 16,32,64,128
sysctltmp 0 0K - 2863 16,64,256,1024
acpitask 1 64K - 1 65536
tidhash 1 64K - 1 65536
callout 7 2696K - 7
umtx 1136 142K - 1136 128
p1003.1b 1 1K - 1 16
SWAP 1 784K - 1
bus 1370 124K - 13206 16,32,64,128,256,512,1024,4096
bus-sc 105 377K - 7039 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
acpisem 31 4K - 31 128
devstat 16 33K - 16 32,4096
epoch 4 1K - 4 128
eventhandler 135 12K - 135 64,128
gtaskqueue 46 37K - 46 16,32,256,8192
kobj 344 1376K - 986 4096
Per-cpu 1 1K - 1 32
kbdmux 6 22K - 6 16,512,1024,2048,16384
rman 719 77K - 987 32,128
sbuf 1 1K - 1799 16,32,64,128,256,4096,32768
toponodes 30 4K - 30 128
taskqueue 30 4K - 30 16,32,64,256
terminal 11 3K - 11 256
Unitno 22 2K - 24510 32,64
vmem 3 648K - 6 4096,8192,65536
ioctlops 0 0K - 4094 256,512,1024,2048,4096
select 71 9K - 71 128
iov 0 0K - 3296161632 16,64,128,256,512
msg 4 30K - 4 2048,4096,8192,16384
sem 4 106K - 4 2048,4096
shm 1 32K - 1 32768
tty 12 12K - 14 1024
pts 0 0K - 2 256
mbuf_tag 0 0K - 577 32
shmfd 1 8K - 1 8192
soname 11 1K - 2914333 16,32,128
pcb 29 2201K - 8666 16,32,64,128,1024,2048,8192
vfscache 4 4145K - 4 256,16384,32768
cl_savebuf 0 0K - 10263 64
vfs_hash 1 2048K - 1
vnodes 1 1K - 1 256
mount 121 6K - 191 16,32,64,128,256
statfs 0 0K - 121439 4096
vnodemarker 0 0K - 44210 512
chacha20random 1 2K - 1 2048
BPF 3 1K - 3 128
ifnet 4 7K - 4 128,2048
ifaddr 71 28K - 77 16,32,64,128,256,512,2048,4096
ether_multi 22 2K - 176 16,32,64,128
clone 7 1K - 7 128
ipsec 3 1K - 3 256
lltable 21 8K - 89 256,512
iflib 43 139K - 51 16,32,64,128,1024,2048,4096,32768
routetbl 48 10K - 249 32,64,128,256,512
vnet 1 1K - 1 64
vnet_data 1 240K - 1
vnet_data_free 1 1K - 1 32
igmp 3 1K - 3 128
in_multi 3 1K - 24 256
ip_moptions 2 1K - 2 64
encap_export_host 12 1K - 12 32,64
sctp_a_it 0 0K - 52 16
sctp_vrf 1 1K - 8 64
sctp_ifa 5 1K - 40 128
sctp_ifn 3 1K - 24 128
sctp_iter 0 0K - 52 256
tfo_ccache 1 128K - 1
hostcache 1 32K - 1 32768
LRO 4 20K - 4 2048,8192
tcpfunc 1 1K - 1 64
syncache 1 68K - 1
in6_multi 15 2K - 120 32,256
ip6_moptions 1 1K - 1 32
ip6opt 4 1K - 11 256
mld 3 1K - 3 128
ip6ndp 4 1K - 4 64,256
inpcbpolicy 243 8K - 972868 32
secasvar 1 1K - 1 1024
sahead 1 1K - 1 1024
ipsecpolicy 2 2K - 2 256,1024
ipsec-saq 2 2K - 2 1024
crypto 2 2K - 2 1024
rpc 2 8K - 2 4096
audit_evclass 230 8K - 285 32
pagedep 9 2049K - 1466 256
inodedep 11 16386K - 42401 512
bmsafemap 9 65K - 12336 256,8192
newblk 8 32768K - 387097 256
indirdep 0 0K - 3498 128,32768
freefrag 0 0K - 24111 128
freeblks 0 0K - 6998 256
freefile 0 0K - 7004 64
diradd 1 1K - 8187 128
mkdir 0 0K - 12 128
dirrem 0 0K - 8075 128
newdirblk 0 0K - 6 64
freework 8 1K - 31314 64,128
freedep 0 0K - 1 64
jaddref 1 1K - 1410 128
jremref 0 0K - 1324 128
jmvref 0 0K - 17 128
jnewblk 0 0K - 637 128
jfreefrag 0 0K - 12 128
jseg 0 0K - 596 128
jsegdep 1 1K - 3383 64
sbdep 0 0K - 5026 64
savedino 0 0K - 767 256
jblocks 10 2K - 10 128,256
softdep 8 4K - 8 512
ufs_dirhash 393 191K - 3279 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
ufs_quota 1 2048K - 1
ufs_mount 24 132K - 32 512,4096,8192
vm_pgdata 1 1K - 1 128
UMAHash 6 64K - 17 512,1024,2048,4096,8192,16384,32768
memdesc 1 4K - 1 4096
pci_link 8 1K - 8 64,128
atkbddev 2 1K - 2 64
acpidev 73 5K - 73 64
CAM SIM 34 9K - 34 256
CAM XPT 107 12K - 331 16,32,64,128,256,512,1024,2048,65536
apmdev 1 1K - 1 128
madt_table 0 0K - 2 128,4096
intr 4 400K - 4 65536
io_apic 1 2K - 1 2048
local_apic 1 16K - 1 16384
MCA 15 2K - 15 64,128
cpus 2 1K - 2 64
msi 36 5K - 36 128
nexusdev 5 1K - 5 16
pf_hash 5 11524K - 5 2048
pf_ifnet 6 4K - 88 256,2048
pf_osfp 1191 123K - 3573 64,128
pf_rule 31 31K - 93 1024
pf_table 3 6K - 12 2048
Code:
pfctl -sm
states hard limit 15000000
src-nodes hard limit 300000
frags hard limit 400000
table-entries hard limit 18000000If I reload or disable PF while the network is hanged, it has no effect
I tried to remove and add again the 2 network interfaces, the issue continues.
Any ideas?
Thank you