Hi, a FreeBSD server I manage has got network problem twice now, with the following error in /var/log/messages:
kernel: [zone: pf states] PF states limit reached
kernel: sonewconn: pcb 0xfffff80100947d20: Listen queue overflow: 193 already in queue awaiting acceptance (358 occurrences)
After the first incident, I followed some online suggestion against this error to add the following to /etc/pf.conf:
set limit { states 1000000, frags 1000000, src-nodes 100000, table-entries 1000000 }
However, after some days the same error happened again, because now "pfctl -ss" is over 1000000 lines.
I followed more online suggestions, so that now I have:
# sysctl -a |grep keep
vfs.nfs.nfs_keep_dirty_on_error: 0
net.inet.tcp.keepidle: 600000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.keepinit: 75000
net.inet.tcp.keepcnt: 4
net.inet.tcp.always_keepalive: 1
still I can see "pfctl -ss" keeps increasing by about 70,000 lines per day, so I'm still doomed.
Another server running the same version of FreeBSD does not have this problem, in fact its "pfctl -ss" is only hundreds of line in total and only fluctuates there.
What could be the problem?
I know upgrading OS may be a good try but I can't do it right now and especially the other server of the exact OS has no such problem.
The following are some relevant files. Thanks!
# uname -a
FreeBSD 11.1-RELEASE-p15 FreeBSD 11.1-RELEASE-p15 #0: Thu Sep 27 06:05:25 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
# cat /boot/loader.conf |egrep -v ^#
accf_http_load="YES"
accf_data_load="YES"
kern.ipc.shmseg="1024"
kern.ipc.soacceptqueue=1024
# cat /etc/pf.conf
ext_if="em0"
int_if="lo0"
set limit { states 1000000, frags 1000000, src-nodes 100000, table-entries 1000000 }
scrub in on $ext_if all fragment reassemble
block return log all
pass on $int_if all keep state
pass out on $ext_if proto udp all keep state
pass out on $ext_if proto tcp all modulate state
pass in on $ext_if proto tcp from any to $ext_if port 443 flags S/SA keep state
pass in on $ext_if proto tcp from 1.2.3.4 to $ext_if port 3306 flags S/SA keep state
table <ssh> { 10.0.0.1 } persist
pass in on $ext_if proto tcp from <ssh> to $ext_if port 22
table <blacklist> persist
block in on $ext_if proto tcp from <blacklist> to any
kernel: [zone: pf states] PF states limit reached
kernel: sonewconn: pcb 0xfffff80100947d20: Listen queue overflow: 193 already in queue awaiting acceptance (358 occurrences)
After the first incident, I followed some online suggestion against this error to add the following to /etc/pf.conf:
set limit { states 1000000, frags 1000000, src-nodes 100000, table-entries 1000000 }
However, after some days the same error happened again, because now "pfctl -ss" is over 1000000 lines.
I followed more online suggestions, so that now I have:
# sysctl -a |grep keep
vfs.nfs.nfs_keep_dirty_on_error: 0
net.inet.tcp.keepidle: 600000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.keepinit: 75000
net.inet.tcp.keepcnt: 4
net.inet.tcp.always_keepalive: 1
still I can see "pfctl -ss" keeps increasing by about 70,000 lines per day, so I'm still doomed.
Another server running the same version of FreeBSD does not have this problem, in fact its "pfctl -ss" is only hundreds of line in total and only fluctuates there.
What could be the problem?
I know upgrading OS may be a good try but I can't do it right now and especially the other server of the exact OS has no such problem.
The following are some relevant files. Thanks!
# uname -a
FreeBSD 11.1-RELEASE-p15 FreeBSD 11.1-RELEASE-p15 #0: Thu Sep 27 06:05:25 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
# cat /boot/loader.conf |egrep -v ^#
accf_http_load="YES"
accf_data_load="YES"
kern.ipc.shmseg="1024"
kern.ipc.soacceptqueue=1024
# cat /etc/pf.conf
ext_if="em0"
int_if="lo0"
set limit { states 1000000, frags 1000000, src-nodes 100000, table-entries 1000000 }
scrub in on $ext_if all fragment reassemble
block return log all
pass on $int_if all keep state
pass out on $ext_if proto udp all keep state
pass out on $ext_if proto tcp all modulate state
pass in on $ext_if proto tcp from any to $ext_if port 443 flags S/SA keep state
pass in on $ext_if proto tcp from 1.2.3.4 to $ext_if port 3306 flags S/SA keep state
table <ssh> { 10.0.0.1 } persist
pass in on $ext_if proto tcp from <ssh> to $ext_if port 22
table <blacklist> persist
block in on $ext_if proto tcp from <blacklist> to any