PF pfctl -ss increases infinitely

J

john2011

Hi, a FreeBSD server I manage has got network problem twice now, with the following error in /var/log/messages:

kernel: [zone: pf states] PF states limit reached
kernel: sonewconn: pcb 0xfffff80100947d20: Listen queue overflow: 193 already in queue awaiting acceptance (358 occurrences)

After the first incident, I followed some online suggestion against this error to add the following to /etc/pf.conf:

set limit { states 1000000, frags 1000000, src-nodes 100000, table-entries 1000000 }

However, after some days the same error happened again, because now "pfctl -ss" is over 1000000 lines.

I followed more online suggestions, so that now I have:

# sysctl -a |grep keep
vfs.nfs.nfs_keep_dirty_on_error: 0
net.inet.tcp.keepidle: 600000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.keepinit: 75000
net.inet.tcp.keepcnt: 4
net.inet.tcp.always_keepalive: 1

still I can see "pfctl -ss" keeps increasing by about 70,000 lines per day, so I'm still doomed.

Another server running the same version of FreeBSD does not have this problem, in fact its "pfctl -ss" is only hundreds of line in total and only fluctuates there.

What could be the problem?

I know upgrading OS may be a good try but I can't do it right now and especially the other server of the exact OS has no such problem.

The following are some relevant files. Thanks!

# uname -a
FreeBSD 11.1-RELEASE-p15 FreeBSD 11.1-RELEASE-p15 #0: Thu Sep 27 06:05:25 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

# cat /boot/loader.conf |egrep -v ^#
accf_http_load="YES"
accf_data_load="YES"
kern.ipc.shmseg="1024"
kern.ipc.soacceptqueue=1024

# cat /etc/pf.conf
ext_if="em0"
int_if="lo0"

set limit { states 1000000, frags 1000000, src-nodes 100000, table-entries 1000000 }

scrub in on $ext_if all fragment reassemble

block return log all
pass on $int_if all keep state

pass out on $ext_if proto udp all keep state
pass out on $ext_if proto tcp all modulate state
pass in on $ext_if proto tcp from any to $ext_if port 443 flags S/SA keep state
pass in on $ext_if proto tcp from 1.2.3.4 to $ext_if port 3306 flags S/SA keep state

table <ssh> { 10.0.0.1 } persist
pass in on $ext_if proto tcp from <ssh> to $ext_if port 22

table <blacklist> persist
block in on $ext_if proto tcp from <blacklist> to any
 
Show some small example of your state table so i can see which port is getting most connection to.

Note: FreeBSD 11.1 is EOL you should upgrade to supported version.
 
I know upgrading OS may be a good try but I can't do it right now
Click to expand...
11.1 has been end-of-life since September 30, 2018.
Upgrading should have been a priority for four years.

Speaking as the person most likely to fix pf bugs: I will just point and laugh at you if you report a bug in 11.1. You're on your own, but you do get the keep the pieces if you break it.
Stop investigating issues with 11.1 and upgrade to a supported version. You're just wasting your time with 11.1.
 
Server getting hammered and/or backend too slow to handle the connection load. "Listen queue overflow" typically means the server isn't able to handle the amount of connections you're receiving. Which could be caused by several different things. I've had this happen once on a server and it turned out to be caused by a broken MySQL table, this caused the server to stall on connections, not handling them quickly enough. Which in turn caused the queue to overflow.

And, as a few people already pointed out, 11.1-RELEASE is end-of-life and not supported anymore.

Topics about unsupported FreeBSD versions
www.freebsd.org

Unsupported FreeBSD Releases

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org
 
Wow, thanks for all your pointers, they are super helpful.

And especially for pointing out elephant in the room. Indeed I should really looking at upgrading the OS now.

Most states are for incoming connections to port 443, and most are in these two states:

TIME_WAIT:TIME_WAIT
FIN_WAIT_2:FIN_WAIT_2

That is, most lines look like:

all tcp m.y.i.p:443 <- 11.22.33.44:33542 TIME_WAIT:TIME_WAIT
all tcp m.y.i.p:443 <- 44.33.22.11:9902 FIN_WAIT_2:FIN_WAIT_2

yet in "netstat -n" those lines are gone long time ago.
 
You can limit the total number of states created by the rule that open port www and limit the maximum number of simultaneous states per given source ip address to 3 or 5. Other good option is to limit the rate of new connection that are established by a single host but you need to consider that some clients may be sharing a single public IP address so it will need some tuning as this will prevent the clients that are behind NAT to access your web site.

For example rules read the "STATEFUL TRACKING OPTIONS" section in the man page of pf.conf(5) and also check the optimization options for expired connections.
 
It would be easier and more efficient to replace HTTP server to the right one than FreeBSD upgrade in this case. For an example, here are some possibly relevant settings from my Hiawatha configuration (which is secure out-of-the-box, without these settings):

Code: 
ConnectionsTotal = 10000
ConnectionsPerIP = 25
CacheSize = 120
#BanOnDeniedBody = 0  (while DenyBody = 'regex')
BanOnFlooding = 10/1:15
BanOnGarbage = 300
# BanOnMaxReqSize = 120 (while MaxRequestSize = 22528)
BanOnMaxPerIP = 60
BanOnInvalidURL = 5
# BanOnSQLi = 0
# BanOnTimeout = 0  (while TimeForRequest = !0)
# BanOnWrongPassword = -:0
KickOnBan = yes
RebanDuringBan = yes
# ShowIndex = no
# WebDAVapp = no
# ExecuteCGI = yes (for specified vhost only)
TimeForCGI = 10
UseFastCGI = PHP-FPM
AccessLogfile = /var/log/hiawatha/access.log
ErrorLogfile = /var/log/hiawatha/errors.log
GarbageLogfile = /var/log/hiawatha/garbage.log
ExploitLogfile = /var/log/hiawatha/exploit.log
SystemLogfile = /var/log/hiawatha/system.log
 
Thank you very much guys.

Since almost all the states are in

TIME_WAIT
FIN_WAIT_2

which are useless since they are even long gone from netstat output, I ended up writing a script to kill such states, and run in cron job. The short script is like:

pfctl -ss |egrep "FIN_WAIT_2:FIN_WAIT_2|TIME_WAIT:TIME_WAIT" |cut -d\ -f3-5 |sed 's/\([^ ]*\) <- \([^ ]*\)/\2 -> \1/; s/:[^ ]*//g' |sort |uniq |\
while read s arrow d
do
pfctl -k $s -k $d 2>/dev/null
done
 
Last edited:
You must log in or register to reply here.
Back
Top