Using FreeBSD 11 for NAT/firewall; performance degrades over time

After a long while using an old, giant Pentium 4 server as my firewall/router/dhcp server, I upgraded to one of these little Qotom boxes. With a Celeron CPU and 4 Intel gigabit ports, it seemed like it should be good at doing the same job.

I am having some performance problems though, and I am in over my head trying to figure it out. Basically, remote terminals (ipads, etc) on wifi may have eventually a hard time connecting to web sites. Try again, and the connection is likely to go through. Once a connection is open, data comes through at the expected rate. (I do not know of a way to replicate this for testing with command line utilities. What's a CLI way to emulate loading a busy web page, and log all the results?)

I realize there could be a LOT of reasons for a browser getting hung up "Connecting..." I believe I have ruled out basics like router cables and DNS problems and wifi signal quality. And here's where it gets weird: I just rebooted the FreeBSD box, and thereafter connections on the various browsers on various gadgets all went through immediately as you would expect. I am expecting it to degrade over the next day or so.

The machine is FreeBSD 11, amd64 src, 2 GB RAM, a GENERIC kernel, runs isc-dhcps, ipnat, ipf, sshd, and that is about all. A slower FreeBSD 8 machine with the same config files did this duty for years without this trouble--though it had more RAM. My ipf ruleset is not too big, just a few forwarding rules for things like Xbox Live.

While I am open to any ideas, I find the fact that a reboot helped the performance of devices on the network to be an interesting lead. In this freshly rebooted state, all seems to be well. What could be failing over time?

Any ideas?
Did you scan the logfiles for hints? grep -Ri 'error|warning|failed|not' /var/log/*'

You don't run ZFS on 2 GB RAM, don't you? When performance deteriorates is worth to do swapinfo for checking swap usage.

with the same config files
Did you just copy the old config files into the new system?

A system also can slow down if CPU temperature is too high.

Try tools like top, netstat to get an idea what's going on ...
Basically, remote terminals (ipads, etc) on wifi may have eventually a hard time connecting to web sites. Try again, and the connection is likely to go through.
Is IPv6 perhaps enabled on the wireless network? If the rest of your network doesn't use IPv6 this can cause delays and connection errors initially.
IPF and IPNAT are broken in FreeBSD 11. Run ipfstat|egrep 'NAT failure'. If the counts start off at 0 and grow over a few hours/days, then that is what's happening. I ran into this as well and the options are to downgrade to 9.3 or use a different firewall setup such as pf or ipfw.

Link to past discussion about this problem:
Guys, thanks much for the replies! Those are the ideas I needed, thank you. I've been using ipnat/ipf for a long time, but pretty casually and have never had to deal with a problem before.

This morning, about 9 hours after the last reboot device performance was very poor again. I checked top and things look fine:

CPU:  0.0% user,  0.0% nice,  0.0% system,  0.2% interrupt, 99.8% idle
Mem: 7844K Active, 233M Inact, 304M Wired, 54M Buf, 1280M Free
Swap: 2689M Total, 2689M Free

I also captured ipfstat -s and that output is attached. Unfortunately I did not get it without the -s argument so I can't check NAT failures.

I decided to try flushing the state table ( ipf -FS) and that did not help. Then I rebooted, and everything was great.

If I flush state and reload rules, is that in effect restarting ipf? I'd like to eliminate anything I can from contention without a system reboot.

getopt -- "Copied configs" in this case meant just ipf/ipnat configs. rc.conf was tuned by hand with only the minimum directives to set up dhcp, ipf, sshd, etc. The box is really lean and is not running zfs. :) However, my ipf/ipnat configs might still suck, though I did have help from a much more experienced user when I set them up. I have not checked the system temperature. I suspect it is OK because CPU perf stays OK, and network perf is OK once a connection is open. I'll look up how to check, though.

I have disabled ipv6 to the best of my ability on FreeBSD. All other devices (consumer wifi router--not doing routing, just access--ipads, etc) all are at their default. The connection problems do happen on wired and wireless connections, though.

Datapanic -- great lead, thanks a bunch! After 19 minutes up uptime, this is the output. Would I expect the number of failures to increment with failed connections? Or will these be huge numbers, like it's tracking packets? I will keep monitoring it and check that thread!

0       input block reason IPv4 NAT failure
0       input block reason IPv6 NAT failure
1       output block reason IPv4 NAT failure
0       output block reason IPv6 NAT failure

Thanks all!


  • ipfstat-s.txt
    1.6 KB · Views: 291
Datapanic -- NAT failures are climbing all right. Have to check that other thread still, not sure how much is too much, but it is climbing steadily with a couple of people doing light web browsing. (And a couple of computers and devices sitting idle, doing whatever they do when the humans aren't looking.)

88      output block reason IPv4 NAT failure

EDIT: The poster in that other thread is describing my symptoms exactly, and the thread provides the tools I need to probe and verify. Awesome, thanks again.
It's been over a year. Has this problem been fixed? I have since painfully migrated my 5 gateways over to pf but still like some of the features of ipf, such as the "log once" option.