PF FreeBSD 10.2 + PF tuning tips

D

Daniel Santos

Hello all,

I am currently responsible for replacing an old DELL 2950 running IPF for a new server, as we are currently having some performance issues.

I want to use PF and the latest stable version of FreeBSD as it seems to me to be the best open source stateful firewall at the moment.

My main concern is that I need to be able to handle 150k concurrent connections + 5k new connections/sec + a throughput of 1.0~1.5Gb/s. The default configuration for PF is probably not well suited for such a large environment, so I need some help with some tuning guidance.

What variables should I consider (state tables, mbufs, etc) ? Is there any tuning guide for large scale routers/firewalls running PF?
 
You will definitely need to increase the state limits. The defaults are very low.

Code: 
set limit states 200000

You can also monitor the states by
Code: 
# pfctl -sm
 
From tuning(7):

Code: 
[FILE]kern.ipc.nmbclusters[/FILE] may be adjusted to increase the number of network
mbufs the system is willing to allocate. Each cluster represents approx-
imately 2K of memory, so a value of 1024 represents 2M of kernel memory
reserved for network buffers. You can do a simple calculation to figure
out how many you need. If you have a web server which maxes out at 1000
simultaneous connections, and each connection eats a 16K receive and 16K
send buffer, you need approximately 32MB worth of network buffers to deal
with it. A good rule of thumb is to multiply by 2, so 32MBx2 = 64MB/2K =
32768. So for this case you would want to set kern.ipc.nmbclusters to
32768. We recommend values between 1024 and 4096 for machines with mod-
erates amount of memory, and between 4096 and 32768 for machines with
greater amounts of memory. Under no circumstances should you specify an
arbitrarily high value for this parameter, it could lead to a boot-time
crash. The -m option to netstat(1) may be used to observe network clus-
ter use. Older versions of FreeBSD do not have this tunable and require
that the kernel config(8) option NMBCLUSTERS be set instead.
 
As a matter of fact, this new box have a total of 64GB of memory available. Don't ask me why as I was not the one that bought it =].

With this in mind, what do you guys consider a safe maximum value for the mbufs?

Thanks in advance for your help.
 
I'd probably give -HEAD a try as there are quite a few enhancements done but if you want to go 10.2...

https://fasterdata.es.net/host-tuning/freebsd/
https://fasterdata.es.net/host-tuning/nic-tuning/ (my em nics uses 1024 running -HEAD by default)
http://blog.cochard.me/2015/09/receipt-for-building-10mpps-freebsd.html

As a side note, you might want to try rebuilding world with CPUTYPE?=core2 in /etc/make.conf or whatever matches your CPU more accurately (don't forget to disable debugging if you run -HEAD). I haven't done any benchmarks but I would expect a few % in boost in performance by setting CPUTYPE. I'd be very reluctant to mess around with the C/CPP flags however.

You probably wont need to change kern.ipc.nmbclusters but I would however do benchmarks between each change as they may/may not decrease or increase performance depending on workload.

Also, setting
Code: 
set optimization aggressive
in pf.conf might be of interest.

//Danne
 
You really don't want to run -CURRENT in production. For testing purposes sure, just not production. -CURRENT is continuously evolving and at times won't even build properly. It's meant to be used by developers. Stick to a -RELEASE version, or if you want, use a -STABLE.
 
-CURRENT works fine in most cases and there are rarely build issues. But as with everything else, do testing first. Pretty much all supported platforms except i386/AMD64 are running -HEAD due to recent development and it's more or less recommended. I've personally used -HEAD on MIPS hardware for years without issues (firewall, gateway), sounds like you haven't even given it a shot. I would however be more careful if it involved a lot of 3rd party software out of base.
//Danne
 
I would really like to keep using the GENERIC kernel as long as possible and stress out the variables tunning. If it comes to the point that a 12 cores 2.5GHz machine with 64GB of RAM requires kernel recompiling, I rather beg my bosses for a commercial solution with proper specific hardware like Paloalto.
 
I would run 10.2-RELEASE and not CURRENT. My tested fw is running on 9.2-RELEASE with 32GB of RAM. No tuning except the state limit. The system uses 5 jails. Each jail is attached to a class C network. Also, every jail runs squid for reverse proxying. The system is a firewall for a big advertising company and squid is being used for reverse proxying of all requests. I get thousands of requests per sec. Never had a problem and I am using GENERIC kernel.
 
diizzy said:
Pretty much all supported platforms except i386/AMD64 are running -HEAD due to recent development and it's more or less recommended. I've personally used -HEAD on MIPS hardware for years without issues (firewall, gateway), sounds like you haven't even given it a shot.
Click to expand...
MIPS is a Tier 3 platform, which means there are no releases for it and the only version that's available is -CURRENT.
 
diizzy said:
SirDice
So? Given your previous response it should be unusable....
ARM users are also recommended to use -CURRENT rather than release.
Click to expand...
ARM is a Tier 2 platform.

18. Support for Multiple Architectures

FreeBSD-CURRENT should not be considered a fast-track to getting new features before the next release as pre-release features are not yet fully tested and most likely contain bugs. It is not a quick way of getting bug fixes as any given commit is just as likely to introduce new bugs as to fix existing ones. FreeBSD-CURRENT is not in any way “officially supported”.
Click to expand...
23.4. Tracking a Development Branch

For FreeBSD versions that match [2], we strongly encourage users to understand what -CURRENT actually means, because there is a common misunderstanding that -CURRENT means "the latest, greatest version". In fact, -CURRENT is the development branch of the FreeBSD operating system. This is where new developments are tested and evaluated, which means that any part of the system may break at any given time. For the most part, -CURRENT should be considered the playground for FreeBSD developers, and for 'adventurous users' who don't mind that their system breaks. These users typically do not require any help to get their systems back into a working state.
Click to expand...
Thread topics-about-unsupported-freebsd-versions.40469

Like I said, for testing purposes sure, use -CURRENT. Just don't use it for production systems.
 
..and you misread the part of where I said do some testing first? MIPS works fine and is used in production, that's a simple fact despite your claims.
There are also several devs who runs -HEAD in production and real world testing doesn't hurt either for the project itself. That said, testing is important as I mentioned earlier and 10.2 might be just "good enough".
//Danne
 
Hello all,

It´s been a while since my last post, but I am still working on my project. I want to thank you all for the help.
Now I am trying to increase the maximum number of tables without success. I have the following lines in my pf.conf:

Code: 
...
set limit states 256000
set limit frags 1024000
set limit tables 10000
set limit table-entries 100000
...

But when I run sudo pfctl -vf /etc/firewall/pf.conf it gives me the following errors:

Code: 
No ALTQ support in kernel
ALTQ related functions disabled
set limit states 256000
set limit frags 1024000
pfctl: Bad pool name.
/etc/firewall/pf.conf:6: unable to set limit tables 10000
set limit table-entries 100000
set require-order yes
...
table <tb_Geo2ipCountry_UM> persist file "/etc/firewall/tables/geo2ip_country_UM.tb"
table <tb_Geo2ipCountry_AS> persist file "/etc/firewall/tables/geo2ip_country_AS.tb"
table <tb_Geo2ipCountry_CA> persist file "/etc/firewall/tables/geo2ip_country_CA.tb"
table <tb_Geo2ipCountry_US> persist file "/etc/firewall/tables/geo2ip_country_US.tb"
/etc/firewall/pf.conf:288: cannot define table tb_Geo2ipCountry_US: Cannot allocate memory
...

Table tb_Geo2ipCountry_US is supposed to have around 50k lines which should be a valid size according to set limit table-entries.
I found some threads about setting table limits on pf but none of them were answered.
Does anybody went through this?

Thanks in advance,
 
I think I may have found a bug.
The allocation memory error occurs because when I set the table-entries limit in pf.conf with:
Code: 
...
set limit table-entries 1024000
...

pfctl -vf does not actually update this limit. If I run vmstat -z, I can see that the default limit is still there. The funny thing is that when I run echo "set limit table-entries 1024000" | pfctl -mf - just after reloading pf's configuration from file, pfctl properly updates the table-entries limit.
Where would be the right place to rerport this bug?
 
You must log in or register to reply here.
Back
Top