PF how much has it been tested in heavy server use? got problem.

C

chrcol

I have had numerous PF issues with PF over the years, some I never got answers to and others I think are bad design rather than bugs but here is my latest problem.

I have a server making outgoing TCP connections to a mysql server on the same lan (but with internet ip, same subnet).

I see in the PF log that return ack packets for 'some' but not all connections are been blocked by the default inbound deny rule. I say 'some' as it seems to be random. There is nothing special about the ones that get blocked.

match): block in on bce0: x.x.x.x.3306 > y.y.y.y.43735: . ack 4136846456 win 520 <nop,nop,timestamp 1920029056 3483089099>
Click to expand...

Code: 
ext_if="bce0"
int_if="lo0"
set optimization normal
set loginterface $ext_if
set block-policy drop
set state-policy floating
set require-order yes
set fingerprints "/etc/pf.os"
set debug misc

set skip on lo0

scrub in all fragment reassemble no-df random-id
scrub out all max-mss 1460

pass in log quick on $ext_if proto { tcp, udp, icmp } from $trusted to any
pass out quick on $ext_if proto { tcp, udp, icmp } from any to $trusted

block drop in log all

antispoof log quick for $ext_if inet

pass out on $ext_if proto { udp, gre, icmp } from any to any keep state
[B]pass out on $ext_if proto tcp from $ext_if to any flags S/SA keep state[/B]
pass in quick on $ext_if proto tcp from any to any port { 22, 21, 53, 80, 443, 110, 995, 25, 465, 143, 993 } flags S/SA keep state

the original ruleset is more then that but I have made it more simple to try and and stop the problem, so no modulate or synproxy state now used or any rate limiting but still the problem. I put the rule in bold which I assume should be allowing all return packets through.
 
PF has been tested very, very heavily in very hostile production environment but ...

  1. It has been written and tested by OpenBSD developers on OpenBSD.
  2. FreeBSD implementation is circa 1.5-2.5 years behind official release
    version (depends on the version of FreeBSD (I think FreeBSD 9.xxx has a
    version from OpenBSD 4.5 or 4.6)
  3. FreeBSD version is not a faithful implementation due to the significant differences in the network stack between FreeBSD and OpenBSD.

Above being said I know lots of very heavy PF users on FreeBSD. I am always confused by that.

Why would anybody who needs PF use FreeBSD instead of OpenBSD? Why would anybody use PF over the native IPFW on FreeBSD?

I am not very familiar with the native IPFW but I would guess that its implementation is much more suitable for FreeBSD. Arguably from my experience with IPFW on OS X the syntax is ugly but it could do most if not all things PF can do.

For the record I use neither IPFW nor OS X and the last time I installed FreeBSD was over three years ago.

Also for the record the syntax of PF has changed so dramatically after huge Henning patches between OpenBSD 4.5 and OpenBSD 4.6 that you will have a very hard time to find any OpenBSD user who will be willing even to look at your old (FreeBSD-specific syntax) pf.conf file.
 
If you know that you are going to have LAN traffic going to your MySQL server, why don't you just add rules for it explicitly, that way no memory is being used to track a state that you know ahead of time is okay.

Code: 
pass out on $ext_if proto tcp from $ext_if to y.y.y.y port 3306
pass in on $ext_if proto tcp from y.y.y.y port 3006 to any
 
gordon correct however its worrying that this behaviour is even happening. What would your answer be if it wasnt a local server? They are local server's to each other but communicating over the internet interface. I will be testing how this behaves tommorow using it as trusted lan traffic in PF.

Oko so what you saying is FreeBSD 8 uses an obselete version of PF? I agree with what you state as well in terms of the code been migrated over and always been too far behind. I am considering going back to ipfw as that in my experience showed no buggy behaviour. My motivation for changing was the weak rate limiting capabilities of ipfw, and its poor logging performance. However it never compromised uptime.

Is PF in FreeBSD 9 going to be obselete by the time it hits STABLE?
 
chrcol said:
I have a server making outgoing TCP connections to a mysql server on the same lan (but with internet ip, same subnet).
Click to expand...

I may be wrong but i think you are in a situation like explain at this page redirection and reflexion.

In other word, you can't connect from a client to a server in the same lan through pf and external ip.

There are some tips after the explications which you can try to adjust to your pf rules (old pf syntax vs new pf syntax but the idea is here).

Let us know if this could help you.
 
2 more things have caught my eye as well.

the stats from when PF was running are 134k/sec searches on state table, and 13.6/sec of mismatches. The first number is very high as there wont even be close to that in connections to the server, so PF is doing more searches than connections and these state mismatches I assume relate to the mysql issues.
 
chrcol said:
2 more things have caught my eye as well.

the stats from when PF was running are 134k/sec searches on state table, and 13.6/sec of mismatches. The first number is very high as there wont even be close to that in connections to the server, so PF is doing more searches than connections and these state mismatches I assume relate to the mysql issues.
Click to expand...

Looks like the state expires while the connection is still alive.

What is the max number of state entries in PF? By default it's 10000, but the timers for expiration are "auto-adaptive". See set timeout in pf.conf.
 
With mysql traffic now going over as trusted the mismatches are now down to 0.3/sec, a massive drop but is interesting some still exist.

searches topped out at around 3k/sec at peak traffic levels. so that is also sharply down. Seems outgoing connections were making it go crazy.

I upped the 10k to 80k value. Was previously at 60k.
 
Just some thoughts...
Don't understand what you see as a problem here. You see dropped packets randomly?! As plamaiziere said, this could be a timeout issue. You should try to tcpdump the MySQL traffic and analyse whats going on within the particular tcp-session befor such a drop occurs. Why do you play with the hard limits for the statetable size? How many sessions do you expect? 80k is a lot. Don't know what else your server is doing, but you should only have a couple (not 10k) of database related sessions, everything else would I call bad application design.

I also don't understand gordons idea. If those rules were intended to implement a stateless behavior, than I think this is wrong. "Keep state" should be the default in pf of FreeBSD 8. Also it's dangourous, as this would allow incoming connections to (e.g.) port 22 from the MySQL server if 3306 is used as source-port.
 
honk said:
I also don't understand gordons idea. If those rules were intended to implement a stateless behavior, than I think this is wrong. "Keep state" should be the default in pf of FreeBSD 8. Also it's dangourous, as this would allow incoming connections to (e.g.) port 22 from the MySQL server if 3306 is used as source-port.
Click to expand...

That would only be true depending on how you architect your rules. I would submit that if you are worried about your database server hacking into your web server, I think you have larger problems. =)
 
gordon@ said:
I would submit that if you are worried about your database server hacking into your web server, I think you have larger problems. =)
Click to expand...

The vertical scrollbar secured my eyes from the last line which already allows port 22 in the example. ;)

I just want to say that a "no state" (and possibly a "quick") statement is missing if a stateless behavior is desired, which should lead to a poor performance as every packets is compared to the ruleset instead of the statetable.
 
What you might be seeing is that the final ACKs of connections that are being closed are seen as out of state because they arrive too late for some reason and the states those ACKs would have belonged to no longer exist.

M0n0wall's doc entry about the very same thing: http://doc.m0n0.ch/handbook/faq-legit-traffic-dropped.html. M0n0wall uses ipfilter instead of pf but the reasons for seeing legitimate traffic being blocked in the logs are the same.
 
honk said:
The vertical scrollbar secured my eyes from the last line which already allows port 22 in the example. ;)

I just want to say that a "no state" (and possibly a "quick") statement is missing if a stateless behavior is desired, which should lead to a poor performance as every packets is compared to the ruleset instead of the statetable.
Click to expand...

Reliability comes before performance on my servers.

However they were using states, now it's using none as I have it going over a LAN interface which is skipped in pf.conf. I was going to try the 'no state' method of doing things but ended up routing it over the lan interface instead.
 
kpa said:
What you might be seeing is that the final ACKs of connections that are being closed are seen as out of state because they arrive too late for some reason and the states those ACKs would have belonged to no longer exist.

M0n0wall's doc entry about the very same thing: http://doc.m0n0.ch/handbook/faq-legit-traffic-dropped.html. M0n0wall uses ipfilter instead of pf but the reasons for seeing legitimate traffic being blocked in the logs are the same.
Click to expand...

Nice explanation, however in my situation it was preventing connections from being established.
 
You must log in or register to reply here.
Back
Top