PF problems

Hi.

I am using PF on a commercial server which is used for both irc and web services.

The main purpose of the PF rules is to stop people on irc accounts from using web ip addresses and to rate limit connections to sshd and web services.

The problem I am having is after a random period of time timeouts will start occuring on new connections, these timouts do not occur on rate limited connections, they are occuring on irc services. They stop occuring when PF is turned off. In addition nothing appears in the PF logs when they occur.

I have only been using PF for around 6 months, there is a severe lack of documentation and discussion on it when compared to ipfw so its very much a learning curve for me, I will provide my ruleset below with usernames and ip's censored out. My ip is 3rd in the trusted ip list and still gets random timeouts when PF is up, suggesting its either the scrub or the stateful part of the firewall having problems.

Some notes about the following rules.

(a) keep-state on the user rules was originally flags S/SA
(b) allow-opts wasnt originally in place
(c) scrub was originally set to 'scrub in on $ext_if fragment reassemble'
(d) I am noticing no problems on the rules above the user rules, eg. sshd is always accessible, www is always accessible and so on.
(e) -edit- sorry to add this is a FreeBSD 6.3 machine so is not the newest PF version.

The timeouts occur on the rules at the end that restrict what ips can be used for outbound connections, there is not an outright blocking of traffic instead connections will timeout and then work again, then timeout then work again and so on.

The S/SA flags were removed of the user rules as I dont want those connections to drop when I flush the states, however they still drop.

Finally another strange occurance is that if I do pfctl -s info this ' searches 104376 0.1/s' constantly increases the rate, so this line I pasted is right after I did a flush and is 0.1/sec if I leave PF running without flushing the rate of searches keeps going up and up and after a few days is about 200/sec.

So summary.

1 - how to stop stateful connections dropping when flush states, seems according to docs this can be done by allowing any packet to create a new state but when I removed flags S/SA this didnt do this.
2 - how to stop random new timeouts when trying to establish new connections, seems to only be affecting the last rules which restrict users.
3 - why is the state search rate counter getting higher with uptime.
4 - how to fully disable scrubbing, to see if this is why I get random timeouts.
5 - any obvious problems with the rules please point out thanks.

ext_if="bge0"
int_if="lo0"
trusted = "{ 208.x.x.x, 85.x.x.x, 87.x.x.x }"
someuser1 = "{ 85.x.x.x, 85.x.x.x, 85.x.x.x, 85.x.x.x }"
someuser2 = "{ 85.x.x.x, 85.x.x.x, 85.x.x.x }"
someuser3 = "{ 85.x.x.x }"
someuser4 = "{ 85.x.x.x }"
someuser5 = "{ 85.x.x.x }"
someuser6 = "{ 85.x.x.x }"
someuser7 = "{ 85.x.x.x }"
main="85.x.x.x"
default="{ <all server ips here except first one> }"
icmp_types="{ echoreq, unreach }"
table <badhosts> persist
table <allowed> { <all server ips here including first one> }

set loginterface $ext_if
set optimization normal
set block-policy drop
set state-policy if-bound
set require-order yes
set fingerprints "/etc/pf.os"
set debug misc

set skip on lo0

scrub on $ext_if reassemble tcp

nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"

rdr pass on $int_if proto tcp from any to any port ftp -> 127.0.0.1 \
port 8021

pass in quick on $ext_if proto { tcp, udp, icmp } from $trusted to any keep state

block drop in log all

block return out log all
block quick on $ext_if from <badhosts> to any

antispoof for $ext_if inet

anchor "ftp-proxy/*"

# Allow for server -> outside connections
pass out on $ext_if proto { udp, gre, icmp } from any to any keep state
pass out on $ext_if proto tcp from $ext_if to any flags S/SA keep state
pass in on $ext_if inet proto tcp from any to $ext_if \
port 10100:10200 keep state

pass in inet proto icmp all icmp-type $icmp_types keep state

# Port opening
pass in quick on $ext_if proto udp from 83.x.x.x to $main port 161
pass in quick on $ext_if proto udp from any to $default port 53
pass in on $ext_if proto tcp from any to $default flags S/SA keep state
pass in quick on $ext_if proto tcp from any to $main port { 42, 25, 2222, 3306, 21, 587, 143, 465, 110, 993, 995, 53, 20000 } flags S/SA keep state (max-src-conn-rate 20/60, overload <badhosts> flush global)

pass in quick on $ext_if proto tcp from any to $ext_if port { 80, 443 } flags S/SA synproxy state (max-src-conn-rate 100/10, overload <badhosts> flush global)

#user name restrictions
block out log on $ext_if proto tcp from any to any user someuser
pass out quick on $ext_if proto tcp from $someuser to any allow-opts user someuser
block out log on $ext_if proto tcp from any to any user someuser2
pass out on $ext_if proto tcp from $someuser2 to any user someuser2 allow-opts keep state
block out log on $ext_if proto tcp from any to any user someuser3
pass out on $ext_if proto tcp from $someuser3 to any user someuser3 allow-opts keep state
block out log on $ext_if proto tcp from any to any user someuser4
pass out on $ext_if proto tcp from $someuser4 to any user someuser4 allow-opts keep state
block out log on $ext_if proto tcp from any to any user someuser5
pass out on $ext_if proto tcp from $someuser5 to any user someuser5 allow-opts keep state
block out log on $ext_if proto tcp from any to any user someuser6
pass out on $ext_if proto tcp from $someuser6 to any user someuser6 allow-opts keep state
block out log on $ext_if proto tcp from any to any user someuser7
pass out on $ext_if proto tcp from $someuser7 to any user someuser7 allow-opts keep state
 
chrcol said:
1 - how to stop stateful connections dropping when flush states
May be it's because you have 'synproxy state' in one of the rules.
"synproxy state (...) includes the functionality of keep state and modulate state." (http://www.openbsd.org/faq/pf/filter.html)
"modulated connections will not recover when the state table is lost (firewall reboot, flushing the state table, etc...)" (man pf.conf)
 
chrcol said:
Hi.

The problem I am having is after a random period of time timeouts will start occuring on new connections, these timouts do not occur on rate limited connections, they are occuring on irc services. They stop occuring when PF is turned off. In addition nothing appears in the PF logs when they occur.

Do you check the number of currently active states? From pf manual page:
Code:
           For example:

                 set timeout tcp.first 120
                 set timeout tcp.established 86400
                 set timeout { adaptive.start 6000, adaptive.end 12000 }
                 set limit states 10000
and
Code:
     set limit
           Sets hard limits on the memory pools used by the packet filter.
           See zone(9) for an explanation of memory pools.

           For example,

                 set limit states 20000

           sets the maximum number of entries in the memory pool used by state
           table entries (generated by keep state rules) to 20000.  Using

                 set limit frags 20000

           sets the maximum number of entries in the memory pool used for
           fragment reassembly (generated by scrub rules) to 20000.  Finally,

                 set limit src-nodes 2000

           sets the maximum number of entries in the memory pool used for
           tracking source IP addresses (generated by the sticky-address and
           source-track options) to 2000.

           These can be combined:

                 set limit { states 20000, frags 20000, src-nodes 2000 }
I've encountered some 'timeout'-like trouble when I used NAT with PF (each connection creates a state), and the number of concurent connections was very high. You may need to check this sysctl:vm.zone:
Code:
# sysctl vm.zone | grep "ITEM\|^pf"
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
pfosfp:           28,        0,    208,    173,     2704
pfospfen:        108,        0,    348,     84,     4524
pfiaddrpl:        92,        0,      0,      0,        0
pfstatescrub:     28,        0,      0,      0,        0
pffrcent:         12,    50141,      0,      0,        0
pffrcache:        48,    10062,      0,      0,        0
pffrag:           48,        0,      0,    156,       10
pffrent:          16,     5075,      0,    203,       18
pfrkentry2:      156,        0,      0,      0,        0
pfrkentry:       156,        0,     56,     94,      721
pfrktable:      1240,        0,     19,     32,      338
pfpooladdrpl:     68,        0,      6,    106,       71
pfaltqpl:        128,        0,      0,      0,        0
pfstatepl:       260,    10005,    629,    706,  8160740
pfrulepl:        604,        0,     66,     72,      787
pfsrctrpl:       100,    10023,      0,      0,        0
 
nice command, here is the output.

Code:
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
pfosfp:           28,        0,    208,    300,    13312
pfospfen:        108,        0,    348,    156,    22272
pfiaddrpl:        92,        0,      0,      0,        0
pfstatescrub:     28,        0,   1273,   6093,  6618612
pffrcent:         12,    50141,      0,      0,        0
pffrcache:        48,    10062,      0,      0,        0
pffrag:           48,        0,      0,    390,   354203
pffrent:          16,     5075,      0,    406,   707530
pfrkentry2:      156,        0,      3,     47,        7
pfrkentry:       156,        0,      0,   1775,     3032
pfrktable:      1240,        0,      1,     32,     1770
pfpooladdrpl:     68,        0,      1,    167,       32
pfaltqpl:        128,        0,      0,      0,        0
pfstatepl:       260,    10005,   1055,   2155, 138451258
pfrulepl:        604,        0,    436,    566,    12381
pfsrctrpl:       100,    10023,    127,    458,   199942

incidently PF has not been flushed for some weeks now, check the amount of searches per second reported.

State Table Total Rate
current entries 1070
searches 648808577 173.0/s

I have had a pm here of someone with the same issue who thinks it may be a bug, I am starting to wonder the same thing. Still getting dropped connections that dont trigger any logging in the PF log.
 
The only thing I may suggest is use of 'pftop', which will show you a lot of things related to pf. From the command output, it seems that your memory alloccation is fine (approx. 1000 states out of approx. 10000).
 
Back
Top