1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

PF problems

Discussion in 'Firewalls' started by chrcol, Mar 5, 2009.

  1. chrcol

    chrcol New Member

    Messages:
    347
    Thanks Received:
    9
    Hi.

    I am using PF on a commercial server which is used for both irc and web services.

    The main purpose of the PF rules is to stop people on irc accounts from using web ip addresses and to rate limit connections to sshd and web services.

    The problem I am having is after a random period of time timeouts will start occuring on new connections, these timouts do not occur on rate limited connections, they are occuring on irc services. They stop occuring when PF is turned off. In addition nothing appears in the PF logs when they occur.

    I have only been using PF for around 6 months, there is a severe lack of documentation and discussion on it when compared to ipfw so its very much a learning curve for me, I will provide my ruleset below with usernames and ip's censored out. My ip is 3rd in the trusted ip list and still gets random timeouts when PF is up, suggesting its either the scrub or the stateful part of the firewall having problems.

    Some notes about the following rules.

    (a) keep-state on the user rules was originally flags S/SA
    (b) allow-opts wasnt originally in place
    (c) scrub was originally set to 'scrub in on $ext_if fragment reassemble'
    (d) I am noticing no problems on the rules above the user rules, eg. sshd is always accessible, www is always accessible and so on.
    (e) -edit- sorry to add this is a FreeBSD 6.3 machine so is not the newest PF version.

    The timeouts occur on the rules at the end that restrict what ips can be used for outbound connections, there is not an outright blocking of traffic instead connections will timeout and then work again, then timeout then work again and so on.

    The S/SA flags were removed of the user rules as I dont want those connections to drop when I flush the states, however they still drop.

    Finally another strange occurance is that if I do pfctl -s info this ' searches 104376 0.1/s' constantly increases the rate, so this line I pasted is right after I did a flush and is 0.1/sec if I leave PF running without flushing the rate of searches keeps going up and up and after a few days is about 200/sec.

    So summary.

    1 - how to stop stateful connections dropping when flush states, seems according to docs this can be done by allowing any packet to create a new state but when I removed flags S/SA this didnt do this.
    2 - how to stop random new timeouts when trying to establish new connections, seems to only be affecting the last rules which restrict users.
    3 - why is the state search rate counter getting higher with uptime.
    4 - how to fully disable scrubbing, to see if this is why I get random timeouts.
    5 - any obvious problems with the rules please point out thanks.

    ext_if="bge0"
    int_if="lo0"
    trusted = "{ 208.x.x.x, 85.x.x.x, 87.x.x.x }"
    someuser1 = "{ 85.x.x.x, 85.x.x.x, 85.x.x.x, 85.x.x.x }"
    someuser2 = "{ 85.x.x.x, 85.x.x.x, 85.x.x.x }"
    someuser3 = "{ 85.x.x.x }"
    someuser4 = "{ 85.x.x.x }"
    someuser5 = "{ 85.x.x.x }"
    someuser6 = "{ 85.x.x.x }"
    someuser7 = "{ 85.x.x.x }"
    main="85.x.x.x"
    default="{ <all server ips here except first one> }"
    icmp_types="{ echoreq, unreach }"
    table <badhosts> persist
    table <allowed> { <all server ips here including first one> }

    set loginterface $ext_if
    set optimization normal
    set block-policy drop
    set state-policy if-bound
    set require-order yes
    set fingerprints "/etc/pf.os"
    set debug misc

    set skip on lo0

    scrub on $ext_if reassemble tcp

    nat-anchor "ftp-proxy/*"
    rdr-anchor "ftp-proxy/*"

    rdr pass on $int_if proto tcp from any to any port ftp -> 127.0.0.1 \
    port 8021

    pass in quick on $ext_if proto { tcp, udp, icmp } from $trusted to any keep state

    block drop in log all

    block return out log all
    block quick on $ext_if from <badhosts> to any

    antispoof for $ext_if inet

    anchor "ftp-proxy/*"

    # Allow for server -> outside connections
    pass out on $ext_if proto { udp, gre, icmp } from any to any keep state
    pass out on $ext_if proto tcp from $ext_if to any flags S/SA keep state
    pass in on $ext_if inet proto tcp from any to $ext_if \
    port 10100:10200 keep state

    pass in inet proto icmp all icmp-type $icmp_types keep state

    # Port opening
    pass in quick on $ext_if proto udp from 83.x.x.x to $main port 161
    pass in quick on $ext_if proto udp from any to $default port 53
    pass in on $ext_if proto tcp from any to $default flags S/SA keep state
    pass in quick on $ext_if proto tcp from any to $main port { 42, 25, 2222, 3306, 21, 587, 143, 465, 110, 993, 995, 53, 20000 } flags S/SA keep state (max-src-conn-rate 20/60, overload <badhosts> flush global)

    pass in quick on $ext_if proto tcp from any to $ext_if port { 80, 443 } flags S/SA synproxy state (max-src-conn-rate 100/10, overload <badhosts> flush global)

    #user name restrictions
    block out log on $ext_if proto tcp from any to any user someuser
    pass out quick on $ext_if proto tcp from $someuser to any allow-opts user someuser
    block out log on $ext_if proto tcp from any to any user someuser2
    pass out on $ext_if proto tcp from $someuser2 to any user someuser2 allow-opts keep state
    block out log on $ext_if proto tcp from any to any user someuser3
    pass out on $ext_if proto tcp from $someuser3 to any user someuser3 allow-opts keep state
    block out log on $ext_if proto tcp from any to any user someuser4
    pass out on $ext_if proto tcp from $someuser4 to any user someuser4 allow-opts keep state
    block out log on $ext_if proto tcp from any to any user someuser5
    pass out on $ext_if proto tcp from $someuser5 to any user someuser5 allow-opts keep state
    block out log on $ext_if proto tcp from any to any user someuser6
    pass out on $ext_if proto tcp from $someuser6 to any user someuser6 allow-opts keep state
    block out log on $ext_if proto tcp from any to any user someuser7
    pass out on $ext_if proto tcp from $someuser7 to any user someuser7 allow-opts keep state
     
  2. BLASTER

    BLASTER New Member

    Messages:
    3
    Thanks Received:
    0
    May be it's because you have 'synproxy state' in one of the rules.
    "synproxy state (...) includes the functionality of keep state and modulate state." (http://www.openbsd.org/faq/pf/filter.html)
    "modulated connections will not recover when the state table is lost (firewall reboot, flushing the state table, etc...)" (man pf.conf)
     
  3. ecazamir

    ecazamir New Member

    Messages:
    215
    Thanks Received:
    27
    Do you check the number of currently active states? From pf manual page:
    Code:
               For example:
    
                     set timeout tcp.first 120
                     set timeout tcp.established 86400
                     set timeout { adaptive.start 6000, adaptive.end 12000 }
                     set limit states 10000
    
    and
    Code:
         set limit
               Sets hard limits on the memory pools used by the packet filter.
               See zone(9) for an explanation of memory pools.
    
               For example,
    
                     set limit states 20000
    
               sets the maximum number of entries in the memory pool used by state
               table entries (generated by keep state rules) to 20000.  Using
    
                     set limit frags 20000
    
               sets the maximum number of entries in the memory pool used for
               fragment reassembly (generated by scrub rules) to 20000.  Finally,
    
                     set limit src-nodes 2000
    
               sets the maximum number of entries in the memory pool used for
               tracking source IP addresses (generated by the sticky-address and
               source-track options) to 2000.
    
               These can be combined:
    
                     set limit { states 20000, frags 20000, src-nodes 2000 }

    I've encountered some 'timeout'-like trouble when I used NAT with PF (each connection creates a state), and the number of concurent connections was very high. You may need to check this sysctl:vm.zone:
    Code:
    # sysctl vm.zone | grep "ITEM\|^pf"
    ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
    pfosfp:           28,        0,    208,    173,     2704
    pfospfen:        108,        0,    348,     84,     4524
    pfiaddrpl:        92,        0,      0,      0,        0
    pfstatescrub:     28,        0,      0,      0,        0
    pffrcent:         12,    50141,      0,      0,        0
    pffrcache:        48,    10062,      0,      0,        0
    pffrag:           48,        0,      0,    156,       10
    pffrent:          16,     5075,      0,    203,       18
    pfrkentry2:      156,        0,      0,      0,        0
    pfrkentry:       156,        0,     56,     94,      721
    pfrktable:      1240,        0,     19,     32,      338
    pfpooladdrpl:     68,        0,      6,    106,       71
    pfaltqpl:        128,        0,      0,      0,        0
    pfstatepl:       260,    10005,    629,    706,  8160740
    pfrulepl:        604,        0,     66,     72,      787
    pfsrctrpl:       100,    10023,      0,      0,        0
     
  4. chrcol

    chrcol New Member

    Messages:
    347
    Thanks Received:
    9
    nice command, here is the output.

    Code:
    ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
    pfosfp:           28,        0,    208,    300,    13312
    pfospfen:        108,        0,    348,    156,    22272
    pfiaddrpl:        92,        0,      0,      0,        0
    pfstatescrub:     28,        0,   1273,   6093,  6618612
    pffrcent:         12,    50141,      0,      0,        0
    pffrcache:        48,    10062,      0,      0,        0
    pffrag:           48,        0,      0,    390,   354203
    pffrent:          16,     5075,      0,    406,   707530
    pfrkentry2:      156,        0,      3,     47,        7
    pfrkentry:       156,        0,      0,   1775,     3032
    pfrktable:      1240,        0,      1,     32,     1770
    pfpooladdrpl:     68,        0,      1,    167,       32
    pfaltqpl:        128,        0,      0,      0,        0
    pfstatepl:       260,    10005,   1055,   2155, 138451258
    pfrulepl:        604,        0,    436,    566,    12381
    pfsrctrpl:       100,    10023,    127,    458,   199942
    


    incidently PF has not been flushed for some weeks now, check the amount of searches per second reported.

    State Table Total Rate
    current entries 1070
    searches 648808577 173.0/s

    I have had a pm here of someone with the same issue who thinks it may be a bug, I am starting to wonder the same thing. Still getting dropped connections that dont trigger any logging in the PF log.
     
  5. ecazamir

    ecazamir New Member

    Messages:
    215
    Thanks Received:
    27
    The only thing I may suggest is use of 'pftop', which will show you a lot of things related to pf. From the command output, it seems that your memory alloccation is fine (approx. 1000 states out of approx. 10000).