pf perfomance problem

[rghq]

Hello :)

Running 7.1 with 3 Jails and if pf is active, the bandwidth drops down - dunno if something is very wrong in the config.

1 Jail is for Mails (Dovecot + Postfix)
1 Jail for Apache + FTP
1 Jail for MySQL

At Apache a Webmail runs sending mails via Port 587 and SQL is used as backend - as well for Dovecot / Postfix.
Finally at 1 external mailhost mail is recieved via POP3

And some usual things like SSH and DKIM-Milter (port 2727) running on the host.
All the internal things on the lo interfaces are also for a monitoring system and due of some errors when running a tcpdump.
Finally - from a backup host a SSH tunnel is connecting to the DB for replica (port 31981)


interface = "fxp0"
interface_db = "lo1"
interface_mail = "lo2"
interface_www = "lo3"
high = "{ 1024:65535 }"
ports = "{ 2812 8025 47821 }"
rogueports = "{ 67 68 135 137 138 139 445 1024 1026 1027 1433 1434 4899 5900 7212 7288 9788 }"
icmptypes = "{ echorep, unreach, squench, echoreq }"

set block-policy return
set loginterface $interface
set limit states 10000
set limit frags 500

set skip on { lo0 }

table <dns> { 123.123.123.123 }
table <backs> { 125.125.125.125 }
table <badhosts> { 127.127.127.127 }
table <ntp> { 128.128.128.128 }
table <mailhosts> { 129.129.129.129 }

master_ip = "130.130.130.130"
second_ip = "130.130.130.131"
db_ip = "127.0.10.1"
mail_ip = "127.0.20.1"
www_ip = "127.0.30.1"

scrub in on $interface all fragment reassemble min-ttl 15 max-mss 1400
scrub in on $interface all no-df
scrub on $interface all reassemble tcp

rdr pass on $interface proto tcp from !<badhosts> to $master_ip port 80 -> $www_ip port 80
rdr pass on $interface proto tcp from any to $master_ip port 443 -> $www_ip port 443
rdr pass on $interface proto tcp from any to $master_ip port 990 -> $www_ip port 990
rdr pass on $interface proto tcp from any to $master_ip port 34700:35000 -> $www_ip port 34700:35000
rdr pass on $interface proto tcp from any to $master_ip port 25 -> $mail_ip port 25
rdr pass on $interface proto tcp from any to $second_ip port 25 -> $mail_ip port 25
rdr pass on $interface proto tcp from any to $master_ip port 465 -> $mail_ip port 465
rdr pass on $interface proto tcp from any to $master_ip port 993 -> $mail_ip port 993
rdr pass on $interface proto tcp from <backs> to $master_ip port 31981 -> $db_ip port 31981

nat on $interface from $db_ip to any -> $master_ip
nat on $interface from $mail_ip to any -> $master_ip
nat on $interface from $www_ip to any -> $master_ip

block in
block out

antispoof log quick for $interface inet

block in quick on $interface proto tcp from <badhosts> to any
block out quick on $interface proto tcp from any to <badhosts>

pass in quick inet proto icmp all icmp-type $icmptypes
pass in quick on $interface proto tcp from any to $master_ip port 22 flags S/SA keep state
pass in quick on $interface proto tcp from any port $high to $master_ip port $ports
pass in quick on $interface proto tcp from any port $high to $db_ip port 31981 flags S/SA keep state
pass in quick on $interface proto tcp from any port $high to $mail_ip port { 25 465 993 }
pass in quick on $interface proto tcp from any port $high to $www_ip port { 80 443 990 34700:35000 }
pass in quick on $interface proto tcp from <backs> to { $db_ip $master_ip }

pass out quick inet proto icmp all icmp-type $icmptypes
pass out quick on $interface proto tcp from $db_ip to <backs>
pass out quick on $interface proto tcp from $master_ip to <backs>
pass out quick on $interface proto tcp from $master_ip port $ports to any
pass out quick on $interface proto tcp from $master_ip port $high to <mailhosts> port { 110 995 }
pass out quick on $interface proto tcp from $mail_ip port { 25 465 993 } to any
pass out quick on $interface proto tcp from $proxy_ip to any port { 80 443 }
pass out quick on $interface proto tcp from $www_ip port { 443 990 34700:35000 } to any

pass out quick on $interface proto udp from $master_ip to <dns> port 53
pass out quick on $interface proto tcp from $master_ip to <dns> port 53
pass out quick on $interface proto udp from $master_ip to <ntp> port 123
pass out quick on $interface proto tcp from $master_ip to any port { 25 465 }
pass out quick on $interface proto tcp from $master_ip to any user root

pass in quick on $interface_mail proto tcp from $mail_ip to $mail_ip port { 25 465 587 993 2727 }
pass in quick on { $interface_mail $interface_www } proto tcp from { $mail_ip $www_ip } to { $mail_ip $www_ip } port { 443 587 993 }
pass in quick on { $interface_db $interface_mail $interface_www } proto tcp from { $db_ip $mail_ip $www_ip } to { $db_ip $mail_ip $www_ip } port { 80 3306 }

pass out quick on $interface_db proto tcp from $db_ip to $db_ip
pass out quick on $interface_mail proto tcp from $mail_ip to $mail_ip
pass out quick on { $interface_mail $interface_www } proto tcp from { $mail_ip $www_ip } to { $mail_ip $www_ip } port { 443 587 993 }
pass out quick on { $interface_db $interface_mail $interface_www } proto tcp from { $db_ip $mail_ip $www_ip } to { $db_ip $mail_ip $www_ip } port { 80 3306 }

block in quick on $interface proto tcp from any to $master_ip port $rogueports

block in log quick from any to any
block out log quick from any to any


Thanks in advance for pointing me to the right direction

[DutchDaemon]

Any specific type/direction of bandwidth that drops down, or is it in general? You're doing a lot of scrubbing. Maybe try a more generic

scrub all fragment reassemble

Oh, and your $proxy_ip is not defined.

[rghq]

Was a typo with proxy_ip - sorry about this.

Will try with the scrubbing - thanks :)

The bandwidth goes down generally - 50 KB/s if at all - doesn't matter then it the service is jailed or not like SSHd - doing a sftp transfer I get the 50 KB/s - without around 3 MB/s which is usual.

Though netstat / lsof report nothing unusual and a boot from CD via KVM with a check of the OS itself also reported no hidden rootkits.

Problem is that I use a very similar configuration at another machine - just another interface and different external IP's though there everything runs without any problems.

And if this matters, load is more than ok and there's enough free memory left.

EDIT: It's not a PF problem - oh - now the bad way begins searching where the problem may be :(

Can it be that fxp still has some problems ? I'm running 7.1p7 though - well - like said, bad network perfomance :(

Thanks for your help :)