[Solved] nat with pf not working [Archive]

Ben

January 17th, 2011, 10:50

Hi,

I have a server at OVH with FreeBSD 8.1 installed. From the host internet works fine.

10.0.0.1: IP of the Jail

I have the following pf.conf:
nat on em0 proto {tcp udp icmp} from 10.0.0.1 to any -> $main_ip

pfctl -s nat says:
nat on em0 inet proto tcp from 10.0.0.1 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto udp from 10.0.0.1 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto icmp from 10.0.0.1 to any -> xxx.xxx.xxx.xxx
rdr on em0 inet proto tcp from any to 188.165.236.45 port = http -> 10.0.0.1
rdr on em0 inet proto tcp from any to xxx.xxx.xxx.xxx port = http -> 10.0.0.1

When I try to connect from inside the jail I get a timeout which means the packets are going out but the response seems not to be redirected back into the jail.

I activated ping in jails (sysctl for raw packets) but nothing works, neither the hostname nor the ip directly.

Does anybody have a hint for me what I can do?

I compared with other servers (not ovh) but couldn't find the mistake.

Hopefully somebody can help.

Thanks in advance,
Ben

SirDice

January 17th, 2011, 10:54

Please post the relevant bits from /etc/rc.conf. Jail config, jail interface etc.

Also host configuration, IP address, routing table etc.

Ben

January 17th, 2011, 10:59

rc.conf of the host:
defaultrouter="xxx.xxx.xxx.254"
hostname="myhost.domain.com"
gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
ifconfig_em0="inet xxx.xxx.xxx.xxx netmask 255.255.255.0 broadcast xxx.xxx.xxx.255"
ifconfig_em0_alias0="10.0.0.1/32"

I'm using ezjail so this is my ezjail-config:
ezjail_jaildir=/usr/jails
ezjail_jailtemplate=${ezjail_jaildir}/newjail
ezjail_jailbase=${ezjail_jaildir}/basejail
ezjail_sourcetree=/usr/src
ezjail_uglyperlhack="YES"
ezjail_mount_enable="YES"
ezjail_devfs_enable="YES"
ezjail_devfs_ruleset="devfsrules_jail"
ezjail_procfs_enable="YES"
ezjail_fdescfs_enable="YES"

From the Jail-Config:

export jail_mysql_hostname="mysql"
export jail_mysql_ip="10.0.0.1"
export jail_mysql_rootdir="/usr/jails/mysql"
export jail_mysql_exec_start="/bin/sh /etc/rc"
export jail_mysql_exec_stop=""
export jail_mysql_mount_enable="YES"
export jail_mysql_devfs_enable="YES"
export jail_mysql_devfs_ruleset="devfsrules_jail"
export jail_mysql_procfs_enable="YES"
export jail_mysql_fdescfs_enable="YES"
export jail_mysql_image=""
export jail_mysql_imagetype=""
export jail_mysql_attachparams=""
export jail_mysql_attachblocking=""
export jail_mysql_forceblocking=""
export jail_mysql_zfs_datasets=""
export jail_mysql_cpuset=""
export jail_mysql_fib=""

And the routing table:

Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default vss-3-6k.fr.eu UGS 11 7716 em0
10.0.0.1 link#1 UHS 0 0 lo0 =>
10.0.0.1/32 link#1 U 0 0 em0
localhost link#3 UH 0 92 lo0
xxx.xxx.xxx.0 link#1 U 3 1410 em0
xxx.ovh.net link#1 UHS 0 4 lo0

Internet6:
Destination Gateway Flags Netif Expire
localhost localhost UH lo0
fe80::%lo0 link#3 U lo0
fe80::1%lo0 link#3 UHS lo0
ff01:3:: fe80::1%lo0 U lo0
ff02::%lo0 fe80::1%lo0 U lo0

SirDice

January 17th, 2011, 11:07

Bind your jail to a clone of lo0. There's no traversal of network interfaces. The 10.0.0.1 address is on the em0 interface itself. So it's supposed to change a source address on it's own interface. I'm doubtful NAT works like that.

Either assign an IP address in the same range as the host or bind the jail to lo1.

Ben

January 17th, 2011, 11:09

On other servers it works fine. This should not be the issue.

You can use NAT on the same interface without problem.

Anyway, I might give it a try.

Ben

January 17th, 2011, 12:52

I tried but it's not working either.

Something is wrong but I don't know how I can debug any further.

I will try to rebuild my pf.conf from the beginning but I don't have too much hope.

I really would appreciate any hint as it's getting urgent.

Thanks for help!

quintessence

January 17th, 2011, 23:09

Hello,

Please, paste the whole pf.conf.

Fot debug you can try to ping from inside the jail some IP address in the internet (for example some of google).
When you perform ping view your firewall states:
pfctl -ss | grep 10.0.0.1 | grep icmp
and then tcpdump:
tcpdump -i em0 host IP_address_which_you_ping - you can set some verbose options here.

Ben

January 18th, 2011, 07:27

Hi,

My pf.conf

lan1 = "{ em0 }"
lans = "{ em0 em1 }"

### TABLES ###
table <bruteforce> persist
table <rfc1918> persist { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0/5 }

### PORTS Jail 1: MySQL ###
# j1_tcp = "{ }"
# j1_udp = "{ }"

### PORTS Jail 2: WWW ###
j2_tcp = "{ 80 }"
j2_udp = "{ 80 }"

icmp_types = "echoreq"

open_tcp = "{ 80 6675 }"
open_udp = "{ 80 6675 }"

### Jail-IPs ###
jails = "{ 10.0.0.0/32 }"
main_ip = xxx.xxx.xxx.xxx

mysql = 10.0.0.1
www = 10.0.0.2

### SETTINGS ###
set block-policy drop
set skip on lo0
set timeout { interval 10, frag 30 }
set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
set timeout { icmp.first 20, icmp.error 10 }
set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 0, adaptive.end 0 }
set limit { states 10000, frags 5000 }
set loginterface em0
set optimization normal
set require-order yes
set fingerprints "/etc/pf.os"
set ruleset-optimization basic

scrub in all fragment reassemble random-id

### RDR ###
# Jail 1: MYSQL #
#rdr on $lan1 proto tcp from any to $lan1 port $j1_tcp -> $mysql
#rdr on $lan1 proto udp from any to $lan1 port $j1_udp -> $mysql

# Jail 2: WWW #
rdr on $lan1 proto tcp from any to $lan1 port $j2_tcp -> $www
rdr on $lan1 proto udp from any to $lan1 port $j2_udp -> $www

### NAT ###
nat on $lan1 proto {tcp udp icmp} from $jails to any -> $main_ip

### RULES ###
block log all
block return
block quick from <bruteforce>
block in quick on $lan1 inet from <rfc1918> to any
antispoof quick for $lan1

### NMAP ERSCHWEREN ###
block in log quick on $lan1 inet proto tcp from any to any flags FUP/FUP
block in log quick on $lan1 inet proto tcp from any to any flags SF/SFRA
block in log quick on $lan1 inet proto tcp from any to any flags /SFRA
block in log quick on $lan1 os NMAP

# IN #
pass in on $lan1 proto tcp from any to any port $open_tcp flags S/SAFR synproxy state (source-track rule, max-src-conn 300, \
max-src-conn-rate 75/2, overload <bruteforce> flush global, if-bound)
pass in on $lan1 proto udp from any to any port $open_udp keep state

# OUT #
pass out quick all keep state

# PING #
pass in on $lan1 inet proto icmp all icmp-type $icmp_types keep state
# TRACEROUTE #
pass in on $lan1 inet proto udp from any to any port 33433 >< 33626 keep state

The tcpdump and pfctl I will check.

Thanks a lot!!

Ben

January 18th, 2011, 07:33

pfctl:
Nothing, just
No ALTQ support in kernel
ALTQ related functions disabled

tcpdump:
07:32:02.295311 IP 10.0.0.1 > cdns.ovh.net: ICMP echo request, id 853, seq 4, length 64

So the request is going out but 10.0.0.1 is not translated?

kpa

January 18th, 2011, 07:51

jails = "{ 10.0.0.0/32 }"

This is wrong, it defines jails to be a single address 10.0.0.0 and your nat rule never matches any traffic. You probably want it to be:

jails = "{ 10.0.0.0/24 }"

Ben

January 18th, 2011, 09:26

No, it doesn't work.

I modified the nat rule to
nat on em0 proto {tcp udp icmp} from 10.0.0.2 to any -> em0
But still doesn't match.

Any idea?

quintessence

January 18th, 2011, 09:46

Hello,

please post the output from:
pfctl -sa

Also,
nat on $lan1 proto {tcp udp icmp} from $jails to any -> $main_ip

should be replaced by

nat on $lan1 proto {tcp udp icmp} from $mysql to any -> $main_ip
( 10.0.0.1 )

or to specify
jails = "{ 10.0.0.1, 10.0.0.2 }"

or nat on $lan1 proto {tcp udp icmp} from { $mysql, $www } to any -> $main_ip

Also, try to restart PF, for me it seems not to passing out packets, but it will be clear when you provide pfctl -sa output:

/etc/rc.d/pf restart

Ben

January 18th, 2011, 09:53

I recognized: After I restarted pf I could do a nslookup in both jails, but a second time would not work anymore. After I restart (not reload) it works again once, then not anymore.

pfctl -sa:

No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xxx.xxx.xxx.xxx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xxx.xxx.xxx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick on em0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_9c059d73_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 300, \
max-src-conn-rate 75/2, overload <bruteforce> flush global, if-bound, src.track 2)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 300, \
max-src-conn-rate 75/2, overload <bruteforce> flush global, if-bound, src.track 2)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xxx.xxx.xxx.xxx:80 <- 77.170.70.124:42179 FIN_WAIT_2:ESTABLISHED
...

SOURCE TRACKING NODES:
82.170.185.2 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/2s )
...

INFO:
Status: Disabled for 0 days 00:01:28 Debug: Urgent

Interface Stats for em0 IPv4 IPv6
Bytes In 10936443 0
Bytes Out 47085545 0
Packets In
Passed 128882 0
Blocked 4541 0
Packets Out
Passed 172977 0
Blocked 2260 0

State Table Total Rate
current entries 562
searches 308663 3507.5/s
inserts 1904 21.6/s
removals 1714 19.5/s
Counters
match 3138 35.7/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 1 0.0/s
synproxy 5556 63.1/s

TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 0 states
adaptive.end 0 states
src.track 0s

LIMITS:
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 100000

TABLES:
__automatic_9c059d73_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded

quintessence

January 18th, 2011, 10:17

Hello,

What is the behaviour with the following pf.conf?

ext_if="em0"
my_ip="your_external_IP_address"
jail_ips = "{ 10.0.0.1, 10.0.0.2 }"
nat on $ext_if inet proto { tcp, udp, icmp } from $jail_ips to any -> $my_ip
pass all

Ben

January 18th, 2011, 10:25

I just wrote pass all in front of the block rules as I can't change too much as this is a live server.

I applied the other changes but no difference.

I reloaded the rules with
pfctl -Fa -f /etc/pf.conf

It seems that the packets pass first after a fresh restart of pf and afterwards they get blocked.

Ben

January 18th, 2011, 12:47

Could it be related to my sysctl-settings:

security.bsd.see_other_uids=0
net.inet.ip.check_interface=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.drop_synfin=1

gpatrick

January 18th, 2011, 15:32

nat should be placed before redirects.

And change:

rdr on $lan1 proto tcp from any to $lan1 port $j2_tcp -> $www

to

rdr on $lan1 proto tcp from any to $main_ip port $j2_tcp -> $www

Ben

January 18th, 2011, 15:53

The redirects are disabled for the moment.

My problem is the NAT. I can't access IPs from inside the jail, only once after pf-restart.

quintessence

January 18th, 2011, 20:56

Could it be related to my sysctl-settings:

security.bsd.see_other_uids=0
net.inet.ip.check_interface=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.drop_synfin=1

Hello,

No.

The problem is not NAT related actually (you will see this by testing with the config I provided you), your whole config is a bit strange, like it is a copy/paste from somewhere and it is not exactly clear what rule what makes.

So, I suggested you to replace your current pf.conf with the one I provided you which passes evething, just doing simple NAT.

Actually, your jail network is matched by block drop in quick on em0 inet from <rfc1918> to any and if you don't want to test with my config, try removing this line from your config and restarting PF. Also, please provide on each "non working" case pfctl -sa to be clear what is your current ruleset.

Ben

January 18th, 2011, 21:11

The rules are a set of rules which I use on several servers for several years.

Can you explain why the same config works on another server? (not em, sk and re)?

Same as before: After the restart of pf it works for a few seconds, then is blocked.
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xxx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xxx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xxx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xxx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick inet from <__automatic_ae8b89f1_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xxx:22 <- 178.82.205.140:49463 ESTABLISHED:ESTABLISHED

SOURCE TRACKING NODES:
178.82.205.140 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Disabled for 0 days 00:04:42 Debug: Urgent

Interface Stats for em0 IPv4 IPv6
Bytes In 54780301 424
Bytes Out 163183405 0
Packets In
Passed 248595 0
Blocked 13591 5
Packets Out
Passed 317028 0
Blocked 6518 0

State Table Total Rate
current entries 373
searches 585750 2077.1/s
inserts 6301 22.3/s
removals 6197 22.0/s
Counters
match 7728 27.4/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 4 0.0/s
state-mismatch 19 0.1/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 3 0.0/s
synproxy 18458 65.5/s

TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 0 states
adaptive.end 0 states
src.track 0s

LIMITS:
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 100000

TABLES:
__automatic_ae8b89f1_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded

quintessence

January 19th, 2011, 09:47

Hello,

With years rules and defaults changes, because of that always rules have to be written from the beginning.
You can start building it with NAT + pass all simple config (for testing NAT), and then insert needed rules.

Can you provide tcpdump output from pflog interface when you perform ping from the jail?

Ben

January 19th, 2011, 10:04

tcpdump on pflog0 while I pinged 213.186.33.99 after a /etc/rc.d/pf restart. Ping worked
tcpdump -i pflog0 -vv host 213.186.33.99
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 96 bytes

At that time pfctl -sa looked like this:
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xx.xx.xx.xx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xx.xx.xx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick on em0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_e698b413_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xx.xx.xx.xx:80 <- 81.71.107.110:51038 FIN_WAIT_2:FIN_WAIT_2

SOURCE TRACKING NODES:
41.22.138.74 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Enabled for 0 days 00:01:49 Debug: Urgent

Interface Stats for em0 IPv4 IPv6
Bytes In 1556595 280
Bytes Out 7597272 0
Packets In
Passed 16816 0
Blocked 822 3
Packets Out
Passed 23366 0
Blocked 328 0

State Table Total Rate
current entries 345
searches 41341 379.3/s
inserts 384 3.5/s
removals 261 2.4/s
Counters
match 656 6.0/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 881 8.1/s

TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 0 states
adaptive.end 0 states
src.track 0s

LIMITS:
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 100000

TABLES:
__automatic_e698b413_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded

Then my ssh connection was killed (no reaction of the shell anymore, so no "you have been disconnected").

After a reconnect via ssh pfctl -sa looked like this:
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xx.xx.xx.xx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xx.xx.xx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick on em0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_e698b413_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xx.xx.xx.xx:22 <- 41.22.138.74:43916 ESTABLISHED:ESTABLISHED

SOURCE TRACKING NODES:
41.22.138.74 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Disabled for 0 days 00:00:34 Debug: Urgent

Interface Stats for em0 IPv4 IPv6
Bytes In 4295142 704
Bytes Out 20467420 0
Packets In
Passed 46836 0
Blocked 1761 8
Packets Out
Passed 69687 0
Blocked 698 0

State Table Total Rate
current entries 166
searches 118999 3500.0/s
inserts 834 24.5/s
removals 890 26.2/s
Counters
match 1309 38.5/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 1992 58.6/s

TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 0 states
adaptive.end 0 states
src.track 0s

LIMITS:
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 100000

TABLES:
__automatic_e698b413_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded

Ben

January 19th, 2011, 17:29

Ok, I set up a different server with exactly the same setup, just a different provider.

pf-rules are the same (main_ip changed and re0 instead of em0) and it works up to now.

Here is the pfctl -sa from the NEW server:
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on re0 inet proto tcp from 10.0.0.0/24 to any -> xxx
nat on re0 inet proto udp from 10.0.0.0/24 to any -> xxx
nat on re0 inet proto icmp from 10.0.0.0/24 to any -> xxx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! re0 inet from xxx.0/24 to any
block drop in quick on ! re0 inet from 10.0.0.0/24 to any
block drop in quick on re0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_d6354acf_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on re0 inet proto tcp all flags FPU/FPU
block drop in log quick on re0 inet proto tcp all flags FS/FSRA
block drop in log quick on re0 inet proto tcp all flags /FSRA
block drop in log quick on re0 from any os "NMAP" to any
pass in on re0 proto tcp from any to any port = 22 flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on re0 proto udp from any to any port = 22 keep state
pass out quick all flags S/SA keep state
pass in on re0 inet proto icmp all icmp-type echoreq keep state
pass in on re0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
all udp xxx:49176 -> 78.46.108.116:123 MULTIPLE:MULTIPLE
all udp xxx:28834 -> 88.198.70.163:123 MULTIPLE:MULTIPLE
all udp xxx:34805 -> 85.214.230.247:123 MULTIPLE:MULTIPLE
all udp xxx:35913 -> 83.169.43.165:123 MULTIPLE:MULTIPLE
re0 tcp xxx:22 <- xx:46877 ESTABLISHED:ESTABLISHED
all udp 10.0.0.1:51113 -> 78.31.69.183:57397 -> 62.141.32.3:53 MULTIPLE:SINGLE

SOURCE TRACKING NODES:
xx -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Enabled for 0 days 00:04:32 Debug: Urgent

Interface Stats for re0 IPv4 IPv6
Bytes In 192600 0
Bytes Out 184625 0
Packets In
Passed 1771 0
Blocked 407 0
Packets Out
Passed 1736 0
Blocked 1 0

State Table Total Rate
current entries 6
searches 3922 14.4/s
inserts 43 0.2/s
removals 37 0.1/s
Counters
match 448 1.6/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 3 0.0/s

TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 0 states
adaptive.end 0 states
src.track 0s

LIMITS:
states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 100000

TABLES:
__automatic_d6354acf_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded

You have an answer for this? Could it be an OVH problem or hardware issue?

EDIT: I forgot to mention that in the above pfctl -sa I always shortened the lists of SOURCE TRACKING NODES and STATES as it was many IPs listed. If this information is necessary I can provide a long list with IPs. Sorry, I forgot to mention that. In the first one it was marked by the ...

quintessence

January 20th, 2011, 09:58

No, it couldn't be issue in your provider (assuming there is no practice colocation provider configuring TTL limitation :D) or hardware issue.

Please, try to do first just NAT + pass rule and a lot more of tcpdump output.

Ben

January 20th, 2011, 10:52

Over night the server has been re-installed (for other reason) and I dared to use the same config as before.

For the moment it's working but I did not set the sysctl-parameters yet.

I will set them one by one and see if there is an effect. It's very strange.

UPDATE: sysctl-Parameters are set, everything still works.

Ben

January 20th, 2011, 15:37

Hopefully final statement:

Everything works as expected with the initial config.

I got a note from OVH that they replaced one harddisk before the re-install even smartd had not complained. I don't know if this was the problem (you said could not be) but I did not change anything else. So somehow it must be related. I don't know if FreeBSD was damaged during install.

Anyway, it still works (since 2.5 hours) and I hope it stays like that.

Thanks for all your input and hints!!!