nat with pf not working

Ben

Ben

Hi,

I have a server at OVH with FreeBSD 8.1 installed. From the host internet works fine.

10.0.0.1: IP of the Jail

I have the following pf.conf:
Code: 
nat on em0 proto {tcp udp icmp} from 10.0.0.1 to any -> $main_ip

pfctl -s nat says:
Code: 
nat on em0 inet proto tcp from 10.0.0.1 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto udp from 10.0.0.1 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto icmp from 10.0.0.1 to any -> xxx.xxx.xxx.xxx
rdr on em0 inet proto tcp from any to 188.165.236.45 port = http -> 10.0.0.1
rdr on em0 inet proto tcp from any to xxx.xxx.xxx.xxx port = http -> 10.0.0.1

When I try to connect from inside the jail I get a timeout which means the packets are going out but the response seems not to be redirected back into the jail.

I activated ping in jails (sysctl for raw packets) but nothing works, neither the hostname nor the ip directly.

Does anybody have a hint for me what I can do?

I compared with other servers (not ovh) but couldn't find the mistake.

Hopefully somebody can help.

Thanks in advance,
Ben
 
Please post the relevant bits from /etc/rc.conf. Jail config, jail interface etc.

Also host configuration, IP address, routing table etc.
 
rc.conf of the host:
Code: 
defaultrouter="xxx.xxx.xxx.254"
hostname="myhost.domain.com"
gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
ifconfig_em0="inet xxx.xxx.xxx.xxx netmask 255.255.255.0 broadcast xxx.xxx.xxx.255"
ifconfig_em0_alias0="10.0.0.1/32"

I'm using ezjail so this is my ezjail-config:
Code: 
ezjail_jaildir=/usr/jails
ezjail_jailtemplate=${ezjail_jaildir}/newjail
ezjail_jailbase=${ezjail_jaildir}/basejail
ezjail_sourcetree=/usr/src
ezjail_uglyperlhack="YES"
ezjail_mount_enable="YES"
ezjail_devfs_enable="YES"
ezjail_devfs_ruleset="devfsrules_jail"
ezjail_procfs_enable="YES"
ezjail_fdescfs_enable="YES"

From the Jail-Config:
Code: 
export jail_mysql_hostname="mysql"
export jail_mysql_ip="10.0.0.1"
export jail_mysql_rootdir="/usr/jails/mysql"
export jail_mysql_exec_start="/bin/sh /etc/rc"
export jail_mysql_exec_stop=""
export jail_mysql_mount_enable="YES"
export jail_mysql_devfs_enable="YES"
export jail_mysql_devfs_ruleset="devfsrules_jail"
export jail_mysql_procfs_enable="YES"
export jail_mysql_fdescfs_enable="YES"
export jail_mysql_image=""
export jail_mysql_imagetype=""
export jail_mysql_attachparams=""
export jail_mysql_attachblocking=""
export jail_mysql_forceblocking=""
export jail_mysql_zfs_datasets=""
export jail_mysql_cpuset=""
export jail_mysql_fib=""

And the routing table:
Code: 
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            vss-3-6k.fr.eu     UGS        11     7716    em0
10.0.0.1           link#1             UHS         0        0    lo0 =>
10.0.0.1/32        link#1             U           0        0    em0
localhost          link#3             UH          0       92    lo0
xxx.xxx.xxx.0      link#1             U           3     1410    em0
xxx.ovh.net   link#1             UHS         0        4    lo0

Internet6:
Destination        Gateway            Flags      Netif Expire
localhost          localhost          UH          lo0
fe80::%lo0         link#3             U           lo0
fe80::1%lo0        link#3             UHS         lo0
ff01:3::           fe80::1%lo0        U           lo0
ff02::%lo0         fe80::1%lo0        U           lo0
 
Bind your jail to a clone of lo0. There's no traversal of network interfaces. The 10.0.0.1 address is on the em0 interface itself. So it's supposed to change a source address on it's own interface. I'm doubtful NAT works like that.

Either assign an IP address in the same range as the host or bind the jail to lo1.
 
On other servers it works fine. This should not be the issue.

You can use NAT on the same interface without problem.

Anyway, I might give it a try.
 
I tried but it's not working either.

Something is wrong but I don't know how I can debug any further.

I will try to rebuild my pf.conf from the beginning but I don't have too much hope.

I really would appreciate any hint as it's getting urgent.

Thanks for help!
 
Hello,

Please, paste the whole pf.conf.

Fot debug you can try to ping from inside the jail some IP address in the internet (for example some of google).
When you perform ping view your firewall states:
Code: 
pfctl -ss | grep 10.0.0.1 | grep icmp
and then tcpdump:
Code: 
tcpdump -i em0 host IP_address_which_you_ping
- you can set some verbose options here.
 
Hi,

My pf.conf

Code: 
lan1 = "{ em0 }"
lans = "{ em0 em1 }"

### TABLES ###
table <bruteforce> persist
table <rfc1918> persist { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0/5 }


### PORTS Jail 1: MySQL ###
# j1_tcp = "{ }"
# j1_udp = "{ }"

### PORTS Jail 2: WWW ###
j2_tcp = "{ 80 }"
j2_udp = "{ 80 }"

icmp_types = "echoreq"

open_tcp = "{ 80 6675 }"
open_udp = "{ 80 6675 }"


### Jail-IPs ###
jails = "{ 10.0.0.0/32 }"
main_ip = xxx.xxx.xxx.xxx

mysql = 10.0.0.1
www = 10.0.0.2


### SETTINGS ###
set block-policy drop
set skip on lo0
set timeout { interval 10, frag 30 }
set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
set timeout { icmp.first 20, icmp.error 10 }
set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 0, adaptive.end 0 }
set limit { states 10000, frags 5000 }
set loginterface em0
set optimization normal
set require-order yes
set fingerprints "/etc/pf.os"
set ruleset-optimization basic

scrub in all fragment reassemble random-id

### RDR ###
# Jail 1: MYSQL #
#rdr on $lan1 proto tcp from any to $lan1 port $j1_tcp -> $mysql
#rdr on $lan1 proto udp from any to $lan1 port $j1_udp -> $mysql

# Jail 2: WWW #
rdr on $lan1 proto tcp from any to $lan1 port $j2_tcp -> $www
rdr on $lan1 proto udp from any to $lan1 port $j2_udp -> $www

### NAT ###
nat on $lan1 proto {tcp udp icmp} from $jails to any -> $main_ip

### RULES ###
block log all
block return
block quick from <bruteforce>
block in quick on $lan1 inet from <rfc1918> to any
antispoof quick for $lan1

### NMAP ERSCHWEREN ###
block in log quick on $lan1 inet proto tcp from any to any flags FUP/FUP
block in log quick on $lan1 inet proto tcp from any to any flags SF/SFRA
block in log quick on $lan1 inet proto tcp from any to any flags /SFRA
block in log quick on $lan1 os NMAP

# IN #
pass in on $lan1 proto tcp from any to any port $open_tcp flags S/SAFR synproxy state (source-track rule, max-src-conn 300, \
max-src-conn-rate 75/2, overload <bruteforce> flush global, if-bound)
pass in on $lan1 proto udp from any to any port $open_udp keep state

# OUT #
pass out quick all keep state

# PING #
pass in on $lan1 inet proto icmp all icmp-type $icmp_types keep state
# TRACEROUTE #
pass in on $lan1 inet proto udp from any to any port 33433 >< 33626 keep state

The tcpdump and pfctl I will check.

Thanks a lot!!
 
pfctl:
Nothing, just
Code: 
No ALTQ support in kernel
ALTQ related functions disabled

tcpdump:
Code: 
07:32:02.295311 IP 10.0.0.1 > cdns.ovh.net: ICMP echo request, id 853, seq 4, length 64

So the request is going out but 10.0.0.1 is not translated?
 
Code: 
jails = "{ 10.0.0.0/32 }"

This is wrong, it defines jails to be a single address 10.0.0.0 and your nat rule never matches any traffic. You probably want it to be:

Code: 
jails = "{ 10.0.0.0/24 }"
 
No, it doesn't work.

I modified the nat rule to
Code: 
nat on em0 proto {tcp udp icmp} from 10.0.0.2 to any -> em0
But still doesn't match.

Any idea?
 
Hello,

please post the output from:
Code: 
pfctl -sa

Also,
Code: 
nat on $lan1 proto {tcp udp icmp} from $jails to any -> $main_ip

should be replaced by

Code: 
nat on $lan1 proto {tcp udp icmp} from $mysql to any -> $main_ip
( 10.0.0.1 )

or to specify
Code: 
jails = "{ 10.0.0.1, 10.0.0.2 }"

or
Code: 
nat on $lan1 proto {tcp udp icmp} from { $mysql, $www } to any -> $main_ip

Also, try to restart PF, for me it seems not to passing out packets, but it will be clear when you provide pfctl -sa output:

Code: 
/etc/rc.d/pf restart
 
I recognized: After I restarted pf I could do a nslookup in both jails, but a second time would not work anymore. After I restart (not reload) it works again once, then not anymore.

pfctl -sa:

Code: 
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xxx.xxx.xxx.xxx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xxx.xxx.xxx.xxx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xxx.xxx.xxx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick on em0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_9c059d73_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 300, \
max-src-conn-rate 75/2, overload <bruteforce> flush global, if-bound, src.track 2)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 300, \
max-src-conn-rate 75/2, overload <bruteforce> flush global, if-bound, src.track 2)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xxx.xxx.xxx.xxx:80 <- 77.170.70.124:42179       FIN_WAIT_2:ESTABLISHED
...

SOURCE TRACKING NODES:
82.170.185.2 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/2s )
...

INFO:
Status: Disabled for 0 days 00:01:28          Debug: Urgent

Interface Stats for em0               IPv4             IPv6
  Bytes In                        10936443                0
  Bytes Out                       47085545                0
  Packets In
    Passed                          128882                0
    Blocked                           4541                0
  Packets Out
    Passed                          172977                0
    Blocked                           2260                0

State Table                          Total             Rate
  current entries                      562               
  searches                          308663         3507.5/s
  inserts                             1904           21.6/s
  removals                            1714           19.5/s
Counters
  match                               3138           35.7/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              1            0.0/s
  synproxy                            5556           63.1/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start                0 states
adaptive.end                  0 states
src.track                     0s

LIMITS:
states        hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   100000

TABLES:
__automatic_9c059d73_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded
 
Hello,

What is the behaviour with the following pf.conf?

Code: 
ext_if="em0"
my_ip="your_external_IP_address"
jail_ips = "{ 10.0.0.1, 10.0.0.2 }"
nat on $ext_if inet proto { tcp, udp, icmp } from $jail_ips to any -> $my_ip
pass all
 
I just wrote pass all in front of the block rules as I can't change too much as this is a live server.

I applied the other changes but no difference.

I reloaded the rules with
Code: 
pfctl -Fa -f /etc/pf.conf

It seems that the packets pass first after a fresh restart of pf and afterwards they get blocked.
 
Could it be related to my sysctl-settings:

Code: 
security.bsd.see_other_uids=0
net.inet.ip.check_interface=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.drop_synfin=1
 
The redirects are disabled for the moment.

My problem is the NAT. I can't access IPs from inside the jail, only once after pf-restart.
 
Ben said:
Could it be related to my sysctl-settings:

Code: 
security.bsd.see_other_uids=0
net.inet.ip.check_interface=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.drop_synfin=1
Click to expand...

Hello,

No.

The problem is not NAT related actually (you will see this by testing with the config I provided you), your whole config is a bit strange, like it is a copy/paste from somewhere and it is not exactly clear what rule what makes.

So, I suggested you to replace your current pf.conf with the one I provided you which passes evething, just doing simple NAT.

Actually, your jail network is matched by
Code: 
block drop in quick on em0 inet from <rfc1918> to any
and if you don't want to test with my config, try removing this line from your config and restarting PF. Also, please provide on each "non working" case [cmd=]pfctl -sa[/cmd] to be clear what is your current ruleset.
 
The rules are a set of rules which I use on several servers for several years.

Can you explain why the same config works on another server? (not em, sk and re)?

Same as before: After the restart of pf it works for a few seconds, then is blocked.
Code: 
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xxx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xxx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xxx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xxx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick inet from <__automatic_ae8b89f1_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xxx:22 <- 178.82.205.140:49463       ESTABLISHED:ESTABLISHED

SOURCE TRACKING NODES:
178.82.205.140 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Disabled for 0 days 00:04:42          Debug: Urgent

Interface Stats for em0               IPv4             IPv6
  Bytes In                        54780301              424
  Bytes Out                      163183405                0
  Packets In
    Passed                          248595                0
    Blocked                          13591                5
  Packets Out
    Passed                          317028                0
    Blocked                           6518                0

State Table                          Total             Rate
  current entries                      373               
  searches                          585750         2077.1/s
  inserts                             6301           22.3/s
  removals                            6197           22.0/s
Counters
  match                               7728           27.4/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            4            0.0/s
  state-mismatch                        19            0.1/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              3            0.0/s
  synproxy                           18458           65.5/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start                0 states
adaptive.end                  0 states
src.track                     0s

LIMITS:
states        hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   100000

TABLES:
__automatic_ae8b89f1_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded
 
Hello,

With years rules and defaults changes, because of that always rules have to be written from the beginning.
You can start building it with NAT + pass all simple config (for testing NAT), and then insert needed rules.

Can you provide tcpdump output from pflog interface when you perform ping from the jail?
 
tcpdump on pflog0 while I pinged 213.186.33.99 after a /etc/rc.d/pf restart. Ping worked
Code: 
tcpdump -i pflog0 -vv host 213.186.33.99
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 96 bytes

At that time pfctl -sa looked like this:
Code: 
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xx.xx.xx.xx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xx.xx.xx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick on em0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_e698b413_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xx.xx.xx.xx:80 <- 81.71.107.110:51038       FIN_WAIT_2:FIN_WAIT_2


SOURCE TRACKING NODES:
41.22.138.74 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Enabled for 0 days 00:01:49           Debug: Urgent

Interface Stats for em0               IPv4             IPv6
  Bytes In                         1556595              280
  Bytes Out                        7597272                0
  Packets In
    Passed                           16816                0
    Blocked                            822                3
  Packets Out
    Passed                           23366                0
    Blocked                            328                0

State Table                          Total             Rate
  current entries                      345               
  searches                           41341          379.3/s
  inserts                              384            3.5/s
  removals                             261            2.4/s
Counters
  match                                656            6.0/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                             881            8.1/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start                0 states
adaptive.end                  0 states
src.track                     0s

LIMITS:
states        hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   100000

TABLES:
__automatic_e698b413_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded

Then my ssh connection was killed (no reaction of the shell anymore, so no "you have been disconnected").

After a reconnect via ssh pfctl -sa looked like this:
Code: 
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on em0 inet proto tcp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto udp from 10.0.0.0/24 to any -> xx.xx.xx.xx
nat on em0 inet proto icmp from 10.0.0.0/24 to any -> xx.xx.xx.xx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! em0 inet from xx.xx.xx.0/24 to any
block drop in quick on ! em0 inet from 10.0.0.0/24 to any
block drop in quick on em0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_e698b413_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on em0 inet proto tcp all flags FPU/FPU
block drop in log quick on em0 inet proto tcp all flags FS/FSRA
block drop in log quick on em0 inet proto tcp all flags /FSRA
block drop in log quick on em0 from any os "NMAP" to any
pass in on em0 proto tcp from any to any port = http flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto tcp from any to any port = ssh flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on em0 proto udp from any to any port = http keep state
pass in on em0 proto udp from any to any port = ssh keep state
pass out quick all flags S/SA keep state
pass in on em0 inet proto icmp all icmp-type echoreq keep state
pass in on em0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
em0 tcp xx.xx.xx.xx:22 <- 41.22.138.74:43916       ESTABLISHED:ESTABLISHED

SOURCE TRACKING NODES:
41.22.138.74 -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Disabled for 0 days 00:00:34          Debug: Urgent

Interface Stats for em0               IPv4             IPv6
  Bytes In                         4295142              704
  Bytes Out                       20467420                0
  Packets In
    Passed                           46836                0
    Blocked                           1761                8
  Packets Out
    Passed                           69687                0
    Blocked                            698                0

State Table                          Total             Rate
  current entries                      166               
  searches                          118999         3500.0/s
  inserts                              834           24.5/s
  removals                             890           26.2/s
Counters
  match                               1309           38.5/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                            1992           58.6/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start                0 states
adaptive.end                  0 states
src.track                     0s

LIMITS:
states        hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   100000

TABLES:
__automatic_e698b413_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded
 
Ok, I set up a different server with exactly the same setup, just a different provider.

pf-rules are the same (main_ip changed and re0 instead of em0) and it works up to now.

Here is the pfctl -sa from the NEW server:
Code: 
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat on re0 inet proto tcp from 10.0.0.0/24 to any -> xxx
nat on re0 inet proto udp from 10.0.0.0/24 to any -> xxx
nat on re0 inet proto icmp from 10.0.0.0/24 to any -> xxx

FILTER RULES:
scrub in all random-id fragment reassemble
block drop log all
block return all
block drop in quick on ! re0 inet from xxx.0/24 to any
block drop in quick on ! re0 inet from 10.0.0.0/24 to any
block drop in quick on re0 inet from <rfc1918> to any
block drop in quick inet from <__automatic_d6354acf_0> to any
block drop quick from <bruteforce> to any
block drop in log quick on re0 inet proto tcp all flags FPU/FPU
block drop in log quick on re0 inet proto tcp all flags FS/FSRA
block drop in log quick on re0 inet proto tcp all flags /FSRA
block drop in log quick on re0 from any os "NMAP" to any
pass in on re0 proto tcp from any to any port = 22 flags S/FSRA synproxy state (source-track rule, max-src-conn 100, \
max-src-conn-rate 15/5, overload <bruteforce> flush global, if-bound, src.track 5)
pass in on re0 proto udp from any to any port = 22 keep state
pass out quick all flags S/SA keep state
pass in on re0 inet proto icmp all icmp-type echoreq keep state
pass in on re0 inet proto udp from any to any port 33433 >< 33626 keep state

STATES:
all udp xxx:49176 -> 78.46.108.116:123       MULTIPLE:MULTIPLE
all udp xxx:28834 -> 88.198.70.163:123       MULTIPLE:MULTIPLE
all udp xxx:34805 -> 85.214.230.247:123       MULTIPLE:MULTIPLE
all udp xxx:35913 -> 83.169.43.165:123       MULTIPLE:MULTIPLE
re0 tcp xxx:22 <- xx:46877       ESTABLISHED:ESTABLISHED
all udp 10.0.0.1:51113 -> 78.31.69.183:57397 -> 62.141.32.3:53       MULTIPLE:SINGLE

SOURCE TRACKING NODES:
xx -> 0.0.0.0 ( states 1, connections 1, rate 0.0/5s )

INFO:
Status: Enabled for 0 days 00:04:32           Debug: Urgent

Interface Stats for re0               IPv4             IPv6
  Bytes In                          192600                0
  Bytes Out                         184625                0
  Packets In
    Passed                            1771                0
    Blocked                            407                0
  Packets Out
    Passed                            1736                0
    Blocked                              1                0

State Table                          Total             Rate
  current entries                        6               
  searches                            3922           14.4/s
  inserts                               43            0.2/s
  removals                              37            0.1/s
Counters
  match                                448            1.6/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               3            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start                0 states
adaptive.end                  0 states
src.track                     0s

LIMITS:
states        hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   100000

TABLES:
__automatic_d6354acf_0
bruteforce
rfc1918

OS FINGERPRINTS:
696 fingerprints loaded

You have an answer for this? Could it be an OVH problem or hardware issue?

EDIT: I forgot to mention that in the above pfctl -sa I always shortened the lists of SOURCE TRACKING NODES and STATES as it was many IPs listed. If this information is necessary I can provide a long list with IPs. Sorry, I forgot to mention that. In the first one it was marked by the ...
 
No, it couldn't be issue in your provider (assuming there is no practice colocation provider configuring TTL limitation :D) or hardware issue.

Please, try to do first just NAT + pass rule and a lot more of tcpdump output.
 
Over night the server has been re-installed (for other reason) and I dared to use the same config as before.

For the moment it's working but I did not set the sysctl-parameters yet.

I will set them one by one and see if there is an effect. It's very strange.

UPDATE: sysctl-Parameters are set, everything still works.
 
Hopefully final statement:

Everything works as expected with the initial config.

I got a note from OVH that they replaced one harddisk before the re-install even smartd had not complained. I don't know if this was the problem (you said could not be) but I did not change anything else. So somehow it must be related. I don't know if FreeBSD was damaged during install.

Anyway, it still works (since 2.5 hours) and I hope it stays like that.

Thanks for all your input and hints!!!
 
You must log in or register to reply here.
Back
Top