PF + route-to + gif weird behavior

schmurfy

schmurfy

Hi,

I have what I consider a bug with pf and I feel like I just hit a wall preventing me from going further.

The server is running FreeBSD 8.2-RELEASE and connected to another FreeBSD host with a gif tunnel (the other host is not really important here except for the fact both are connected so I will not speak much about it), I have most of the things I want working with the configuration files below.

My problem is with port forwarding, I want any connection on the external interface on a specific address/port (10.11.12.212 and 80 here) to be forwarded to another host behind the gif tunnel (192.168.0.23), the rules below almost work but it fails, here is what happens:

- the packet enters via the external interface (em0) with destination 10.11.12.212 on port 80
- the rdr rule translates the packet with destination 192.168.0.23 and port 80
- the pass rule with route-to routes the packet to the gif1001 interface
- the rule "pass out log on $env1c1_tunnel tagged e1c1" allows the packet to exit via gif1001

In a wonderful world that is what would happen, but with my current configuration here's what tcpdump catches exiting via em0 (I added line breaks to keep things readable):

Code: 
IP 10.11.12.212 > 10.11.20.1:
IP 10.11.12.5.57151 > 192.168.0.23.80:
Flags [S], seq 1515274791, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 721395077 ecr 0,sackOK,[|tcp]> (ipip-proto-4)

In this packet the source address for the ipip header is wrong but I don't see anything wrong in my rules, it may be a simple mistake and I sure hope it is but it really looks like a bug somewhere.

Here is the relevant part of my rc.conf file:

Code: 
# Router
ifconfig_em0="inet 10.11.12.212/24"
defaultrouter="10.11.12.253"
gateway_enable="YES"

static_routes="gif_endpoint"
route_visp="10.11.20.1/32 10.11.12.213"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"

# IPIP tunnels
gif_interfaces="gif1001"

ifconfig_em0_alias0="inet 10.11.20.2/32"
ifconfig_em0_alias1="inet 192.168.254.1/32"
gifconfig_gif1001="10.11.20.2 10.11.20.1"
ifconfig_gif1001="inet 1.2.3.1 1.2.3.2 netmask 255.255.255.252"

And here is my pf.conf:

Code: 
phys_if = "em0"

env1c1_tunnel = "gif1001"
env1c1_tunnel_dst = "10.11.20.1"
env1c1_tunnel_src = "10.11.20.2"
env1c1_escape = "10.11.12.212"

set skip on lo0
set block-policy drop

scrub in on $phys_if

####
## NAT / RDR
####

nat on $phys_if tagged e1c1 -> $env1c1_escape
nat on $phys_if tagged e1c2 -> $env1c2_escape

# forwarded port (http)
rdr on $phys_if proto tcp from any to $env1c1_escape port 80 tag e1c1 -> 192.168.0.23 port 80


####
## FILTERS
####

# block any packet with no match
block log all

# allow our own services to work
pass in on $phys_if proto tcp from any to $phys_if port ssh synproxy state
pass in on $phys_if proto icmp from any to $phys_if
pass out on $phys_if proto udp from any to any port { ntp, domain }


# allow ipip traffic (tunnels)
pass in on $phys_if from $env1c1_tunnel_dst to $env1c1_tunnel_src
pass out on $phys_if from $env1c1_tunnel_src to $env1c1_tunnel_dst

pass in on $phys_if from $env1c2_tunnel_dst to $env1c2_tunnel_src
pass out on $phys_if from $env1c2_tunnel_src to $env1c2_tunnel_dst

# Port forwarding
pass in log on $phys_if route-to ( $env1c1_tunnel  1.2.3.2 ) proto tcp from any to 192.168.0.23 port 80 tagged e1c1
pass out log on $env1c1_tunnel tagged e1c1

# NAT (the server's gateway will be used, traffic is returned to the tunnel)
pass in log on $env1c1_tunnel tag e1c1
pass out log on $phys_if reply-to $env1c1_tunnel tagged e1c1

pass in log on $env1c2_tunnel tag e1c2
pass out log on $phys_if reply-to $env1c2_tunnel tagged e1c2

# sites => sites
pass in on $env1c1_tunnel route-to $env1c1_tunnel to $env1c1_sites tag e1c1
pass in on $env1c2_tunnel route-to $env1c2_tunnel to $env1c2_sites tag e1c2

The only real resource I found on the internet about this would be a bug report on NetBSD: http://mail-index.netbsd.org/netbsd-bugs/2006/09/06/0000.html

I hope someone can help me on that :\
 
There's no need for route-to. Just add a static route for 192.168.0.0/24 and point it to the tunnel endpoint. Normal routing will take care of the rest.
 
Thanks for your answer but it appears I may have over simplified my configuration :P

My complete configuration will have many tunnels side by side (one per customer) and each of them can have any private network(s) on the other side (two customers could both use 192.168.0.0/24) of the tunnel so simply adding a static route is not really an option that is why I am trying with route-to.

I could use multiple routing table with setfib but with the limit of 16 routing table it is a door I'd rather keep closed to not introduce ridiculously low limits (it would limit us to 16 customers per server here...).

Each customer has a dedicated IP address on the server used for NAT and incoming forwarded connection, they need to be isolated from each other.
 
Yeah, that does change things a bit ;)

Not sure if it's going to work but you could setup a binat on the gif interface and map each customer to it's own 10.x.x.x/24 subnet. The route-to might get tricky if 2 customers have 192.168.0.23 to route to.
 
I just tested a similar ruleset on an OpenBSD 4.9 virtual machine and it works as I expected there. Looks like I will have to make a choice now.

Edit:
For reference, here is the OpenBSD 4.9 ruleset:

Code: 
phys_if = "em0"

env1c1_tunnel = "gif0"
env1c1_tunnel_dst = "192.168.1.2"
env1c1_tunnel_src = "192.168.1.1"
env1c1_escape = "192.168.0.33"

set skip on lo0
set block-policy drop


####
## FILTERS
####

block in quick on $phys_if proto udp from any to any port { 17500 }

# block any packet with no match
block log all

# allow our own services to work
pass in on $phys_if proto tcp from any to $phys_if port ssh synproxy state
pass in on $phys_if inet proto icmp from any to $phys_if
pass out on $phys_if proto udp from any to any port { ntp, domain }


# allow ipip traffic (tunnels)
pass in on $phys_if from $env1c1_tunnel_dst to $env1c1_tunnel_src
pass out on $phys_if from $env1c1_tunnel_src to $env1c1_tunnel_dst

# Port forwarding
pass in log on $phys_if proto tcp from any to $env1c1_escape port 80 tag "e1c1" route-to ( $env1c1_tunnel  169.254.0.1 ) rdr-to 192.168.3.1
pass out log on $env1c1_tunnel tagged e1c1

The only real difference with the FreeBSD version is that with latest pf the rdr rule no longer exists and was merged with the pass rule, which makes things much cleaner by the way.
 
Try using the same config on FreeBSD. As far as I know the pass rdr should work on the FreeBSD version of PF too.
 
In the FreeBSD version what is supported is to replace this:

Code: 
rdr on $phys_if proto tcp from any to $env1c1_escape port 80 tag e1c1 -> 192.168.0.23 port 80
pass in log on $phys_if route-to ( $env1c1_tunnel  1.2.3.2 ) proto tcp from any to 192.168.0.23 port 80 tagged e1c1

by this:

Code: 
rdr pass log on $phys_if proto tcp from any to $env1c1_escape port 80 tag e1c1 -> 192.168.0.23 port 80

but the route-to option can only be used in a pass rule not in a rdr rule so I cannot really test it :(

I would really prefer to stick to FreeBSD since my experience with OpenBSD is not so great until now: I have troubles compiling the tools I use currently on FreeBSD/Linux, the nice thing however is that the install is really nice, no horrible and useless ncurses GUI, just less than 10 questions in console mode with default answers and you are good to go! I never installed an operating system so fast.

Is there anyone who could confirm that I hit a bug so I can fill a report? I just checked and none of the reported bugs on pf match my problem.
 
You can always open a PR but I'm not sure if it's really a bug or not.

Perhaps it's best if you posted your problem to the freebsd-pf@ mailinglist. There aren't a lot of developers on this forum but they should be on the mailinglist.
 
I eventually gave up on FreeBSD.
I just turned toward OpenBSD for this machine since my ruleset works as expected there and I seem to be the only one to consider the FreeBSD behavior as a bug (I posted on the freebsd-pf mailing list with similar result).
 
You must log in or register to reply here.
Back
Top