PF; rdr rules; unable to keep the connection

_martin

_martin

I've described the similar problem here already: http://forums.freebsd.org/viewtopic.php?f=7&t=44953 . I did suggest the workaround too.

Recently I shared somewhat big files (~30GB) over http. FreeBSD 10/PF/jails/apache22 under the hood. I encountered the same problem as in VPN connection described above - download was aborted prematurely. When I looked at traffic with tcpdump I saw incoming RST packet from the FreeBSD server just before connection was dropped. Web server itself is working (meaning semi-static html contents is displayed correctly).
I had to move apache out of jail (to avoid using rdr in PF) to make it all work.

I see this behavior in FreeBSD 10. Have any of you encountered the similar problem ?
Setup is the following:

ext_if - some public IP on external interface
int_if / loopback - internal or dummy interface

basic PF skeleton:

Code: 
# LOCAL_NET - local net on dummy/internal iface
# IP_PUB - nat-to IP (on ext_if)
# PORT_WWW -  port to rdr
# IP_JAIL_WEBSERVER  - IP from LOCAL_NET

nat pass on $ext_if from $LOCAL_NET to any -> $IP_PUB
rdr pass on $ext_if proto tcp to $IP_PUB port $PORT_WWW -> $IP_JAIL_WEBSERVER

pass in all
pass out all
As it is vacation period I didn't have much time to do deeper tests on my machine. I will try to do deeper tests before submitting PR. I am, however, curious if somebody else did encounter the same problem.
 
I can't remember where I read it but it might be that the rdr pass construction doesn't imply keep-state on the firewall rule that it creates like you would expect. Try moving the firewalling of the NAT'ed connections to separate rules.
 
To my understanding keep-state means to keep the status of the connection in a status table. This is mainly due to performance and a security. The way I see it it should not matter if I do have keep-state or not (in the sense that session should not be closed prematurely in either setup). Sure I can avoid nat pass and pass the traffic in filtering section; I'll try to do it during my tests.
 
@kpa Hm, so I did some tests today. It seems that nat pass is OK, but rdr pass is a trouble here. I cut the pf.conf to the very basic as following (public IP changed here):

Code: 
ext_if="em0"

IP_PUB="192.0.2.1"
IP_JAIL_SANDBOX="192.168.252.1"
NET_LO252="192.168.252.0/24"
PORT_JAIL_SANDBOX_SSH="5222"

set skip on lo0
scrub in all

nat pass on $ext_if from $NET_LO252 to any -> $IP_PUB
rdr pass on $ext_if proto tcp to $IP_PUB port $PORT_JAIL_SANDBOX_SSH -> $IP_JAIL_SANDBOX port 22

block in all
pass out all

# (XXX: keep the door open!)
pass in on $ext_if proto tcp from any to $IP_PUB port 22
I have a FreeBSD machine running a jail which I access from outside to a custom port. Local address is bind to a custom loopback interface. Config above does NOT work (i.e. RST is sent at random) on my machine (10.0-RELEASE-p6 GENERIC). However, if I do a slight modification:

Code: 
ext_if="em0"

IP_PUB="192.0.2.1"
IP_JAIL_SANDBOX="192.168.252.1"
NET_LO252="192.168.252.0/24"
PORT_JAIL_SANDBOX_SSH="5222"

set skip on lo0
scrub in all

nat pass on $ext_if from $NET_LO252 to any -> $IP_PUB
rdr on $ext_if proto tcp to $IP_PUB port $PORT_JAIL_SANDBOX_SSH -> $IP_JAIL_SANDBOX port 22

block in all
pass out all

# pass sandbox rdr to IP_JAIL_SANDBOX
pass in quick proto tcp from any to $IP_JAIL_SANDBOX port 22

# (XXX: keep the door open!)
pass in on $ext_if proto tcp from any to $IP_PUB port 22
Now the connection is working perfectly.

To make it worse though, I can't reproduce this behavior in VM. Right now the main difference is that my VM was on the same network as a machine I was doing tests from (but should that matter ? ) opposed to my server which I'm reaching behind the gateway.
 
Last edited by a moderator:
Hi all,

I can confirm that today we had the same problem with the "rdr pass" rules in our pf.conf/FreeBSD 10.1 STABLE
uname -K
Code: 
1001510
uname -U
Code: 
1001510
I can also confirm that in FreeBSD 9 STABLE these rules just work perfectly!
 
devil_devil The sad thing is that I tried to reproduce this issue in 3 different ways:

  • 0) in VM
  • 1) on a totally different physical HW
  • 2) on a same type of a HW (server) where problem occurred, just different node

Neither of these scenarios helped - issue didn't occur. 0) 1) examples would make me think OK, it's something more HW-related, maybe Intel's driver ? Hard to say. But when I tested the 2) scenario I was really bumped. True that uptime was a bit small (a day or so), but this issue does occur on my HW even after reboot, maybe few minutes (traffic) later.

My network adapter is Intel 82579LM on a S1200BT motherboard.
 
You must log in or register to reply here.
Back
Top