Failed to redirect to jail outside host.

sixtydoses

sixtydoses

Hi,

I have this problem in redirecting traffic from a public IP (host) to an private IP (jail) on port 80. I have several jails, and it works fine if I were to redirect the traffic to a jail that resides on the same host. But if I try to redirect it to a jail outside of the host, it failed.

Both hosts are in the same subnet. Each server has 2 NICs, em0 and em1. em0 is assigned with public IPs, and em1 is assigned with private IPs. Each jail has two IPs, both public and private. I have tried redirecting to the jail's public IP as well but to no avail.


Rule in /etc/pf.conf (web01 is the host public IP, and wj01 is the jail):
Code: 
rdr pass log (all) on $ext_if inet proto tcp from any to $web01 port 80 -> $wj01

tcpdump output when I set the rule to rdr to local jail:
Code: 
00:00:00.327793 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.21396 > 192.168.0.21.80: Flags [.], ack 878, win 8280, options [nop,nop,TS val 4847531 ecr 2117623831], length 0
00:00:04.672817 rule 0..16777216/0(match): rdr out on em0: 176.9.77.72.80 > x.x.x.x.21396: Flags [F.], seq 878, ack 722, win 1237, options [nop,nop,TS val 2117628832 ecr 4847531], length 0
00:00:00.228523 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.21396 > 192.168.0.21.80: Flags [.], ack 879, win 8280, options [nop,nop,TS val 4852432 ecr 2117628832], length 0
00:00:00.009993 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.21396 > 192.168.0.21.80: Flags [F.], seq 721, ack 879, win 8280, options [nop,nop,TS val 4852442 ecr 2117628832], length 0
00:00:00.000015 rule 0..16777216/0(match): rdr out on em0: 176.9.77.72.80 > x.x.x.x.21396: Flags [.], ack 723, win 1237, options [nop,nop,TS val 2117629070 ecr 4852442], length 0
00:00:00.468363 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.46103 > 192.168.0.21.80: Flags [S], seq 51293176, win 65535, options [mss 1452,nop,wscale 3,sackOK,TS val 4852910 ecr 0], length 0


tcpdump output when I set the rule to rdr to external jail:
Code: 
00:00:00.000000 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.63811 > 192.168.0.30.80: Flags [S], seq 502839335, win 65535, options [mss 1452,nop,wscale 3,sackOK,TS val 5298641 ecr 0], length 0
00:00:00.256880 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.25358 > 192.168.0.30.80: Flags [S], seq 1736699979, win 65535, options [mss 1452,nop,wscale 3,sackOK,TS val 5298892 ecr 0], length 0
00:00:02.741578 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.63811 > 192.168.0.30.80: Flags [S], seq 502839335, win 65535, options [mss 1452,nop,wscale 3,sackOK,TS val 5301641 ecr 0], length 0
00:00:00.258529 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.25358 > 192.168.0.30.80: Flags [S], seq 1736699979, win 65535, options [mss 1452,nop,wscale 3,sackOK,TS val 5301892 ecr 0], length 0
00:00:02.941571 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.63811 > 192.168.0.30.80: Flags [S], seq 502839335, win 65535, options [mss 1452,nop,wscale 3,sackOK,TS val 5304841 ecr 0], length 0
00:00:00.257690 rule 0..16777216/0(match): rdr in on em0: x.x.x.x.25358 > 192.168.0.30.80: Flags [S], seq 1736699979, win 65535, options [mss 1452,nop,wscale 3,sackOK,TS val 5305092 ecr 0], length 0

I'm still playing around with the rules, but to this point nothing seems to work. It's like I'm missing something obvious, but I just can't figure out what it is :(.

Appreciate if someone could enlighten me with some points.

Thanks.
 
If I understand you correctly you're trying to redirect incoming traffic back out to another host again? You cannot 'bounce' traffic back out on the same interface with PF. You can only redirect it to a different interface on the same host (so you have to use a separate network to route between the hosts, e.g. using a cross-connect).
 
Actually I'm trying to get relayd to work but it failed, so I decided to try on pf alone and see if the redirection works.

Just wondering, is it possible if I redirect traffic from em0 to em1, then em1 redirects to the jail private IP? As I mentioned in previous post, em0 is binded with public IPs and em1 is binded with private IPs.
 
That could work, but that means you'd have competing services on port 80 on the outside IP address, so you can't use port 80 for anything else once traffic to port 80 is being redirected to either a local or a remote jail.

So, to stay in your setup, this would probably work:

Code: 
rdr pass on $ext_if inet proto tcp from any to $ext_if port 80 -> 192.168.0.30

assuming that 192.168.0.30 is directly connected to a different interface on that server (e.g. on the LAN behind the other interface).

I tested the above with this setup:

Code: 
public IP-[router]-10.0.0.1
                         |
                         |
                         |
                         |
                   10.0.0.2-[workstation]

and this rule on the router:

Code: 
rdr pass on $ext_if inet proto tcp from any to $ext_if port 33333 -> 10.0.0.2 port 33333

and this is what I saw on the workstation after a tcp connection to the router's external IP address, port 33333

Code: 
IP a.b.c.d > 10.0.0.2.33333: Flags [S] (so that's a different host on the LAN behind 10.0.0.1)
IP 10.0.0.2.33333 > a.b.c.d: Flags [R.] (host refuses connection, port isn't open)

and this is what I saw on the external IP address making that connection:

Code: 
IP a.b.c.d > w.x.y.z.33333: Flags [S] (that's the connection over the Internet to the router's public IP)
IP w.x.y.z.33333 > a.b.c.d: Flags [R.] (host's refusal routed back via NAT, and over the net)

So the entire connection and return traffic are OK (the [R] means that that port wasn't actually open, so the tcp connection is properly refused and reset).
 
Thanks, DD. Turned out that my /etc/pf.conf was missing the pass in rules. The redirection works with the following setup.

Say I am accessing the public IP of server1, and the traffic will be routed to server2's jails.

Code: 
server1:
ext_if = 176.x.x.x (em0)
int_if = 192.168.0.x (em1)

Code: 
server2:
ext_if = 176.y.y.y (em0)
int_if = 192.168.0.y (em1)
jail01 = 192.168.0.1 (em1)
jail02 = 192.168.0.2 (em1)

In server1 /etc/pf.conf:
Code: 
table <webpool> persist { $jail01 $jail02 }

nat proto tcp from { $jail01 $jail02 } port 80 to any -> $ext_if
rdr on $ext_if proto tcp to $server1 port 80 -> <webpool> round-robin

pass in on $ext_if proto tcp from any to { $jail01 $jail02 } port 80 flags S/SA keep state


In server2 /etc/pf.conf:
Code: 
pass in quick on $int_if reply-to ( $int_if 192.168.0.x ) proto tcp from any to { $jail01 $jail02 } port 80 keep state


tcpdump output when I'm making a connection to server1:
Code: 
00:00:00.206484 rule 0/0(match): rdr in on re0: a.b.c.d.22156 > 192.168.0.1.80: 
00:00:00.207429 rule 0/0(match): rdr in on re0: a.b.c.d.61087 > 192.168.0.2.80: 
00:00:00.000105 rule 0/0(match): rdr out on re0: 176.x.x.x.80 > a.b.c.d.61087:  
00:00:00.227124 rule 0/0(match): rdr out on re0: 176.x.x.x.80 > a.b.c.d.22156:
 
You must log in or register to reply here.
Back
Top