How to get pf to rdr on FreeBSD 8.2

[Xosted]

Hi all,

I am getting a headache on trying to set pf properly. But first:

Environment:
- A FreeBSD 8.2 box with 5 NICs (em0 to em4).
- em0 and em1 are connected to routers and so parts of two local networks: A 192.168.0.* and B 192.168.1.*
- Each router has its own access to the Internet.
- em2 to em4 are connected to the Internet provider, each receiving its own IP address.
- Several jails are running on the FreeBSD box, each having as IP address an alias on em0 (192.168.0.15* so visible to the A network).

Objectives:
- 1) Give access to the Internet to the jails via A network router.
- 2) Make services in the jails available to A network.
- 3) Make services in the jails available to Internet via A network.

This is my pf.conf:

net_if0="em0"
net_if1="em1"
ext_if2="em2"
ext_if3="em3"
ext_if4="em4"

net0_addr="192.168.0.90"

serv1="192.168.0.152"
serv2="192.168.0.155"


scrub in all

#Internet for jails through Network A
nat on $net_if0 from {$serv1,$serv2} to any -> ($net_if0)

#Redirect Network A requests to jails
rdr pass log(all) on $net_if0 inet proto { tcp, udp } from any to ($net_if0) port 10090 -> $serv1 port 80

pass in all
pass out all

With this, the jails are getting access to the Internet.
A computer on Network A can access the service on serv1 jail.
A computer from the Internet getting through Network A to serv1 jail hangs until timeout.

This is the output of tcpdump -n -e -ttt -i pflog0 host IP_OF_PC:

00:00:03.026752 rule 0/0(match): rdr in on em0: IP_OF_PC.57069 > 192.168.0.152.80: [|tcp]
00:00:00.000036 rule 0/0(match): rdr out on em2: 192.168.0.90.10090 > IP_OF_PC.57069: [|tcp]
00:00:02.999186 rule 0/0(match): rdr out on em2: 192.168.0.90.10090 > IP_OF_PC.57069: [|tcp]
00:00:05.999998 rule 0/0(match): rdr out on em2: 192.168.0.90.10090 > IP_OF_PC.57069: [|tcp]
00:00:11.999960 rule 0/0(match): rdr out on em2: 192.168.0.90.10090 > IP_OF_PC.57069: [|tcp]

So the question is... why does it try to reply on em2 while replying normally on em0 in the case of a PC inside Network A (so with an 192.168.0.* IP)?

Thank you for your help.

 

[SirDice]

Look at your routing tables. Look where traffic destined to IP_OF_PC is routed to.

 

[Xosted]

I checked the output of nestat -r and found this line:

A.B.C.0/21    link#3             U           0      205    em2

I guess that this is the problem, considering that the IP_OF_PC is equal to A.B.C+5.33.
Indeed, it is on one of our connections so I guess it is related to how the provider allocates the IP addresses.

But... then what to do? How do I force it to consider where the packets are coming from over "where they should go"?

 

[SirDice]

You might be able to use PF's route-to but I've never used it so I'm not sure how to use it properly.

 

[Xosted]

After some days looking at it I have to admit I don't get it. At the same time, it replies through em2 ok but this shouldn't matter should it? I mean, the important thing is the destination, not the route. If the browser sends a request through the given address/IP, the reply should get there even if the route is different, shouldn't it?

Is it possible that pf would actually block the traffic through em2 because there are no rules about it? Is there a way to turn on a general logging so pf would report all packets that it handles no matter how?

 

[SirDice]

I mean, the important thing is the destination, not the route. If the browser sends a request through the given address/IP, the reply should get there even if the route is different, shouldn't it?

Routing is done based on the destination address, nothing else. The system doesn't care where the data came from, it'll only look at the destination address to determine where to route the traffic to next.

 

[Xosted]

Yes so that's exactly what I was saying.
But still, it doesn't work. Of course, it could be anything that drops the packets after they have left through em0.

I am going to make another post for help on forcing the packets to come out the same route.