FreeBSD 8.3 PF CARP redirect

T

triumdh

Hello,

I have a basic testing setup with four machines.

Code: 
                 -[FreeBSD 8.3 gateway 1]-
               -                           -
             -                               -
[Windows PC]                                   [FreeBSD 8.3 server 1]
            -                                -
              -                            -
                 -[FreeBSD 8.3 gateway 2]-

I am testing CARP and port redirection. Gateway 1 and 2 have CARP interfaces.

gateway 1 rc.conf
Code: 
cloned_interfaces="carp0"
ifconfig_carp0="vhid 1 pass PASSWORD 10.0.0.18 netmask 255.255.255.0"
ifconfig_igb0_name="wan0"        # ISP
ifconfig_wan0="10.0.0.17/24"

gateway 2 rc.conf
Code: 
cloned_interfaces="carp0"
ifconfig_carp0="vhid 1 pass PASSWORD advskew 100 10.0.0.18 netmask 255.255.255.0"
ifconfig_igb0_name="wan0"        # ISP
ifconfig_wan0="10.0.0.16/24"

gateway 1 and 2 share same pf.conf
Code: 
## TRANSLATION RULES (NAT)
rdr on wan0 inet proto { tcp, udp } from any to wan0 port = 8822 -> 192.168.1.97 port 8822

## FILTER RULES 
pass out log all keep state
pass in log all keep state

When I set server 1 with the gateway 10.0.0.18 (the CARP interface for gateway 1) I cannot ssh on port 8822 to server 1 from the windows PC.

When I set server 1 with the gateway 10.0.0.17 (the actual interface interface for gateway 1) I can ssh on port 8822 to server 1 from the windows PC.

This seems to say that the CARP interface is the issue.

I have read that FreeBSD does not have the ability to bind a CARP interface to a specific interface. Is this the problem?

ifconfig on the gateways shows an apparent bind to the loopback address.

Code: 
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 10.0.0.18 netmask 0xffffff00
        carp: MASTER vhid 1 advbase 1 advskew 0

Any help always appreciated.
 
Hi,

The CARP interface is not the issue, the issue is in your pf.conf.

Try something like that and tell us if it works.

Code: 
carp_if="10.0.0.18"

## TRANSLATION RULES (NAT)
rdr on wan0 inet proto { tcp, udp } from any to $carp_if port = 8822 -> 192.168.1.97 port 8822

## FILTER RULES 
pass out log all keep state
pass in log all keep state
 
I tried that with the same results. The connection works when I don't use the CARP IP as the default gateway on the server 1.

BTW What was your theory for adding that command to the pf.conf?
 
A simple tcpdump should tell you exactly what is wrong:

[CMD=""]# tcpdump -n -e -ttt -i pflog0[/CMD]

However, I believe that you need to do the NAT on the carp interface. You can try:

Code: 
rdr on $carp_if inet proto { tcp, udp } from any to $carp_if port = 8822 -> 192.168.1.97 port 8822

See if this works.
 
triumdh said:
I tried that with the same results. The connection works when I don't use the CARP IP as the default gateway on the server 1.

BTW What was your theory for adding that command to the pf.conf?
Click to expand...


Carp interfaces are just virtual interfaces, it's like an alias. If you want to filter the traffic by interfaces on pf, you have to do it on the physical interface that the carp interface depends. It's why in your pf.conf you write on PHYSICAL_INTERFACE.

On the other side if you want to allow or deny traffic by ip, you use the carp address. Take a look at this http://www.openbsd.org/faq/pf/carp.html#RulesetTips

And what I wrote before could be simplified like this:

Code: 
## TRANSLATION RULES (NAT)
rdr on wan0 inet proto { tcp, udp } from any to carp0 port = 8822 -> 192.168.1.97 port 8822

## FILTER RULES 
pass out log all keep state
pass in log all keep state

Could you post the output of pfctl -vnf /etc/pf.conf with these lines ?
 
Thank you for the reply,

Result of [cmd=]pfctl -vnf /etc/pf.conf[/cmd]

Code: 
SYN_ONLY = "S/FSRA"
set loginterface lan0
carp_if = "192.168.1.98"
rdr on wan0 inet proto tcp from any to 192.168.10.100 port = 8822 -> 10.0.0.97 port 8822
rdr on wan0 inet proto udp from any to 192.168.10.100 port = 8822 -> 10.0.0.97 port 8822
pass out log all flags S/SA keep state
pass in log all flags S/SA keep state

The result of [cmd=]tcpdump -n -e -ttt -i pflog0[/cmd] shows no traffic going to the destination server. All results of tcpdump on any interface show me that the traffic stops after going to the gateway. When I change the default gateway on the destination server to the physical interface on the gateway, tcpdump shows traffic going to the destination server.

The gateway CARP interface is on the server 1 (destination) server side of the gateway so

Code: 
rdr on wan0 inet proto { tcp, udp } from any to carp0 port = 8822 -> 192.168.1.97 port 8822

wouldn't help. Sorry wasn't clearer on the setup.


Code: 
               Wan0                   Carp0/Lan0
                 -[FreeBSD 8.3 gateway 1]-
               -                           -
             -                               -
[Windows PC]                                   [FreeBSD 8.3 server 1]
            -                                -
              -                            -
                 -[FreeBSD 8.3 gateway 2]-
               Wan0                   Carp0/Lan0

When I set the default gateway on "server 1" to the carp0 IP address SSH does not connect.
When I set the default gateway on "server 1" to the Lan0 IP address SSH does connect.
 
It's very strange.

Where does these addresses come from? 192.168.10.100 and 10.0.0.97?

Can you post a more detailed scheme with interfaces, addresses and netmasks of the client, the gateways and the server please?
 
gkontos said:
Does your rc.conf include:

Code: 
gateway_enable=YES"
Click to expand...

When I set server 1 with the gateway 10.0.0.17 (the actual interface interface for gateway 1) I can ssh on port 8822 to server 1 from the windows PC.
Click to expand...


From what he said IP forwarding is enabled otherwise what he told would not work.
 
I think I gave some misinformation about one of the ip addresses. I also took out gateway 2. Just to clarify.

Code: 
               wan0                   carp0/lan0
                 -[FreeBSD 8.3 gateway 1]-
               -                           -
             -                               -
[Windows PC]                                   [FreeBSD 8.3 server 1]


gateway 1

Code: 
#ifconfig
lan0:   inet 192.168.1.251 netmask 0xffffff00 broadcast 192.168.1.255

wan0:   inet 10.0.0.17 netmask 0xffffff00 broadcast 10.0.0.255

carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 10.0.0.18 netmask 0xffffff00
        carp: MASTER vhid 1 advbase 1 advskew 0

When I set the default gateway on server 1 to the carp0 interface, the PC cannot ssh to server 1.
When I set the default gateway on server 1 to the lan0 interface, the PC can ssh to server 1.

This rules out everything but CARP or PF. There is a minimal PF config allowing everything through.
 
triumdh said:
I think I gave some misinformation about one of the ip addresses. I also took out gateway 2. Just to clarify.

Code: 
               wan0                   carp0/lan0
                 -[FreeBSD 8.3 gateway 1]-
               -                           -
             -                               -
[Windows PC]                                   [FreeBSD 8.3 server 1]
Click to expand...

You need two CARP interfaces, one on wan0 and one on lan0. You need also a pfsync interface to synchronize packet filter states between the two firewalls.

Something like

Code: 
carp0 : 192.168.1.254/24 on lan0
carp1 : 10.0.0.254/24 on wan0

Set the sysctl net.inet.carp.preempt to '1' (see man carp)

Then use the CARP addresses for gateway.

Regards.
 
Thank you, plamaiziere.

Getting CARP set up is not the issue. Getting PF to redirect through a CARP interface is the issue.

When I set the default gateway on server 1 to the carp0 interface, the PC cannot ssh to server 1. When I set the default gateway on server 1 to the lan0 interface, the PC can ssh to server 1.
Click to expand...

Everything in this setup works as expected. We even have a CARP setup in a live environment in several locations for several months but no one seems to have an answer to the PF redirect issue.

We are going to roll back to our original setup using Nagios to monitor interfaces and Event Handlers to control Interface state.

Thanks for everyone's help on this.
 
You must log in or register to reply here.
Back
Top