CARP - both nodes MASTER

F

Fatman

Hi Everyone,

I've been trying to setup CARP for a redundant firewall but I have run into some problems. I need this to do a demo but I can't seem get my traffic passed the firewall to the internet router. This is going to be a long post but hopefully it will help others.

I've been following the advice from A No-Nonsense Guide to the
OpenBSD Firewall and information on the net.

Here's my setup:

InternetRouter
|
Switch
| |
FWA--FWB (PFSYNC Between the two using CrossOver Cable)
| |
Switch
|
LAN

Here's some outputs that will help:

FirewallA# ifconfig
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:02:55:3f:49:ae
inet 10.1.0.1 netmask 0xfffffffc broadcast 10.1.0.3
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
rl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:17:3f:ce:fd:1a
inet 192.168.20.103 netmask 0xffffff00 broadcast 192.168.20.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
rl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:17:3f:cf:00:c3
inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu 1500
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
inet 192.168.1.105 netmask 0xffffff00
carp: MASTER vhid 1 advbase 1 advskew 0
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
inet 192.168.20.1 netmask 0xffffff00
carp: MASTER vhid 2 advbase 1 advskew 0

FirewallA## sysctl net.inet.carp
net.inet.carp.allow: 1
net.inet.carp.preempt: 0
net.inet.carp.log: 1
net.inet.carp.arpbalance: 0
net.inet.carp.suppress_preempt: 2

FirewallB# sysctl net.inet.carp
net.inet.carp.allow: 1
net.inet.carp.preempt: 0
net.inet.carp.log: 1
net.inet.carp.arpbalance: 0
net.inet.carp.suppress_preempt: 0

FWA - RC.CONF
cloned_interfaces="carp0 carp1"
ifconfig_rl0="inet 192.168.20.103/24"
ifconfig_rl1="inet 192.168.1.103/24"
ifconfig_fxp0="inet 10.1.0.1/30"
ifconfig_carp0="inet 192.168.1.105/24 vhid 1 pass foo"
ifconfig_carp1="inet 192.168.20.1/24 vhid 2 pass bar"
ifconfig_pfsync0="syncdev fxp0"
defaultrouter="192.168.1.1"
gateway_enable="YES"

FWB - RC.CONF
cloned_interfaces="carp0 carp1"
ifconfig_rl0="inet 192.168.20.104/24"
ifconfig_rl1="inet 192.168.1.104/24"
ifconfig_fxp0="inet 10.1.0.2/30"
ifconfig_carp0="inet 192.168.1.105/24 vhid 1 advskew 100 pass foo"
ifconfig_carp1="inet 192.168.20.1/24 vhid 2 advskew 100 pass bar"
ifconfig_pfsync0="syncdev fxp0"

FirewallB# ifconfig
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:02:55:bf:0d:fe
inet 10.1.0.2 netmask 0xfffffffc broadcast 10.1.0.3
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
rl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:21:27:dd:c9:3a
inet 192.168.20.104 netmask 0xffffff00 broadcast 192.168.20.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
rl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:1d:0f:cf:14:6e
inet 192.168.1.104 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu 1500
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=0<> metric 0 mtu 33204
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
inet 192.168.1.105 netmask 0xffffff00
carp: MASTER vhid 1 advbase 1 advskew 100
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
inet 192.168.20.1 netmask 0xffffff00
carp: MASTER vhid 2 advbase 1 advskew 100

I noticed that both FWA and B carp interfaces are MASTER. From the LAN, I can ping carp0 and carp1 but traffic not not make it to the internet router.

I also cannot redirect SSH from the internet router to CARP0. However, I can redirect SSH to FWA rl0 interface. I did a quick search and found that SSHing to CARP0 does not work.

Anyone know why both CARP0/1 on FWA and FWB MASTERS? Is this due to pfsync not working properly.

Here's my PF.CONF file:
ext_if="rl1"
int_if="rl0"
#dmz_if="fxp0"
carpdev="{carp0, carp1}"
syncdev="fxp0"

table <private_net> { 0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12}

# blacklist host
#table <blacklist> persist file "/usr/local/etc/pf/pf.blacklist"

#dmz_nets="172.16.0.0/24"
internal_nets="{192.168.20.0/24}"

ssh_www="172.16.0.80"
www_server="172.16.0.80"
vpn_server="172.16.0.200"
ssh_vpn="172.16.0.200"
update_server="172.16.0.169"

FW="172.16.0.1"

Davor="99.224.25.91"

trusted_vpn="{99.246.179.73, 207.112.122.90, 192.168.1.1, 209.87.232.162 }"
trusted_www="{99.246.179.73, 207.112.122.90, 192.168.1.1, 209.87.232.162 }"

trusted_nets = "{ 192.168.20.0/24 }"
trusted_hosts = "{ 192.168.20.0/24 }"

client_out="{ftp-data, cvspserver, ftp, ssh, domain, pop3, auth, nntp, http, https, 8080, 8000}"
www = "{80, 443}"
udp_services = "{domain, ntp}"
sshport="22669"
rdp="3389"
vpn="1194"

## GLOBAL OPTIONS

set block-policy return
set loginterface $ext_if
set optimization normal
set skip on {lo0, $syncdev}
## TRAFFIC NORMALIZATION

scrub in all no-df
scrub out all no-df

#nat on $ext_if from $dmz_nets to any -> ($ext_if)
nat on $ext_if from $internal_nets to any -> ($ext_if)

#rdr on $ext_if proto {tcp, udp} from $trusted_vpn to any port $rdp -> $rdp_server
rdr on $ext_if proto tcp from $trusted_www to any port $www -> $www_server
rdr on $ext_if proto {tcp, udp} from $trusted_vpn to any port 1194 -> $vpn_server

block log all

# block all private ip addresses
#block in quick on $ext_if from { <private_net> }

pass on $carpdev proto carp keep state

pass inet proto tcp from any to ($ext_if) port $sshport keep state

pass in inet proto {tcp, udp} from any to $vpn_server port $vpn keep state
pass in inet proto tcp from $trusted_www to $www_server port $www keep state

#pass out log quick on $dmz_if proto tcp from $trusted_www to $www_server port $www keep state
#pass out log quick on $dmz_if proto tcp from $FW to $ssh_www port 22

#pass out log quick on $dmz_if proto {tcp, udp} from any to $vpn_server port $vpn keep state
#pass out log quick on $dmz_if proto tcp from $FW to $ssh_vpn port 22

#pass quick on $dmz_if proto {tcp, udp, icmp} from any to any keep state
#pass quick on $dmz_if proto {tcp, udp, icmp} from $update_server to any keep state

#Let traffic out for the External Interface
pass out quick proto tcp from $ext_if to any flags S/SA keep state
pass out quick proto udp from $ext_if to any keep state
pass out quick proto icmp from $ext_if to any keep state

#Let Internal Traffic Flow Freely to DMZ
#pass in quick on $dmz_if inet proto tcp from $trusted_hosts to $dmz_nets keep state

#Let Internal Traffic Flow out
pass inet proto tcp from $internal_nets to any port $client_out
pass inet proto udp from $internal_nets to any port $udp_services
pass inet proto icmp from $internal_nets to any keep state

#Let Trusted Host Anywhere
pass inet proto {tcp, udp, icmp} from $trusted_hosts to any keep state

Hopefully someone read all of this!! :p

Thanks for reading.

FATMAN
 
Another quick question, re-reading A No-Nonsense Guide to the
OpenBSD Firewall, I see a line that talks about a sysctl value that needs be set for all CARP hosts.

**Snipit From Chapter**

Setting the net.inet.carp.preempt variable means that on hosts with more than one network interface, such as our gateways, all CARP interfaces will set their advskew (the meaning of which we will explain more thoroughly in a moment) to the extremely high value of 240 in order to prod other hosts in the CARP group to start failover when one of the interfaces goes down. This
setting needs to be identical on all hosts in the CARP group.

Are they reffering to all backup host or does it apply to both master and backup?
 
Hi guys,

Just to give you a heads up, I deceided to restart from scratch and try setting it up with a buddy. I will keep you posted on the results.
 
Ok,

So I rebuilt my setup without the help of my buddy. CARP seems to be working since shutting down an interface cause FWB to take over.

I still have one problem though. When the backup unit takes over, i'm forced to run pfctl -Fa -f /etc/pf.conf so that the traffic continues to flow.

Anyone know why that is?

Oh by the way, I followed the link below to help me get this setup:
http://www.familywilson.ca/pf-carp-freebsd-redundant-firewall/
 
Alright,

For some odd reason, i rebooted both systems and everything is working. Carp is fantastic!!!

I have one question though... Should the syncpeer reflect the IP of the adjacent connection to FWB? I tried setting the following in my rc.conf but it doesn't make the change.

ifconfig_pfsync0="up syncif fxp0"
pfsync_syncdev="fxp0"
pfsync_syncpeer="10.1.0.4"

Here's the output:
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
syncpeer: 224.0.0.240 maxupd: 128

What is necessary in the rc.conf file for pfsync to be working? By leaving the above or removing the first line going to have any unwanted effects?

Hmm... Will I find my own answers before someone replies to this post????? Kudo's to those who made it to the bottom of this thread.

Later,

Fatman
 
Code: 
ifconfig_pfsync0="up syncif fxp0"
This seems wrong to me.. there is no syncif option that I can see in pfsync(4).

I would change it to:
Code: 
ifconfig_pfsync0="up"
and leave the other lines the same.
 
Thanks for the advice, I went a head and made the change. I noticed a minor problem with CARP. Even though i have net.inet.carp.preempt set to 1 on both boxes, bringing down an interface does not cause complete failover. The system remains redundant and the backup interface takes over.

My understanding is that with preempt set to one, bringing down one interface, brings down all and the fail over unit takes over.

Any suggestion as to where I should start looking for the problem??
 
I think I've found the problem but can't find anything in the man pages or online for changing the value.

When comparing the output of sysctl -a | grep carp, I see a difference with the suppress_preemp.

FirewallA# sysctl -a | grep carp
net.inet.ip.same_prefix_carp_only: 0
net.inet.carp.allow: 1
net.inet.carp.preempt: 1
net.inet.carp.log: 1
net.inet.carp.arpbalance: 0
net.inet.carp.suppress_preempt: 2

FirewallB# sysctl -a | grep carp
net.inet.ip.same_prefix_carp_only: 0
net.inet.carp.allow: 1
net.inet.carp.preempt: 1
net.inet.carp.log: 1
net.inet.carp.arpbalance: 0
net.inet.carp.suppress_preempt: 1

Here's a snip it regarding suppress_preempt from the man pages:
A read only value showing the status of preemption suppression. Preemption can be suppressed if link on an interface is down or when pfsync(4) interface is not synchronized. Value of 0 means that preemption is not suppressed, since no problems are detected. Every problem increments suppression counter.

Is the fact that my suppression counters are different the cause of the issue? Carp seems to work but preemption being enabled has no effect on bringing down all the other interface if a failure occurs on a single interface. However, when you check the status of the carp interfaces, the carp interface on FWA shows as init and FWB a shows as master. All other interfaces on FWA remain MASTER but should of flipped over due to preempt being set to 1.
 
Just in case anyone is following this thread, I was told that the problem could be caused by faulty or bad nics. I did install a couple of 6 dollar NICs made in CHINA in these two firewalls. I will go out today and buy some new ones. Lets see if this corrects the issue.
 
Hmm.. I'm not sure that would fix the problem. When I initially setup CARP a few years ago, I don't remember the preempt support working. I haven't had time to sit down and do some real testing.
 
I'll tell you tonight whether it's the NICs. I just bought 4 new ones.

I think your right though about preempt not working. I tripled checked my configs and they seems to be accurate. CARP works because when I bring down an interface on FWA, the NIC state for the same lan on FWB becomes master. Traffic continues to flow through FWB but cause problems since the other interfaces don't go to MASTER.

Thanks BRD for posting your thoughts.

FATMAN
 
Sorry for taking so long to post my results. I still had the same problems after changing all the NICs. I'm not sure what I've done wrong however rebuilding the servers using OpenBSD solved the problem.

Thanks to all those who tried helping!

Fatman
 
You must log in or register to reply here.
Back
Top