Other HA Firewall and carp – not all interfaces change when disconnect a nic

[amilis]

I have two firewalls running carp(4) between them for more than one+ year now. One 10 gbps nic for (14) internal VLANs and one 10 gbps nic for internet (10 external IPs). Three 1 gbps nics to other non VLAN network and one nic for pfSync. Both is up to date and running 13.1-RELEASE-p3. Everything work, except..

If FW1 is master and I do a reboot, FW2 takes over directly and vice versa. No problem.

But if I disconnect a nic, say the external nic (to internet) on FW1 (master), FW2 will take over, but only on the external nic. FW1 will have all other carp interfaces on master except the external that I disconnected. FW2 will only have the external nic on master and all other as backup.

Why? What do I miss?
On OpenBSD it put all carp-interface to the other server if only one nic fails/disconnect.

Fast cut and past.
Both servers is identical hardware HP DL360 G9 and same configs except IPs (FW1 have .1 and FW2 have .2 – virt IP .5) and advskew is 100 on FW2.

In /boot/loader.conf I have:

### PF CARP module
carp_load="YES"

in /etc/sysctl.conf I have:

### PF CARP module
net.inet.carp.preempt=1

In /etc/rc.conf I have (only the network/carp stuff..):

#### Clone ####
cloned_interfaces="vlan50 AND MANY MORE…."
#### Defaultrouter ####
defaultrouter="A.A.A.A"

#### External interface ####
ifconfig_oce0="inet A.A.A.B netmask 255.255.255.240"
ifconfig_oce0_alias10="vhid 10 advskew 0 pass PASSWORD1 alias A.A.A.C/32"
ifconfig_oce0_alias11="vhid 11 advskew 0 pass PASSWORD2 alias A.A.A.D/32"
# And more of the same above “aliasXX”, one for every external IP

#### Internal interface ####
ifconfig_oce1="up mtu 9000"
ifconfig_vlan50="inet 10.50.1.1 netmask 255.255.255.0 vlan 50 vlandev oce1 mtu 9000"
ifconfig_vlan51="inet 10.51.1.1 netmask 255.255.255.0 vlan 51 vlandev oce1 mtu 9000"
# And more of the same above “vlanXX”, one for every vlan

ifconfig_vlan50_alias100="vhid 100 advskew 0 pass PASSWORD20 alias 10.50.1.5/32"
ifconfig_vlan51_alias101="vhid 101 advskew 0 pass PASSWORD21 alias 10.51.1.5/32"
# And more of the same above “vlanXX_aliasXX”, one for every vlan

#### pfSync ####
ifconfig_em0="inet 172.25.37.1 netmask 255.255.255.0"
pfsync_enable="YES"
pfsync_syncdev="em0"

In /etc/pf.conf.. there is a lot, 1497 rows today, but I pass on pfsync and carp

########## CARP firewall failover ##########
pass quick on $PFSync proto pfsync keep state (no-sync)
pass quick on $ext_if proto carp keep state (no-sync)
pass quick on $int_if proto carp keep state (no-sync)
pass quick on $net50 proto carp keep state (no-sync)
pass quick on $net51 proto carp keep state (no-sync)
# And many more carp-lines, one for every network/vlan

What do I miss?

 

[Jose]

I'm going to guess it's your choice of using an advskew of 0. The man page for carp(4) says

This is the setup for host A (advskew is above 0 so it could be overwritten in the emergency situation from the other host)

Interestingly, the man page for ifconfig(8) states that 0 is not an "acceptable" value for advskew, even though it is the default.

 

[amilis]

Hmm.. interesting.. I really missed that one!
Just changed the settings to 50 on FW1 and have the same 100 on FW2. I will go to the racks tomorrow or the day after and unplug some cables. I come back after the test.