CARP stuck in init

joel@ · Feb 11, 2019

I've setup CARP on two hosts for testing but CARP gets stuck in init. I've used CARP before, but this is the first time with vlans. Output from ifconfig looks like this after rebooting both hosts and waiting a few minutes (real ip addresses removed):

Host1:

Code:

vlan60: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:f1
    inet A.A.A.1 netmask 0xffffff80 broadcast A.A.A.127 vhid 10
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 60 vlanpcp: 0 parent interface: igb1
    carp: INIT vhid 10 advbase 1 advskew 100
    groups: vlan
vlan61: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:f1
    inet B.B.B.249 netmask 0xfffffff8 broadcast B.B.B.255 vhid 20
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 61 vlanpcp: 0 parent interface: igb1
    carp: INIT vhid 20 advbase 1 advskew 100
    groups: vlan
vlan62: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:f1
    inet C.C.C.1 netmask 0xffffffe0 broadcast C.C.C.31 vhid 30
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 62 vlanpcp: 0 parent interface: igb1
    carp: INIT vhid 30 advbase 1 advskew 100
    groups: vlan

Host2:

Code:

vlan60: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:97
    inet A.A.A.1 netmask 0xffffff80 broadcast A.A.A.127 vhid 10
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 60 vlanpcp: 0 parent interface: igb1
    carp: INIT vhid 10 advbase 1 advskew 200
    groups: vlan
vlan61: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:97
    inet B.B.B.249 netmask 0xfffffff8 broadcast B.B.B.255 vhid 20
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 61 vlanpcp: 0 parent interface: igb1
    carp: INIT vhid 20 advbase 1 advskew 200
    groups: vlan
vlan62: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:97
    inet C.C.C.1 netmask 0xffffffe0 broadcast C.C.C.31 vhid 30
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 62 vlanpcp: 0 parent interface: igb1
    carp: INIT vhid 30 advbase 1 advskew 200
    groups: vlan

Logs on both machines are also filled with:

Code:

arprequest: cannot find matching address
arprequest: cannot find matching address
arprequest: cannot find matching address
arprequest: cannot find matching address
arprequest: cannot find matching address
arprequest: cannot find matching address
arprequest: cannot find matching address
arprequest: cannot find matching address

Configuration on both machines looks like this:

host01:

Code:

ifconfig_igb1="up -tso -lro -vlanhwtso"
cloned_interfaces="vlan60 vlan61 vlan62"
ifconfig_vlan60="inet A.A.A.1/25 vlan 60 vlandev igb1 vhid 10 advskew 100 pass secret"
ifconfig_vlan61="inet B.B.B.249/29 vlan 61 vlandev igb1 vhid 20 advskew 100 pass secret"
ifconfig_vlan62="inet C.C.C.1/27 vlan 62 vlandev igb1 vhid 30 advskew 100 pass secret"

host02:

Code:

ifconfig_igb1="up -tso -lro -vlanhwtso"
cloned_interfaces="vlan60 vlan61 vlan62"
ifconfig_vlan60="inet A.A.A.1/25 vlan 60 vlandev igb1 vhid 10 advskew 200 pass secret"
ifconfig_vlan61="inet B.B.B.249/29 vlan 61 vlandev igb1 vhid 20 advskew 200 pass secret"
ifconfig_vlan62="inet C.C.C.1/27 vlan 62 vlandev igb1 vhid 30 advskew 200 pass secret"

CARP settings are identical on both hosts:

Code:

# sysctl net.inet.carp
net.inet.carp.ifdown_demotion_factor: 240
net.inet.carp.senderr_demotion_factor: 240
net.inet.carp.demotion: 0
net.inet.carp.log: 1
net.inet.carp.preempt: 1
net.inet.carp.allow: 1

Any ideas?

SirDice · Feb 11, 2019

Your hosts are not connected, that's the first thing that comes to mind. You're using vlan(4) tagged interfaces, are your switch ports correctly configured for trunking?

joel@ · Feb 11, 2019

Yes, I believe switch ports are configured correctly.

Here is a similar config that works:

Host 1:

 

ifconfig_igb1="up -tso -lro -vlanhwtso"

cloned_interfaces="vlan60 vlan61 vlan62"

ifconfig_vlan60="inet A.A.A.2/25 vlan 60 vlandev igb1"

ifconfig_vlan60_alias0="inet A.A.A.1/32 vhid 10 advskew 100 pass secret"

Host 2:

 

ifconfig_igb1="up -tso -lro -vlanhwtso"

cloned_interfaces="vlan60 vlan61 vlan62"

ifconfig_vlan60="inet A.A.A.3/25 vlan 60 vlandev igb1"

ifconfig_vlan60_alias0="inet A.A.A.1/32 vhid 10 advskew 200 pass secret"

Code:

vlan60: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:f1
    inet A.A.A.2 netmask 0xffffff80 broadcast A.A.A.127
    inet A.A.A.1 netmask 0xffffffff broadcast A.A.A.1 vhid 10
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 60 vlanpcp: 0 parent interface: igb1
    carp: MASTER vhid 10 advbase 1 advskew 100
    groups: vlan

Code:

vlan60: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:1b:21:df:e5:97
    inet A.A.A.3 netmask 0xffffff80 broadcast A.A.A.127
    inet A.A.A.1 netmask 0xffffffff broadcast A.A.A.1 vhid 10
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 60 vlanpcp: 0 parent interface: igb1
    carp: BACKUP vhid 10 advbase 1 advskew 200
    groups: vlan

This is not how I want to do it however - I want CARP to just have a single IP that is shared between the machines. Also, the /32 is applied to the wrong interface in this example

SirDice · Feb 11, 2019

You need 'physical' addresses. Typically you use .2 for one node, .3 for the other node and .1 as a virtual IP.

joel@ · Feb 11, 2019

Hm, Ok. That's odd. I've seen setups using CARP with a /30.

A quick search in the forum archive:

Solved - Routing on carp, interface question

Hello guys, I hope to get some suggestions from you. I have multiple FreeBSD machines currently running on 10.0-RELEASE-p9. 2 machines running with a CARP setup and provide the Internet/WAN Connection. They are connected to the ISP router. The ISP provides a x.x.67.0/30 subnet for the WAN...

forums.freebsd.org

Solved - CARP on /30 subnet

Hello folks, I have the following situation: +-----------+ 172.31.255.6/24 | | 192.168.100.1/24 +--------+ silver1 +---------+...

forums.freebsd.org

SirDice · Feb 11, 2019

CARP works with multicast and the multicast needs a source address. Not sure how this would work without one.

Code:

root@p1:~# tcpdump -ni igb0 -T carp carp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:08:23.696885 IP 172.17.40.251 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=8453463975696288273
12:08:24.722554 IP 172.17.40.251 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=8453463975696288274
12:08:25.771684 IP 172.17.40.251 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=8453463975696288275
12:08:26.829151 IP 172.17.40.251 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=8453463975696288276
12:08:27.876249 IP 172.17.40.251 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=8453463975696288277
^C
5 packets captured
2361 packets received by filter
0 packets dropped by kernel
root@p1:~# ifconfig igb0
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:25:90:c6:25:a4
        hwaddr 00:25:90:c6:25:a4
        inet 172.17.40.251 netmask 0xffffff00 broadcast 172.17.40.255
        inet 172.17.40.250 netmask 0xffffff00 broadcast 172.17.40.255 vhid 1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: MASTER vhid 1 advbase 1 advskew 0

joel@ · Feb 11, 2019

I'm not saying you're wrong.

I see lots of references to using carp with a /30 however (that's why I'm experimenting with a single ip), for example this thread: https://lists.freebsd.org/pipermail/freebsd-net/2014-October/040114.html

SirDice · Feb 11, 2019

Use tcpdump(8) to have a look "on the wire". Make sure you can actually see the multicast on both nodes.

phoenix · Feb 11, 2019

[Note: we retired our last FreeBSD systems using CARP back in the 10.x days, and I haven't used one since then. Configs below taken from our backups. But, we did use CARP and vlans successfully for several years with 9.x and 10.x.]

Split your vlan configuration out from your CARP configuration, and see if that helps. There's some timing issues with doing it all in one go (if I'm remembering correctly) via rc.conf.

The setup we used back in the 10.x days (haven't used CARP on 11+) was similar to:

Code:

# em1  is the second  NIC port from the top of the quad-port NIC
# This interface is connected to the fibre network, and is the device
# used for the vlans.  It just needs to be up, no IPs assigned.
ifconfig_em1="up "
vlans_em1="vlan100 vlan120 vlan130 vlan140 vlan150 vlan160 vlan180"

# School 1
create_args_vlan100="vlan 100"
ifconfig_vlan100="vhid 100 pass nxsp4ss100 A.A.A.1/27"

# School 2
create_args_vlan110="vlan 110"
ifconfig_vlan110="vhid 110 pass nxsp4ss110 A.A.A.33/27"

# School 3
create_args_vlan120="vlan 120"
ifconfig_vlan120="vhid 120 pass nxsp4ss120 A.A.A.161/27"
ifconfig_vlan120_alias0="vhid 120 pass nxsp4ss120 B.B.B.9/29 alias"

# School 4
create_args_vlan130="vlan 130"
ifconfig_vlan130="vhid 130 pass nxsp4ss130 A.A.A.97/27"
ifconfig_vlan130_alias0="vhid 130 pass nxsp4ss130 C.C.C.33/29 alias"

This uses some newer rc.conf syntax for vlans, and made things much nicer to work with for us. Especially compared to the 9.x days.

On our other pair of FreeBSD 10.x boxes, we did everything in one line in rc.conf, but those systems didn't use vlans for anything:

Code:

# igb0  is the first   NIC port from the top of the quad-port NIC (top)
# This interface is connected to the Internet
defaultrouter="B.B.B.129"

ifconfig_igb0="inet B.B.B.130/32 vhid 30 pass romp4ss030 advskew 1 -lro -tso -vlanhwtso"
ifconfig_igb0_alias0="inet B.B.B.251/25"

# igb1  is the second NIC port from the top of the quad-port NIC
ifconfig_igb1="inet 10.10.10.254/32 vhid 31 pass romp4ss031 advskew 1 -lro -tso -vlanhwtso"
ifconfig_igb1_alias0="inet 10.10.10.251/24"

# igb2  is the third NIC port from the top of the quad-port NIC
ifconfig_igb2="inet 192.168.0.1/32 vhid 32 pass romp4ss032 advskew 1 -lro -tso -vlanhwtso"
ifconfig_igb2_alias0="inet 192.168.0.242/24"

# igb3  is the bottom NIC port on the quad-port NIC
ifconfig_igb3="inet 10.20.0.1/32 vhid 33 pass romp4ss033 advskew 1 -lro -tso -vlanhwtso"
ifconfig_igb3_alias1="inet 10.20.0.251/16"

CARP stuck in init

joel@

SirDice

Administrator

joel@

SirDice

Administrator

joel@

Solved - Routing on carp, interface question

Solved - CARP on /30 subnet

SirDice

Administrator

joel@

SirDice

Administrator

phoenix