CARP config... again

kazix · Oct 27, 2016

hi, i need help for CARP configuration.

i have 2 host: A and B and 5 local IP address.
Host have 4 networks card (link agregation as LACP) - works fine.

Host A (network configuration)

 

# LAGG 0

ifconfig_bce0="up"

ifconfig_em0="up"

cloned_interfaces="lagg0 lagg1 lo1"

ifconfig_lagg0="up laggproto lacp laggport bce0 laggport em0 10.10.1.101/24"



# IP for Jails

ifconfig_lagg0_alias01="inet vhid 1 advskew 1 pass paswd alias 10.10.1.145/32"

ifconfig_lagg0_alias02="inet vhid 2 advskew 1 pass paswd alias 10.10.1.143/32"

ifconfig_lagg0_alias03="inet vhid 3 advskew 10 pass paswd alias 10.10.1.139/32"

ifconfig_lagg0_alias04="inet vhid 4 advskew 10 pass paswd alias 10.10.1.140/32"

ifconfig_lagg0_alias05="inet vhid 5 advskew 1 pass paswd alias 10.10.1.142/32"

Host B (network configuration)

 # LAGG for Jails

ifconfig_bce0="up"

ifconfig_igb0="up"

cloned_interfaces="lagg0 lagg1 lo1"

ifconfig_lagg0="up laggproto lacp laggport bce0 laggport igb0 10.10.1.102/24"



# IP for Jails

ifconfig_lagg0_alias01="inet vhid 1 advskew 10 pass paswd alias 10.10.1.145/32"

ifconfig_lagg0_alias02="inet vhid 2 advskew 10 pass paswd alias 10.10.1.143/32"

ifconfig_lagg0_alias03="inet vhid 3 advskew 1 pass paswd alias 10.10.1.139/32"

ifconfig_lagg0_alias04="inet vhid 4 advskew 1 pass paswd alias 10.10.1.140/32"

ifconfig_lagg0_alias05="inet vhid 5 advskew 10 pass paswd alias 10.10.1.142/32"

Host A and Host B CARP config:

 

root@A:~ # sysctl -a | grep carp

net.inet.carp.ifdown_demotion_factor: 240

net.inet.carp.senderr_demotion_factor: 240

net.inet.carp.demotion: 1200

net.inet.carp.log: 1

net.inet.carp.preempt: 0

net.inet.carp.allow: 1

like i write in previous post https://forums.freebsd.org/threads/57943/
and like you said - better restore after crach by hand.
And there is problem - i dont know who will be MASTER after crash!

Host A - master for vhid 1,2,5

Code:

    carp: MASTER vhid 1 advbase 1 advskew 1
    carp: MASTER vhid 2 advbase 1 advskew 1
    carp: BACKUP vhid 3 advbase 1 advskew 10
    carp: BACKUP vhid 4 advbase 1 advskew 10
    carp: MASTER vhid 5 advbase 1 advskew 1

Host B - master for vhid 3,4

 

carp: BACKUP vhid 1 advbase 1 advskew 10

    carp: BACKUP vhid 2 advbase 1 advskew 10

    carp: MASTER vhid 3 advbase 1 advskew 1

    carp: MASTER vhid 4 advbase 1 advskew 1

    carp: BACKUP vhid 5 advbase 1 advskew 10

reboot Host A,
Host B is MASTER for all

 

tail -f /var/log/messages

Oct 27 10:58:55 B kernel: carp: VHID 1@lagg0: BACKUP -> MASTER (master down)

Oct 27 10:58:55 B kernel: carp: VHID 2@lagg0: BACKUP -> MASTER (master down)

Oct 27 10:58:56 B kernel: carp: VHID 5@lagg0: BACKUP -> MASTER (master down)

everything is FINE now

But when Host A is back, he takeover some vhid as MASTER (in this example vhid 5), but this is randomize - each time different vhid, one or all
Host B

 

tail -f /var/log/messages

Oct 27 11:01:22 B kernel: carp: VHID 5@lagg0: MASTER -> BACKUP (more frequent advertisement received)

and Host A

 

carp: BACKUP vhid 1 advbase 1 advskew 1

    carp: BACKUP vhid 2 advbase 1 advskew 1

    carp: BACKUP vhid 3 advbase 1 advskew 10

    carp: BACKUP vhid 4 advbase 1 advskew 10

    carp: MASTER vhid 5 advbase 1 advskew 1

preempt is disabled net.inet.carp.preempt: 0

what is happened? why host A and B randomizing MASTER/BACKUP.

SirDice · Oct 27, 2016

The first interface that comes up will be the master. The advskew really only comes into play when both interfaces come up at the same time, the advskew will add a little delay causing the interface with the lowest value to come up first and therefor be master.

kazix · Oct 27, 2016

ok, i understand - but problem is: host A crash and host B is master - that's ok, host A back and he should wait for sysadmin intervention (like you said in previous post https://forums.freebsd.org/threads/57943/). Wake up host should wait, not became MASTER for randomize IP's. This is very strange...

kazix

SirDice · Oct 27, 2016

Yesterday I upgraded two 9.3 machines that were using CARP to 10.3. Initially I had some problems, switching by hand just didn't work at all for some reason, it looked like preemption was on but it wasn't. I then ran freebsd-update(8) to get the system up to date with the latest 10.3 patch level and the problem disappeared. You may want to try that too, even if it's just to rule it out as a possible cause.

kazix · Oct 28, 2016

updated host A and B

Code:

FreeBSD ... 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11

reboot - no change

I will install (clean installation) FreeBSD v11 on both host and try, maybe work correctly

Two things:

I use lagg(4) (Link agregation LACP) - maybe this is problem. Someone use CARP with LAGG?

When host A is starting, in host B is something like this:

Code:

Oct 28 06:09:16 B kernel: carp: VHID 101@lagg0: MASTER -> BACKUP (more frequent advertisement received)
Oct 28 06:09:16 B kernel: carp: VHID 102@lagg0: MASTER -> BACKUP (more frequent advertisement received)
Oct 28 06:09:17 B kernel: carp: VHID 105@lagg0: MASTER -> BACKUP (more frequent advertisement received)

What is "more frequent advertisement received"?

kzx

kazix · Nov 2, 2016

hello,

install fresh FreeBSD v 11 and this is my network config:
HOST A
/etc/rc.conf

Code:

# CARP
    kld_list="carp"
#LAGG0 (jails)
    ifconfig_bce0="up"
    ifconfig_em0="up"
    ifconfig_lagg0="up laggproto lacp laggport bce0 laggport em0 10.10.1.101/24"
    ifconfig_lagg0_alias0="inet vhid 1 advskew 0 pass pass0 alias 10.10.1.145/32"
    ifconfig_lagg0_alias1="inet vhid 2 advskew 0 pass pass1 alias 10.10.1.143/32"
    ifconfig_lagg0_alias2="inet vhid 3 advskew 100 pass pass2 alias 10.10.1.139/32"
    ifconfig_lagg0_alias3="inet vhid 4 advskew 100 pass pass3 alias 10.10.1.140/32"
    ifconfig_lagg0_alias4="inet vhid 5 advskew 0 pass pass4 alias 10.10.1.142/32"
  #LAGG0

HOST B
/etc/rc.conf

Code:

# CARP     
    kld_list="carp"    
  # LAGG0
    ifconfig_lagg0_alias0="inet vhid 1 advskew 100 pass pass0 alias 10.10.1.145/32"
    ifconfig_lagg0_alias1="inet vhid 2 advskew 100 pass pass1 alias 10.10.1.143/32"
    ifconfig_lagg0_alias2="inet vhid 3 advskew 0 pass pass2 alias 10.10.1.139/32"
    ifconfig_lagg0_alias3="inet vhid 4 advskew 0 pass pass3 alias 10.10.1.140/32"
    ifconfig_lagg0_alias4="inet vhid 5 advskew 100 pass pass4 alias 10.10.1.142/32"
  # END LAGG0

preempt is disabled, all other defaults.
Work like I want - no automatic restore Master -> Backup.

thanks, kazix

phoenix · Nov 3, 2016

If all the shared IPs are in the same subnet, put them into the same VHID. That will prevent the situation where some IPs are MASTER on host A and some IPs are MASTER on host B.

leebrown66 · Nov 4, 2016

I've always liked to put the CARP traffic on a separate VLAN and script my own up/down events for the shared address.

It doesn't clutter my tcpdump
It's not prone to security issues

dch · Nov 5, 2016

leebrown66 said:
I've always liked to put the CARP traffic on a separate VLAN and script my own up/down events for the shared address.

It doesn't clutter my tcpdump

It's not prone to security issues

I've not used VLANs at all, is this straightforwards? Can you share some rc.conf snippets?

leebrown66 · Nov 5, 2016

So in this example, igb0 is my NIC, which has a few 10.1.1.0/24 addresses native (ie no VLAN tag).
The interface named vlan1300 for example has a bunch of 10.1.200.0/24 addresses on with a VLAN id of 1300. It doesn't have to be named vlan1300, you can call the interface Radio if you like, but if you have more than a handful it's a simple, consistent scheme.

Code:

vlans_igb0="vlan4001 vlan1300 vlan1301 vlan4002 vlan10 vlan3100"
create_args_vlan10="vlan 10"
create_args_vlan1300="vlan 1300"
create_args_vlan1301="vlan 1301"
create_args_vlan4001="vlan 4001"
create_args_vlan4002="vlan 4002"
ifconfig_igb0="inet 10.1.1.6 netmask 255.255.255.0"
ifconfig_igb0_aliases="\
 inet 10.1.1.3 netmask 255.255.255.255 \
 inet 10.1.1.7 netmask 255.255.255.255 \
 inet 10.1.1.8 netmask 255.255.255.255 \
 inet 10.1.1.25 netmask 255.255.255.255"

ifconfig_vlan4001_aliases="\
 vhid 100 pass 2bf7b5430bf68db2 10.1.250.100/24 up"
ifconfig_vlan1300_aliases="\
 inet 10.1.200.3 netmask 255.255.255.255 \
 inet 10.1.200.101 netmask 255.255.255.255 \
 inet 10.1.200.100 netmask 255.255.255.255 \
 inet 10.1.200.99 netmask 255.255.255.255 \
 inet 10.1.200.98 netmask 255.255.255.255 \
 inet 10.1.200.97 netmask 255.255.255.255 \
 inet 10.1.200.96 netmask 255.255.255.255"
ifconfig_vlan1301_aliases="\
 inet 10.1.201.100 netmask 255.255.255.255 \
 inet 10.1.201.101 netmask 255.255.255.255"
ifconfig_vlan10_aliases="\
 inet 10.1.10.3 netmask 255.255.255.255"
ifconfig_vlan4002_aliases="\
 inet 10.1.253.2 netmask 255.255.255.255"

which looks like this

Code:

$ ifconfig
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
   ether 0c:c4:7a:01:e3:9c
   inet 10.1.1.3 netmask 0xffffffff broadcast 10.1.1.3
   inet 10.1.1.7 netmask 0xffffffff broadcast 10.1.1.7
   inet 10.1.1.8 netmask 0xffffffff broadcast 10.1.1.8
   inet 10.1.1.25 netmask 0xffffffff broadcast 10.1.1.25
   inet 10.1.1.6 netmask 0xffffff00 broadcast 10.1.1.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
vlan4001: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 0c:c4:7a:01:e3:9c
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 4001 parent interface: igb0
vlan1300: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 0c:c4:7a:01:e3:9c
   inet 10.1.200.3 netmask 0xffffffff broadcast 10.1.200.3
   inet 10.1.200.101 netmask 0xffffffff broadcast 10.1.200.101
   inet 10.1.200.100 netmask 0xffffffff broadcast 10.1.200.100
   inet 10.1.200.99 netmask 0xffffffff broadcast 10.1.200.99
   inet 10.1.200.98 netmask 0xffffffff broadcast 10.1.200.98
   inet 10.1.200.97 netmask 0xffffffff broadcast 10.1.200.97
   inet 10.1.200.96 netmask 0xffffffff broadcast 10.1.200.96
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 1300 parent interface: igb0
vlan4002: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 0c:c4:7a:01:e3:9c
   inet 10.1.253.2 netmask 0xffffffff broadcast 10.1.253.2
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 4002 parent interface: igb0
vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 0c:c4:7a:01:e3:9c
   inet 10.1.10.3 netmask 0xffffffff broadcast 10.1.10.3
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 10 parent interface: igb0
vlan1301: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=303<RXCSUM,TXCSUM,TSO4,TSO6>
   ether 0c:c4:7a:01:e3:9c
   inet 10.1.201.100 netmask 0xffffffff broadcast 10.1.201.100
   inet 10.1.201.101 netmask 0xffffffff broadcast 10.1.201.101
   inet 10.1.201.102 netmask 0xffffffff broadcast 10.1.201.102
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   vlan: 1301 parent interface: igb0
$

CARP config... again

kazix

SirDice

Administrator

kazix

SirDice

Administrator

kazix

kazix

phoenix

leebrown66

dch

leebrown66