CARP configuration, advskew value

[kazix]

I'm testing CARP configuration for two host: A (master) and B (slave).
Problem is with advskew value - who (A or B) should have smaller value?
If A should be MASTER, then I give them advskew = 1, and B SLAVE advskew = 100?
Documentation say that: https://www.freebsd.org/doc/handbook/carp.html

But in my case that not work is vice versa...
If i give A advskew = 1 and B advskew = 100 then B is MASTER
If i give A advskew = 10 and B advskew = 1 then A is MASTER

This configuration work like i want (A-Master, B-SLAVE), but this is strange...

My config from host A, rc.conf:

ifconfig_lagg0="up laggproto lacp laggport bce0 laggport em0 10.10.1.101/24"
ifconfig_lagg0_alias01="inet vhid 1 advskew 10 pass passwd alias 10.10.1.145/32"
ifconfig_lagg0_alias02="inet vhid 2 advskew 10 pass passwd alias 10.10.1.143/32"
ifconfig_lagg0_alias03="inet vhid 3 advskew 10 pass passwd alias 10.10.1.139/32"
ifconfig_lagg0_alias04="inet vhid 4 advskew 10 pass passwd alias 10.10.1.141/32"
ifconfig_lagg0_alias05="inet vhid 5 advskew 10 pass passwd alias 10.10.1.142/32"

My config from host B, rc.conf:

ifconfig_lagg0="up laggproto lacp laggport bce0 laggport igb0 10.10.1.102/24"
ifconfig_lagg0_alias01="inet vhid 1 advskew 1 pass passwd alias 10.10.1.145/32"
ifconfig_lagg0_alias02="inet vhid 2 advskew 1 pass passwd alias 10.10.1.143/32"
ifconfig_lagg0_alias03="inet vhid 3 advskew 1 pass passwd alias 10.10.1.139/32"
ifconfig_lagg0_alias04="inet vhid 4 advskew 1 pass passwd alias 10.10.1.141/32"
ifconfig_lagg0_alias05="inet vhid 5 advskew 1 pass passwd alias 10.10.1.142/32"

ifconfig output from A:

lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500  options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
    ether 00:21:5a:a7:2a:94
    inet 10.10.1.101 netmask 0xffffff00 broadcast 10.10.1.255 
    inet 10.10.1.145 netmask 0xffffffff broadcast 10.10.1.145 vhid 1 
    inet 10.10.1.143 netmask 0xffffffff broadcast 10.10.1.143 vhid 2 
    inet 10.10.1.139 netmask 0xffffffff broadcast 10.10.1.139 vhid 3 
    inet 10.10.1.141 netmask 0xffffffff broadcast 10.10.1.141 vhid 4 
    inet 10.10.1.142 netmask 0xffffffff broadcast 10.10.1.142 vhid 5 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: active
    carp: MASTER vhid 1 advbase 1 advskew 10
    carp: MASTER vhid 2 advbase 1 advskew 10
    carp: MASTER vhid 3 advbase 1 advskew 10
    carp: MASTER vhid 4 advbase 1 advskew 10
    carp: MASTER vhid 5 advbase 1 advskew 10
    laggproto lacp lagghash l2,l3,l4
    laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: bce0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

ifconfig output from B:

lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
    ether 00:26:55:1e:ce:e2
    inet 10.10.1.102 netmask 0xffffff00 broadcast 10.10.1.255 
    inet 10.10.1.145 netmask 0xffffffff broadcast 10.10.1.145 vhid 1 
    inet 10.10.1.143 netmask 0xffffffff broadcast 10.10.1.143 vhid 2 
    inet 10.10.1.139 netmask 0xffffffff broadcast 10.10.1.139 vhid 3 
    inet 10.10.1.141 netmask 0xffffffff broadcast 10.10.1.141 vhid 4 
    inet 10.10.1.142 netmask 0xffffffff broadcast 10.10.1.142 vhid 5 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: active
    carp: BACKUP vhid 1 advbase 1 advskew 1
    carp: BACKUP vhid 2 advbase 1 advskew 1
    carp: BACKUP vhid 3 advbase 1 advskew 1
    carp: BACKUP vhid 4 advbase 1 advskew 1
    carp: BACKUP vhid 5 advbase 1 advskew 1
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: bce0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

Any idea???

 

[SirDice]

This is probably because preemption is turned off by default. So it doesn't automatically fall back. Personally I don't like the automatic fall back, I like to have control when to fall back or not.

     net.inet.carp.preempt                     Allow virtual hosts to preempt
                                               each other.  When enabled, a
                                               vhid in a backup state would
                                               preempt a master that is
                                               announcing itself with a lower
                                               advskew.  Disabled by default.

See carp(4).

 

[kazix]

SirDice - how you control who is MASTER and when change it?
by script (via devd)?
Master to Slave work fine, but how to config restore MASTER after restart.

change preempt =1 and set
Host A advskew = 1
Host B advskew = 10
and Host A is Master - ok.
Restart Host A and host B now is Master - ok.
Host A "back to life" but now he is SLAVE, and host B is master...

this situation is very, very bad
Host A should always be MASTER when is ON.

kzx

 

[SirDice]

SirDice - how you control who is MASTER and when change it?

ifconfig em0 vhid 1 state backup as explained in the handbook.

Master to Slave work fine, but how to config restore MASTER after restart.

If you want to automatically fall back enable preemption: sysctl net.inet.carp.preempt=1

 

[kazix]

ok, but not work

/etc/sysctl.conf on both host

net.inet.carp.preempt=1

restart Host A and host B.
A is Master, B is Backup.
Reboot host A, host B is now Master.
Host A "back to the life" but as Backup, not Master (host B is still Master).

if i give command on host A:

ifconfig lagg0 vhid 1 state master
...
ifconfig lagg0 vhid 5 state master

host A became Master and work

maybe you have right, better is change Master/Backup via command.
especially i want configure CARP + HAST and automatic changes can do very BIG MESS...

or maybe someone have CARP with preempt=1 and it works??

kzx

 

[SirDice]

maybe you have right, better is change Master/Backup via command.

Yeah, to be honest I generally keep it like that. If, for some reason, the master server has a problem CARP can bounce back and forth between the master and slave, not an ideal situation. It's fine to automatically switch to B if A is down but switching back is usually better done by hand.

 

[kazix]

ok, diabled prempt in /etc/sysctl.conf.

net.inet.carp.preempt=0

everything will be done via ifconfig...

but i see little problem (again) - reboot or power restore.
Host B is faster for starting up, and who first is Master - stay Master and have to change this only via ifconfig...
have to try slow down host B - or maybe think somthing else...

thanks
kzx

 

[dch]

This is what I've come up with, using the preempt sysctl.

- 2 FreeBSD 11.0p1 hosts running CARP
- 2 externally routable IPv4 and IPv6 addresses
- haproxy for app-level failover "bound" to those IP addresses

# /etc/rc.conf

# network
ifconfig_igb0="up"
ifconfig_igb1="up"
defaultrouter="123.45.9.1"
cloned_interfaces="lagg0"
ifconfig_lagg0="inet 123.45.9.3/28 laggproto lacp laggport igb0 laggport igb1"
ifconfig_lagg0_ipv6="inet6 abcd:1234:3000:4::3/64"
ipv6_defaultrouter="abcd:1234:3000:4::1"
# carp on
kld_list="carp"
ifconfig_lagg0_aliases="\
  inet  vhid 1 advskew 100 pass pass1 123.45.9.4/32 \
  inet6 vhid 2 advskew 100 pass pass2 abcd:1234:3000:4::4/64 \
  inet  vhid 3 advskew   0 pass pass3 123.45.9.5/32 \
  inet6 vhid 4 advskew   0 pass pass4 abcd:1234:3000:4::5/64"

on the other node, the advskew is on vhid 3 & 4.

On planned & unplanned restarts, the load is correctly distributed
cross both systems, and haproxy does a good job of load balancing
across the IPv4 and IPv6 pairs whether they are on a single node
(failover scenario) or both nodes (happy place).

I noticed that if the ifconfig_* stanzas are not 100% correct then
the carp setup may not work as expected. Double check!

I do think that auto-start up of services that are behind these proxies
needs careful consideration -- in my case, haproxy's liveness detection
ensures we are not opening up before things are ready.