CARP setup Hetzner (vSwitch)

nbari

nbari

I am testing CARP in two dedicated server from Hetzner having both servers atached to a vSwitch with IPv4 addresses, this is the what I have in /etc/rc.conf:

Code: 
cloned_interfaces="vlan0"
create_args_vlan0="vlan 4000 vlandev em0 mtu 1400 fib 1"
ifconfig_vlan0="ether random"

I need a VLAN in order to configure the public IPv4 from the vSwitch, I can't set the VIP as an alias to the main NIC.

And in /etc/rc.local (the same on both servers only difference is that server 2 uses advskew 100)

Code: 
ifconfig vlan0 inet 137.23.7.140/29 vhid 1 advskew 0 pass xxxyyyzzz 
route add default 137.23.7.137 -fib 1
sleep 3
if `ifconfig vlan0 | grep -q "carp: MASTER"`; then
    setfib 1 /usr/local/sbin/arping -i vlan0 -w 3 -U 137.23.7.137
fi

I am using /etc.rc.local, because I need to change the mac address of the VLAN, and in case of rebooting the sever I just want to ping the gateway if CARP is "MASTER", but this is indeed my question/problem. (I had to change the mac address of the VLAN because otherwise the mac address of the main host will advertize into the vSwitch and this is not allowed by Hetzner)

The setup works fine, but after approximately 5 minutes the vSwithh stop routing packets to the server holding the VIP, what I notice is that periodically (cron every minute) I need to advertise the mac address of the server holding the VIP ( arping) If I don't do this, the vSwitch stops routing packets.

Is there a way to periodically advertise the mac address? I tried without luck
Code: 
sysctl net.link.ether.inet.max_age=60

In both servers I have also this in /etc/devd/carp.conf

Code: 
notify 0 {
    match "system"      "CARP";
    match "subsystem"   "[0-9]+@[0-9a-z]+";
    match "type"        "(MASTER|BACKUP)";
    action "/root/carpcontrol.sh $subsystem $type | logger &";
};

This calls /root/carpcontrol.sh:

Code: 
if [ $# -lt 2 ]; then
    echo 'missing arguments subsystem and (BACKUP|MASTER)'
    exit 1
fi

IP_GW="137.23.7.137"

# arping and ping to speed up the switch
if [ $2 = "MASTER" ]; then
    setfib 1 /usr/local/sbin/arping -i vlan0 -w 3 -U ${IP_GW}
fi

It works fine, but the only If I keep using this cron:

Code: 
*/1 * * * * ifconfig vlan0 | grep -q 'carp: MASTER' && setfib 1 /usr/local/sbin/arping -i vlan0 -w 3 -U 137.23.7.137 > /dev/null 2>&1

Wondering what could be wrong that I need to keep advertising/pinging and if there is a way to prevent this.

Any thoughts?
 
probably your vlan mac expires in gw's arp cache and it's request is not answered
do you have a route for 137.23.7.136/29 via vlan0 in fib0 ?
if not try to add one
 
Jose said:
This is exactly what CARP does. It periodically advertises a MAC address.


Try using tcpdump(1) to see what changes after 5 minutes.
Click to expand...
Hi, the provider told me that multicast and real multicast aren't possible inside the vSwitch for technical reasons and that all packets will be handled like broadcast inside the vSwitch.
What I notice via tcpdump after the vSwitch forgets the mac address are messages like:

Code: 
  f2:0b:a4:d1:20:01    Broadcast    ARP    56    Who has 137.23.7.140? Tell 137.23.7.137

I also notice that CARP is indeed announcing to address as expected to 224.0.0.18 (VRRP) wondering if there could be a way to use like UNICAST like keepalive I think in where I could define the endpoint to send the announcements?
 
covacat said:
probably your vlan mac expires in gw's arp cache and it's request is not answered
do you have a route for 137.23.7.136/29 via vlan0 in fib0 ?
if not try to add one
Click to expand...
Hi, I can't I get :

Code: 
fib 0: Network is unreachable
 
VladiBG said:
CARP is based on multicast. If your provider doesn't support it then don't use CARP.
Click to expand...
Agree, but partially works if using arping periodically or using a private IP.

Any alternatives or suggestions? my goal is to configure an HAproxy in HA mode having a single VIP.
 
covacat said:
try route add -net 137.23.7.136/29 -interface vlan0 -fib 0
Click to expand...
Thanks, that created the route but still not working, I need to call arping to make it work:

Code: 
/usr/local/sbin/arping -i vlan0 -w 3 -U 137.23.7.137
 
try without fibs first
just set a static route to some external box via 137.23.7.137 and see if that scenario works
 
covacat said:
try without fibs first
just set a static route to some external box via 137.23.7.137 and see if that scenario works
Click to expand...
I tried without luck:

Code: 
route add 137.23.7.143 137.23.7.137 
and

route add 137.23.7.143 137.23.7.137 -fib 1

From my understanding having the routes should allow replying to the broadcast messages from the switch?
 
Something that I notice, is that if I don't use CARP and just fix/configure the VIP, when running arp -an the mac is present as "vlan0 permanent" (I don't need to arping, it works fine without problems)

Code: 
$ setfib 1 arp -na
? (137.23.7.137) at f2:0b:a4:d1:20:01 on vlan0 expires in 1199 seconds [vlan]
? (137.23.7.140) at e2:0c:ec:f6:f2:36 on vlan0 permanent [vlan]

But when using CARP this is the output, notice there is no entry for the VIP, only for the GW:

Code: 
$ setfib 1 arp -na
? (137.23.7.137) at f2:0b:a4:d1:20:01 on vlan0 expires in 1199 seconds [vlan]

After running "arping -i vlan0 -w 3 -U 137.23.7.137", things start to work, this is part of the tcpdump output:

Code: 
e2:0c:ec:f6:f2:36     Broadcast    ARP    58    Who has 137.23.7.137? Tell 137.23.7.140 (duplicate use of 137.23.7.140 detected!)
`--- [Duplicate IP address detected for 137.23.7.140 (e2:0c:ec:f6:f2:36) - also in use by 00:00:5e:00:01:01 (frame 368)]
...

fe:6e:94:89:c7:c9    JuniperN_a5:f7:e4    ARP    42    137.23.7.140 is at 00:00:5e:00:01:01

Something interesting that I notice is that for some reason the MAC address that was advertised is 00:00:5e:00:01:01 but that is not the mac address of the vlan0 interface: e2:0c:ec:f6:f2:36 . (this could explain why arping works since it is announcing the mac of the VLAN, but also because of that I get the duplicate use of <IP> detected!

Any idea about how to make CARP send the MAC address of the current interface used, in this case, a VLAN?
 
Jose CARP announcements between the nodes are send using multicast. In pfsense you can overwrite it to use unicast by specifying the other node address. The announcement of the active (master) node toward the LAN segment is done using gratuitous arp. Other protocols like HSRP and VRRP works on the same principle.
 
VladiBG said:
Jose CARP announcements between the nodes are send using multicast. In pfsense you can overwrite it to use unicast by specifying the other node address. The announcement of the active (master) node toward the LAN segment is done using gratuitous arp. Other protocols like HSRP and VRRP works on the same principle.
Click to expand...
How could this be done with pf ? or it is something unique to pfsense?
Currently, I am using arping to do the gratuitous ARP, and pf route-to because I have the VIP in another route (fib 1) something like this:

Code: 
pass in on vlan0 reply-to (vlan0 137.23.7.137) inet proto tcp from any to any port = https flags S/SA keep state

So far is working, but I would like to have a cleaner setup to get rid mainly of the "duplicated mac addresses message" and the arping that I have in a cron
 
VladiBG said:
Jose CARP announcements between the nodes are send using multicast.
Click to expand...
You're absolutely right. I was being ignorant. The CARP advertisements are multicast on the local subnet. Details on this are hard to find. This is the best explanation I found, and it's for Openbsd CARP:
www.packetmischief.ca

OpenBSD CARP Notes

CARP is the Common Address Redundancy Protocol. It's a secure, free alternative to the Virtual Router Redundancy Protocol and the Hot Standby Router Protocol. CARP was created and is maintained by the OpenBSD project. The notes here apply to OpenBSD 5.0 and higher. Protocol Information Virtual...
www.packetmischief.ca www.packetmischief.ca

CARP absolutely will not work without multicast. In its absence, both nodes will think they're the master because they can't "hear" the backup, and will send gratuitous ARP to grab the VIP. This explains the duplicate IP address messages Nbari is seeing. Both nodes are trying to grab the VIP at the same time. I'm not sure how ARP handles conflicts like this. It might be undefined behavior.
 
nbari said:
How could this be done with pf ?
Click to expand...
It's not about the firewall (pf) it's how the CARP is modified in pfsense. There's the same discussion from 2020 on netgate forum regarding the vSwitch and CARP.
forum.netgate.com

CARP issues due to Layer 2 switching

Hi all, I am having an issue with CARP using Hetzner's vSwitch to connect servers across different DC's. As CARP wasn't working out of the box, talked to the...
forum.netgate.com forum.netgate.com

If the vSwitch handles gARP properly but not allows multicast then you need another way to connect the nodes together so they can talk to each other. Maybe with gif or gre tunnels between them.

Here's another example from OpenBSD and syncpeer :

OpenBSD PF: Firewall Redundancy (CARP and pfsync)

www.openbsd.org
 
Jose said:
You're absolutely right. I was being ignorant. The CARP advertisements are multicast on the local subnet. Details on this are hard to find. This is the best explanation I found, and it's for Openbsd CARP:
www.packetmischief.ca

OpenBSD CARP Notes

CARP is the Common Address Redundancy Protocol. It's a secure, free alternative to the Virtual Router Redundancy Protocol and the Hot Standby Router Protocol. CARP was created and is maintained by the OpenBSD project. The notes here apply to OpenBSD 5.0 and higher. Protocol Information Virtual...
www.packetmischief.ca www.packetmischief.ca

CARP absolutely will not work without multicast. In its absence, both nodes will think they're the master because they can't "hear" the backup, and will send gratuitous ARP to grab the VIP. This explains the duplicate IP address messages Nbari is seeing. Both nodes are trying to grab the VIP at the same time. I'm not sure how ARP handles conflicts like this. It might be undefined behavior.
Click to expand...
Interesting that in my case they do work, indeed one server is in BACKUP and the other as MASTER, also when doing:

Code: 
ifconfig vlan0 vhid 1 state master
or
ifconfig vlan0 vhid 1 state backup

They work as expected so I don't know if this is something within the vSwitch that partially supports it or the CARP states are only broadcasted.
 
multicast is blocked in many environments, therefore wondering if there is another alternative or if implementing something on top of broadcast could work? just wondering if when broadcasting one can send the vhid, password, etc, some data that could give room to create/start drafting something, ideally this should be at layer 2 but maybe something interesting for starting a project.
 
from what i gathered incoming packets should come to the virtual carp mac 00:00:5e:00:01:VHID and outgoing should have the real nic mac addr and this may confuse/or be denied in some configuration
i don't have a carp install to test
 
I only see the VLAN mac address (when using ifconfig) but when using tcpdump I notice 00:00:5e:00:01:VHID
 
VladiBG said:
It's not about the firewall (pf) it's how the CARP is modified in pfsense. There's the same discussion from 2020 on netgate forum regarding the vSwitch and CARP.
forum.netgate.com

CARP issues due to Layer 2 switching

Hi all, I am having an issue with CARP using Hetzner's vSwitch to connect servers across different DC's. As CARP wasn't working out of the box, talked to the...
forum.netgate.com forum.netgate.com

If the vSwitch handles gARP properly but not allows multicast then you need another way to connect the nodes together so they can talk to each other. Maybe with gif or gre tunnels between them.

Here's another example from OpenBSD and syncpeer :

OpenBSD PF: Firewall Redundancy (CARP and pfsync)

www.openbsd.org
Click to expand...
I created a gif/gre tunnel, which works but I can't find a way to add an alias or use a tap interface so that I could use CARP on it, any ideas?

In both cases I get:
Code: 
ifconfig: SIOCGVH: Operation not supported
 
For some reason, the TAP interface seems to be always down, and CARP remains in INIT
 
You must log in or register to reply here.
Back
Top