VIP / Carp issues with FreeBSD VMs on Linux KVM Hypervisors

I

iRobbery

I two linux hypervisors (CentOS) with each a freebsd VM with carp setup. The VRRP part works fine, switching backup <-> master works as expected. However the VIP on the master it does not seem to work.

I have been trying to get it to work for a longer time now, I've read many forum posts and tried many settings and suggestions I could find, and just now I figured out that if I use arping on the master freebsd VM with the VIP as source address to somewhere, gateway, broadcast address e.g. in the network, then it works, until I stop arping then it lingers for about ten seconds max and it will cease to work again.

I'm looking forward for suggestions to try and or check.
Many thanks in advance.
 
iRobbery said:
until I stop arping then it lingers for about ten seconds max and it will cease to work again.
Click to expand...
This sounds like the MAC address is timing out on the 'virtual' switch. Which begs the question, how are the interfaces and networking configured on the host side?

With VMWare and Virtualbox you need to put the virtual interface in 'promiscuous' mode for CARP to work for example.
 
This is just linux libvirtd / KVM, no VMware or virtual box, I did find all the posts about this and tried to set all interfaces involved in promiscuous mode. Though no change unfortunately.

And the ip only temporarily works if I arping with the Mac address of the virtio interface not the CARP 00:00:5e:00:01:x Mac address.
 
I suspect either the linux bridges don't forward CARP packets properly or the host/hypervisor or switch can't cope with the mac address being taken over and/or the CARP interfaces actually having 2 mac addresses (the virtual and the shared mac). have you checked what happens to the traffic on the bridges during a CARP failover via tcpdump or wireshark? can you check the arp cache on the switch?

I'm running several OpenBSD routers with multiple CARP'ed interfaces on bhyve and the only thing I needed to set to get CARP working reliably was disabling the net.link.bridge.pfil_onlyip sysctl, otherwise after a failover the peers would constantly (and asynchronously) switch between MASTER and SLAVE as if they were loosing connection to each other. This was only resolved after rebooting both VMs and could be triggered again by a failover.

However, I once also had problems with CARP failover on some pesky "smart" switch that refused to acknowledge that the mac address changed from one port to another. It took up to 2 minutes until it was finally sending packets destined to the virtual ip/mac to the port of the failover host. So ARP caching is another thing I'd check when seeing such problems (hence the request to check what happens to the traffic during a failover - does it actually reach the failover host?)
 
Thanks for your suggestions and help so far.

I'll be doing some more testing, but it almost seems as if the current KVM hosts doesn't consider the virtual mac address as something valid, simply ignores it.
I got a spare machine to do some testing with, see if I can reproduce and more easily can change KVM settings as the others are live and have other production systems on it as well.

For now on te live setup, I see the arp messages coming by (ip addresses altered for 'privacy' :) )
.234 is a ssh host
.254 is the gateway

Code: 
22:04:22.026053 ARP, Request who-has 6.1.2.83 tell 6.1.2.234, length 46
22:04:22.026582 ARP, Reply 6.1.2.83 is-at 00:00:5e:00:01:64, length 28
22:04:53.911119 ARP, Request who-has 6.1.2.83 tell 6.1.2.254, length 46
22:04:53.911389 ARP, Reply 6.1.2.83 is-at 00:00:5e:00:01:64, length 28
22:05:01.026051 ARP, Request who-has 6.1.2.83 tell 6.1.2.234, length 46
22:05:01.026252 ARP, Reply 6.1.2.83 is-at 00:00:5e:00:01:64, length 28
22:05:40.026075 ARP, Request who-has 6.1.2.83 tell 6.1.2.234, length 46
22:05:40.026284 ARP, Reply 6.1.2.83 is-at 00:00:5e:00:01:64, length 28

But if I then do an ip neigh show / arp -n on the KVM host, the 6.1.2.83 shows as incomplete

Code: 
[root@vmh06 ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
6.1.2.56            ether   52:54:00:31:ed:eb   C                     vlan100
6.1.2.10            ether   52:54:00:e0:fc:3f   C                     vlan100
6.1.2.199           ether   82:52:85:6f:6d:a6   C                     vlan100
6.1.2.234           ether   52:54:00:bb:1b:2e   C                     vlan100
6.1.2.83                    (incomplete)                              vlan100
6.1.2.254           ether   b2:94:fb:96:bd:6c   C                     vlan100
6.1.2.54            ether   ac:1f:6b:b3:e8:d6   C                     vlan100
[root@vmh06 ~]# ip neigh show
6.1.2.56 dev vlan100 lladdr 52:54:00:31:ed:eb STALE
6.1.2.10 dev vlan100 lladdr 52:54:00:e0:fc:3f STALE
6.1.2.199 dev vlan100 lladdr 82:52:85:6f:6d:a6 STALE
6.1.2.234 dev vlan100 lladdr 52:54:00:bb:1b:2e REACHABLE
6.1.2.83 dev vlan100  FAILED
6.1.2.254 dev vlan100 lladdr b2:94:fb:96:bd:6c STALE
6.1.2.54 dev vlan100 lladdr ac:1f:6b:b3:e8:d6 STALE

The switch connected to the KVM hosts I have inspected many times, but it doesn't have anything really to check except VLANS. And the issue applies to connectivity between VM hosts on the same KVM host, so I suspect that would not involve the switch in anyway. Funny how these weird issues make you second guess your knowledge about everything you knew so well already :)
 
On illumos/smartOS one can set the properties "nics.*.allow_mac_spoofing" and "nics.*.allow_unfiltered_promisc" for VMs (KVM and bhyve), which are related to the guest being able to use multiple/custom mac addresses.
from the vmadm manpage:
Code: 
       nics.*.allow_mac_spoofing:

           With this property set to true, this VM will be able to send packets
           from this nic with MAC addresses that don´t match the mac property.

           type: boolean
           vmtype: ANY
           listable: yes (see above)
           create: yes
           update: yes
           default: false

       nics.*.allow_unfiltered_promisc:

           With this property set to true, this VM will be able to have multiple
           MAC addresses (eg. running SmartOS with VNICs).  Without this option
           these packets will not be picked up as only those unicast packets
           destined for the VNIC´s MAC will get through.  Warning: do not enable
           this option unless you fully understand the security implications.

           type: boolean
           vmtype: HVM
           listable: yes (see above)
           create: yes
           update: yes
           default: false

I tried to find some explanation what options those properties actually corelate to in the hypervisors, but wasn't able to find anything. However, searching for "KVM mac spoofing" it seems most questions are about how to actually *do* mac filtering or prevent spoofing, suggesting that by default KVM doesn't even care what MAC addresses the guests are using.
Given that smartOS runs everything in zones and anything in zones (as with jails) works on a "default deny/restrict" basis (as it should be), this might be specific to illumos/smartOS but not apply to KVM or linux in general.

However, searching for KVM and promisc mode, on linux this seems to require the host to set all involved (physical) interfaces to set to promisc mode so it will work for the guest. Might be worth a try?

TBH this looks very much like a linux/KVM-specific problem with the host not properly forwarding packets to the VM if it uses multiple MACs on the same interface; so I'm not sure if you can get find much help here...
 
I suspect this MacVTap thing might be interfering. Have you tried setting it up with the 'traditional' bridge and tap/tun interfaces?
 
You must log in or register to reply here.
Back
Top