FreeBSD unable to use multicast IP as network gateway.

C

creepingfur

Recently we upgraded our gateway/NAT boxes to an active/active setup using OpenBSD/CARP. Because it's active/active - the IP used for the gateway is multicast.

It seems that after the change, every FreeBSD host on the network (physical and virtual - 9.x and 10-CURRENT) cannot ping/ traceroute/etc. - let alone actually connect through the gateway via IPv4 (we don't use IPv6 internally - so it hasn't been tried). More so - it seems completely isolated to FreeBSD. Everything from NetBSD, OpenBSD, Linux, OS X, Ubiquiti APs, VXwork, and even Windows all work fine. No issues.

If we point FreeBSD to the individual nodes in the cluster - it works fine, but the CARP IP - it just doesn't work. Even a basic install - no firewalls or anything enabled - exhibits the same behavior.

We've tried everything we know of - I've found nothing in sysctl that seems to help. The only error we get - and it's spewing to the logs pretty constantly is:

Code: 
in_arp: 01:00:5e:00:01:06 is multicast (which, we of course know)

I've searched the forums/Google and haven't found anything that helps (I've seen stuff regarding similar issues with certain load balancing setups, but none have helped). Does anyone know a solution or has had this problem before? We can work around it - but we would rather use the HA/CARP IP than individual nodes in the cluster.

Cheers and thanks,
-bp
 
You should assign a regular unicast address to the CARP interface. CARP uses multicast to communicate with the other hosts but the interface itself should have a unicast address.
 
I was also curious about that.

So I did some further looking - the other OSes (at least Open/NetBSD and OS X - though as all other tested non-FreeBSD OSes work - I figure they don't either) - don't think it's a multicast address (I just assumed it was multicast because FreeBSD thinks it is). I should have looked a little bit closer at the systems not having problems - to them it's your standard unicast gateway address. Nothing strange. Nothing out of the ordinary. No functional problems or issues.

If look at what is basically my carp configuration:
Code: 
inet xx.xx.xx.xx 255.255.255.0 xx.xx.xx.255 balancing ip carpnodes 32:0,33:100 carpdev em0 pass blahblahblah
It's all very generic and boring. It's something I've done many times before. I also tried switching the balancing mode to arp - it doesn't seem to have any effect at all.

So now I'm REALLY curious - why the heck does FreeBSD (9.x and 10-CURRENT, virtual and physical) think it's multicast and every other OS doesn't? And how do I get it not to think that and/or work & behave correctly.
 
I figured out a solution (on the CARP side - FreeBSD really doesn't seem to like multicast gateways and I don't think there is much of a way around it).

I re-read the OpenBSD CARP man page. It appears that CARP has four load balancing modes - arp (which I know won't work on my network because of certain thing like wireless bridges), ip (multicast, what I was using), ip-unicast and ip-stealth.

I switched CARP first to ip-unicast. OpenBSD remained unaffected. FreeBSD was once again happy (wired and wireless). But ICMP (outside the local subnet), traceroute broke, etc on OS X (wireless only - wired hosts where fine, as where, oddly, iOS wireless - they usually), NetBSD (wired) behaved like wireless OS X clients and had some UDP issues - as did Linux hosts (wireless clients being noticeably worst).

So I switched it to ip-stealth, which uses the same faked MAC address as ip-unicast, but CARP never sends packets with its virtual MAC address as source. Switching to that made all the servers happy and content, though caused a little heartburn with a Juniper switch - but that's because someone configured it funny and that was a quick fix to resolve.

But - that all said - freeBSD FreeBSD will not work with a multicast gateway (or I never found a way), despite most everything else being able to, which makes me wonder how some VRRP shops or places with certain types of load balancers don't run into issues.

Thanks to all for all the sanity checks.
 
You must log in or register to reply here.
Back
Top