Solved Routing on carp, interface question

Hello guys,

I hope to get some suggestions from you.

I have multiple FreeBSD machines currently running on 10.0-RELEASE-p9.
2 machines running with a CARP setup and provide the Internet/WAN Connection. They are connected to the ISP router. The ISP provides a x.x.67.0/30 subnet for the WAN Interface (one address is for my gateway and the other address is the ISP gateway). Thankfully the new CARP protocol can now handle this.

I learned that the new CARP implementation is very different from the old one. I'm still a bit confused about the /32 subnet mask for the CARP alias. But I could set[]up a working configuration. The master rc.conf for the igb0 (WAN Interface) looks like this. Unnecessary output is omitted:
Code:
# igb0 WAN Interface
ifconfig_igb0_alias0="vhid 100 advskew 0 pass Test alias x.x.67.2/32"

# Static Routes
static_routes="default"
route_default="-net 0.0.0.0 -interface igb0"
With this setup the backup host fails over if the master goes down or if a fail-over is triggered manually.

My problem was the defaultroute option. The default route was not set. When the CARP interface is in backup mode the routing entry is deleted or not present.

I added the route like you see in the rc.conf above on master and backup host.

Routing table looks like this now:
Code:
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            00:1b:21:45:23:A0  US          0  4473175   igb0
      x.x.62.2    link#5             UHS         0        0    lo0 =>
      x.x.62.2/32 link#5             U           0        0   igb0

Would you prefer a other way to get a functional routing?

I'm asking because I have another couple of FreeBSD machines which provide firewalling and routing for internal networking with link aggregation and VLAN tagging. On these machines I want to set[]up CARP like described.

On these machines it would also be necessarily to add manually static routing entries for every VLAN interface?

On FreeBSD 9 the routing table was "automatically" build because every interface was configured with an IP address and then the virtual CARP device was created. So the system knows for example that network 192.168.10.1/24 was connected via vlan10.

It would be great to get some suggestions. Thanks a lot!
 
Last edited by a moderator:
Usually I would assign a unique IP address to each host, then add a CARP alias to each with a third, shared address. Obviously this isn't possible when you only have a /30, and without testing I don't have an exact answer, although I have a few suggestions.

Based on the ISP using x.x.67.1 on their end, and you having x.x.67.2 to work with.
  1. The static route looks strange to me. May be required, but I would expect it to just have x.x.67.1 as the default gateway
  2. If you only have one address to work with, I can't see how using an alias will work. Specifying alias in the ifconfig command suggests there's already an IP assigned to the interface, which there can't be if you only have 1 address.

The FreeBSD carp(4) man page does show some examples where the alias keyword isn't used so I'm assuming here that it's possible to assign just a CARP address to an interface, without any other address assigned. If that's the case, try the following:

Primary:
Code:
ifconfig_igb0="vhid 1 pass password x.x.67.2/30"
defaultrouter="x.x.67.1"

Backup:
Code:
ifconfig_igb0="vhid 1 advskew 100 pass password x.x.67.2/30"
defaultrouter="x.x.67.1"

The ifconfig lines above taken from the example in the man page. If you have a private interface on those servers that provide the gateway to your network, that will likely need a carp address as well. The examples in the man page show this, as well as the required net.inet.carp.preempt sysctl to make all the interfaces failover together (so when the backup takes over the IP for the ISP link, it also starts accepting traffic from the lan).
 
Last edited by a moderator:
1) You are right. But it was the only option that worked.

2) That was the same opinion I had after reading man pages.

Your suggestion doesn't work. It was also my first try. Because CARP uses an alias with /32 subnet mask the routing table had no entry for the x.x.67.1/30 address. The machine doesn't know that network.
And when I try to add the route I get following:
Code:
add net default: gateway x.x.67.1 fib 0: Network is unreachable

I know the parameter net.inet.carp.preempt and it's set.
 
Last edited by a moderator:
Hmm, I've just tested on some VMs and I was able to get two 10.0 machines to accept the following commands fine using my 192.168.0.0/24 LAN as a test:

Code:
ifconfig em0 vhid 1 pass mypass 192.168.0.100/24
route add default 192.168.0.10

Code:
ifconfig em0 vhid 1 advskew 100 pass mypass 192.168.0.100/24
route add default 192.168.0.10

Neither complained when adding the default route. The backup machine was unable to ping the gateway (expected as it has no other address other than the CARP address which was in backup mode), but the pings magically started working a few seconds after downing the interface on the master.

The "Network is unreachable" error suggests that the IP isn't assigned to the interface, or has been assigned with a /32 mask.

The only issue I had was the master VM couldn't ping out after being in backup mode then put back into master mode although that could be some issue with my virtual machines.

Edit: Although all the commands were accepted and it appears to function somewhat, it doesn't seem to work robustly either with preempt on or off. The backup seems to work fine and take over, but the master seems to regularly lose network connectivity when switching modes. I don't know if the new carp system is just buggy when used like this or if it's something to do with my VMs)

Edit 2: Seems to be more stable with both VMs in Virtualbox. I've actually had it switch between the two successfully a few times now. In fact it seems to have become fairly stable with both preempt on and off.
 
I ended up with the following configuration on my VMs. This seems to work as expected for me. If both my hosts (master and backup) accept a gateway of 192.168.0.10, I can't see why yours shouldn't accept x.x.67.1, seeing as it's a valid address in the x.x.67.0/30 network.

/etc/sysctl.conf
Code:
net.inet.carp.preempt=1

/etc/rc.conf
Code:
ifconfig_em0="vhid 10 pass mypass 192.168.0.100/24 up"
defaultrouter="192.168.0.10"

Exactly the same configuration as above on backup host although the ifconfig line changed as follows:
Code:
ifconfig_em0="vhid 10 advskew 100 pass mypass 192.168.0.100/24 up"

I found that without the up keyword on the end, I would get a
Code:
carp: demoted by 240 to 240 (interface down)
message during boot and it would stick in INIT mode even though the interface was up. I had to down/up the interface to get it to start. With up on the end it works as expected. The master boots into master mode and the backup boots straight to backup mode. The backup takes over when the master is down, and because of the preempt setting, if I bring the master back online it immediately takes over again. (Without preempt the backup stays master until it goes down, at which point the original master takes over again).
 
I think the confusion about the subnet mask is because it's advised to have a /32 netmask on an alias. But because the CARP address is the only address on that interface it should be set as the primary address using the /30 netmask. Normally you'd use three IP addresses, one for each "leg" and one for the CARP address. In which case the CARP settings would be set on an alias of the interface. I had a look on one of our machines that uses CARP and I've used the "normal" subnet mask for it. But this is on FreeBSD 9.3, on 10.0 and higher it may be different.
 
Yes, when you assign an address such as 192.168.0.1/24 to an interface, FreeBSD now knows that the entire 192.168.0.0/24 network is accessible via that interface and will add a route similar to the following:

Code:
192.168.0.0/24    link#1             U           0   624593    em0

(Quite simply, packets to that /24 network can be delivered directly via em0)

When you then add an alias, either for carp or just a normal alias, you don't need to tell FreeBSD the network again, so you use a /32. If it was an IP in a different range, you would need the correct mask so that FreeBSD could correctly route that second network to the interface.

The primary address on an interface has to have the correct mask, whether it's for carp or not.
 
Thanks for the many tips! I got it now.

I misunderstood the handbook and thought the carp alias had to be a /32 subnet.
Like you said this makes only sense when the physical interface ("leg") had also an IP address with the network subnet mask assigned.
In my case the CARP alias is the one and only address assigned. Then it must have the subnet mask of the connected network (a /30).

now the rc.conf look like this:

Code:
defaultrouter="x.x.67.1"

# igb0 WAN Interface
ifconfig_igb0_alias0="vhid 100 advskew 0 pass Test alias x.x.67.2/30"

netstat -nr
show this:

Code:
Internet:
Destination  Gateway  Flags  Refs  Use  Netif Expire
default x.x.67.1  UGS  0 17177160  igb0
x.x.7.0/30 link#5  U  0  0  igb0
x.x.67.2  link#5  UHS  0  0  lo0

The routing table looks much better and the machine is working!
 
Is that all you have in /etc/rc.conf?

If it's working then it may be best left alone but I still don't see a need to use _alias0 or the alias keyword.
In /etc/rc.conf, the first address is set using ifconfig_X. If you then want a second address (alias), you use ifconfig_X_alias0, then ifconfig_X_alias1 and so on.

I think you're still getting mixed up between FreeBSD 9 CARP configuration, and FreeBSD 10 CARP configuration.

On FreeBSD 10, if the CARP address is the first (and only) address, then you should be able to set it just using ifconfig_X:
Code:
ifconfig_igb0="inet x.x.67.2/30 vhid 100 pass Test"
I don't believe the inet keyword is required but may as well add it. Also advskew defaults to 0 so you don't need to set that unless you actually want it greater than 0 (i.e. on your backup host).
 
You are right. Thanks!
The manual clearly says that only a vhid and a IP is necessary at minimum...
 
Back
Top