CARP and HA for routers

We are using FreeBSD server as a router to the upstream provider, for use by a dozen of servers. All servers have real IP addresses.

For example:

Internet <> upstream provider < 11.12.13.1/29 -- 11.12.13.2/29> our gateway <33.44.55.1/27> our servers 33.44.55.0/27

On non-gateway servers, 33.44.55.1 is declared to be the gateway. The gateway has two interfaces, and the upstream one uses 11.12.13.2, with 11.12.13.1 being the gateway.

Now, we'd like to survive the death of the gateway machine.

The general idea is to equip another server with the second Ethernet interface, and allow it to, when the active gateway dies:

1. bring the second interface up and start to serve 11.12.13.2;
2. start to act as a gateway;
3. steal 11.12.13.1 from the dead gateway by aliasing on the 'internal' interface, and start to act as a gateway;
4. continue to do its own serving job with the original IP address on the 'internal' interface.

Of course, we assume that the second interface on the second server is already wired to the same switch as gateway and upstream provider.

Is CARP an adequate tool for this? Any pointers to using CARP for HA of gateways?

BondedBSD said:

Is CARP adequate tool for this? Any pointers to using CARP for HA of gateways?

Click to expand...

CARP would be excellent for this. With it you create a 'virtual' IP address that's bound to both gateways. The clients all use the virtual address instead of the physical addresses of the gateway. And CARP will accept the data on one of them, if the machine dies CARP automatically switches to the other machine.

30.10. Common Address Redundancy Protocol (CARP)

Note: For best results, use FreeBSD 10.0 or newer. CARP support was re-worked in FreeBSD 10.0, and it is so much nicer to work with than in previous releases. We're using it on two sets of routers. One set is doing VLAN tagging and routing across 20-odd VLANs/subnets, with some Dummynet traffic shaping. The other set is doing IPFW packet filtering (stateless) and routing across four gigabit NICs. Everything is working nicely, and fail-over is virtually instantaneous (nobody notices when things fail-over).

I tried and tried and tried to get it working on previous versions of FreeBSD using the built-in CARP, ucarp from ports, and freevrrpd from ports, and could not get things to work reliably. And it wasted a lot of IPs in the process.

Use FreeBSD 10+ for any CARP setups.

phoenix said:

I tried and tried and tried to get it working on previous versions of FreeBSD using the built-in CARP, ucarp from ports, and freevrrpd from ports, and could not get things to work reliably. And it wasted a lot of IPs in the process.

Click to expand...

I have it working reliably on 9.x. I'm using it to on two HAProxy machines to provide fail-over. I actually used it yesterday to upgrade both machines to 9.3-RELEASE without interruption to the users. I upgraded the 'backup' first and then triggered a failed-over to do the other one. Nobody noticed anything.

For how many IPs and interfaces?

I was trying to get it to work on a router with four physical NICs and 20-odd VLANs. Routing and IPFW-based traffic shaping (strictly Dummynet and routing, no packet filtering or NAT). I spent a good, solid week working on it. Eventually I got a working setup with freevrrpd, but it was fragile. I could not get a stable setup. This was with FreeBSD 8.x and 9.0. Too many carpX interfaces and wasted IPs.

I upgraded to 10-CURRENT (January 2013) and had a working setup the first day. It's been in production since February 2013. It's failed over (and back) several times since without anyone noticing.

I replicated a similar setup using 10.0-RELEASE on a firewall with four physical NICs including proper double-NAT between private networks, using CARP for all public IPs. /etc/rc.conf.local is very nice and clean compared to what the previous CARP setup would require.

Many people have managed to get CARP working on pre-10.0 systems. But I haven't heard of any of them actually liking how it works. And very few who have upgraded to 10.0 have complained about the nicer way CARP works.

If you're starting from scratch, I'd recommend starting with 10.0 and just avoiding the old way of doing things.

phoenix said:

For how many IPs and interfaces?

Click to expand...

We don't need any firewalls or traffic shaping. All machines should be directly visible from the public Internet, so we need to route packets from/to upstream provider to 30 or so machines. The reason we need to do this is that upstream provider doesn't want to - they will just send packets destined to all our public IPs to one IP, which they call router.

I don't want to introduce two new expensive boxes (two - because we need HA, expensive - we are talking 1 Gbps cumulative traffic), so the idea is to use one of the servers to do routing as described above (interfacing upstream provider on one NIC, and relaying to the other NIC, to which all other servers are connected and use it as a default route), and another server to be a backup router if the first one dies (there is no third one - we do not protect against double hardware failure).

This is relatively straightforward to do manually, but automated failover is better (it's expensive to add live DevOps eng to the cage).

I am uncomfortable with CARP, as it relies on the single probe to determine the death of the machine. Servers can die in many ways, and we usually use 2-3 independent monitoring methods which all have to agree.

Plus:

• We would have to run CARP on both sides, and there is nothing there to guarantee that they both fail over simultaneously (i.e. yanking an Ethernet cable from the the upstream NIC would be a catastrophe, as 'outside' CARP would fail over, but the 'inside' one would not, and packets from servers would end up nowhere).
• There is CARP traffic and CARP IPs that open potential attack surface.

I guess what we really need is a system to unambiguously and reliably determine real death of one server, make sure it stays dead forever, and communicate this fact to the backup server. The rest is easy.