CARP auth not working on 11.2-RELEASE-p9

asv0 · Feb 25, 2019

Good morning to you all,
I'm experiencing an annoying issue with the new CARP implementation (changed since FBSD 10+) on two identical FreeBSD 11.2-RELEASE-p9 servers.
Here the configuration strings in /etc/rc.conf (password same length and type but modified):
SRV1: inet vhid 1 advskew 0 pass add44c97ec0dcd4568b63770c1fe11ef alias 10.2.2.130/32
SRV2: inet vhid 1 advskew 100 pass add44c97ec0dcd4568b63770c1fe11ef alias 10.2.2.130/32

--- IFCONFIG OUTPUT
SRV1:

Code:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>                                              
        ether 1e:44:d1:79:bb:2e
        hwaddr 1e:44:d1:79:bb:2e
        inet 10.2.2.134 netmask 0xfffffe00 broadcast 10.2.3.255
        inet 10.2.2.130 netmask 0xffffffff broadcast 10.2.2.130 vhid 1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        carp: MASTER vhid 1 advbase 1 advskew 0

SRV2:

Code:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 02:ec:50:e2:9c:16
        hwaddr 02:ec:50:e2:9c:16
        inet 10.2.2.128 netmask 0xfffffe00 broadcast 10.2.3.255 
        inet 10.2.2.130 netmask 0xffffffff broadcast 10.2.2.130 vhid 1 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        carp: BACKUP vhid 1 advbase 1 advskew 100

As I did't enable the preemption I'm testing the failover manually and it works well just as connectivity to the service IP .130 does.
Everything works very well BUT authentication!
As I've read around some apparent issues with passwords longer than 32 chars and I remember years back having troubles with a similar issue for a redundant PBX setup, I've tested different kind of passwords:

8-10 chars random with special chars
8 chars random no special chars
8 chars only letters (big/small)
8 chars only letters (no capitals)
no password at all

The result is always the same. Therefore, having in production 2 PFsense clusters at their latest version (2.4.4-RELEASE powered by FreeBSD 11.2-RELEASE-p6) I've been digging a bit more and I've realised that CARP authentication "works" on the PFsense cluster.
I've dissected the advertisement packets on both, sanitizing with "--" the public IP addresses present in the packets, and here is what I see:

--- Packet on FreeBSD, fields of interest between squared brackets

Code:

22:57:50.622956 IP 10.2.2.128 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36
        0x0000:  45 10 00 38 00 00 40 00 ff [70] 8e b1 [0a 02 02 80]  E..8..@..p......
        0x0010:  [e0 00 00 12] [21] 01 64 07 [00] 01 8c 06 -- -- -- --  ....!.d.........
        0x0020:  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  ................
        0x0030:  -- -- -- -- -- -- -- --                      ........

70 = Protocol VRRP
0a 02 02 80 = Source IP Address
e0 00 00 12 = Destination IP Address
21 = VRRP version 2
00 = Auth Type: NO AUTHENTICATION

--- Packet on PFsense, fields of interest between squared brackets

Code:

23:14:19.556562 IPv4 (0x0800), length 60: XX.XX.XX.XX > 224.0.0.18: VRRPv2, Advertisement, vrid 101, prio 105, authtype simple, intvl 3s, length 20
        0x0000:  45 c0 00 28 00 00 00 00 ff [70] 56 20 [-- -- -- --]  E..(.....pV.....
        0x0010:  [e0 00 00 12] [21] 65 69 01 [01] 03 ce ff -- -- -- --  ....!ei.........
        0x0020:  [6e 4f 63 6e 4f 63 00 00] 00 00 00 00 00 00       nOcnOc........

70 = Protocol VRRP
-- -- -- -- = Source IP Address
e0 00 00 12 = Destination IP Address
21 = VRRP version 2
01 = Auth Type: Simple Text Authentication [RFC 2338] / Reserved [RFC 3768] (1)
6e 4f 63 6e .... = Auth string: nOcnOc
____
The output is quite clear, the FreeBSD packets are not using any authentication while PFsense does BUT the simple text one and not the SHA1-HMAC as expected and I'd love to know why.

The related FreeBSD documentation in the handbook doesn't go into the details at any time (https://www.freebsd.org/doc/handbook/carp.html) therefore it's better to get the info from the source.
On OpenBSD FAQ pages there's much more: https://www.openbsd.org/faq/pf/carp.html
and in the specific, under "CARP Operation":

"In order to prevent a malicious user on the network segment from spoofing CARP advertisements, each group can be configured with a password. Each CARP packet sent to the group is then protected by an SHA1 HMAC."

This should be most likely a modified version of the one specified in RFC2338 which states MD5-HMAC but that's another story as CARP is a re-writing of VRRP (to my understanding), moreover the RFC is very old and MD5 is no longer considered secure, so that's good.

About the password type/lenght limitation no mention anywhere:

carp(4) - OpenBSD manual pages

ifconfig(8) - OpenBSD manual pages

ifconfig(8)

www.freebsd.org

carp

www.freebsd.org

So, in conclusion:

FreeBSD 11.2 with the aforementioned setup does not accomplish any authentication during the VRRP packet exchange.
PFsense does but just the Simple Text Authentication instead of the SHA1-HMAC expected one.

Am I missing something trivial?
Any hint would be highly appreciated.

Thanks in advance.

QUOTING SOME RFC REFERENCE

The RFC3798 doesn't even mention "authentication". So let's talk only about RFC2338 which has 1 related chapter, the 10th.
10. Security Considerations
..........
The protocol includes several
authentication methods ranging from no authentication, simple clear
text passwords, and strong authentication using IP Authentication
with MD5 HMAC.
..........

10.1 No Authentication

The use of this authentication type means that VRRP protocol
exchanges are not authenticated. This type of authentication SHOULD
only be used in environments were there is minimal security risk and
little chance for configuration errors (e.g., two VRRP routers on a
LAN).

10.2 Simple Text Password

The use of this authentication type means that VRRP protocol
exchanges are authenticated by a simple clear text password.
..........
This type of authentication does not protect against
hostile attacks where the password can be learned by a node snooping
VRRP packets on the LAN.
..........

10.3 IP Authentication Header
The use of this authentication type means the VRRP protocol exchanges
are authenticated using the mechanisms defined by the IP
Authentication Header [AUTH] using "The Use of HMAC-MD5-96 within ESP
and AH", [HMAC].
..........

SirDice · Feb 25, 2019

What does RFC-3798 have to do with CARP?

Code:

   This memo defines a MIME content-type that may be used by a mail user
   agent (MUA) or electronic mail gateway to report the disposition of a
   message after it has been successfully delivered to a recipient.

RFC-2338 is about VRRP. VRRP and CARP serve the same purpose but are not the same protocol.

asv0 · Feb 25, 2019

SirDice said:
What does RFC-3798 have to do with CARP?

Nothing but was mentioned in the protocol description (that comes from wireshark), so I've mentioned it and discarded it.

SirDice said:
RFC-2338 is about VRRP. VRRP and CARP serve the same purpose but are not the same protocol.

I know. What concerns me is that CARP should use SHA1-HMAC to encrypt the authentication when a password is present but it doesn't seem to work.

SirDice · Feb 25, 2019

OpenBSD's CARP is not the same as FreeBSD's CARP.

asv0 · Feb 25, 2019

SirDice said:
OpenBSD's CARP is not the same as FreeBSD's CARP.

I don't think I've wrote that anywhere but as it was ported from the OpenBSD one I would assume that has the same encryption algo.

This should be most likely a modified version of the one specified in RFC2338 which states MD5-HMAC but that's another story as CARP is a re-writing of VRRP (to my understanding)

I fully understand that it's not the same. But I would expect to do what it claims and I'm trying to figure out if it's not due to my mistake or due to a bug.

carp

www.freebsd.org

To use carp, the administrator needs to configure at a minimum a common
virtual host ID (vhid), and attach at least one IP address to this vhid
on each machine which is to take part in the virtual group. Additional
parameters can also be set on a per-vhid basis: advbase and advskew,
which are used to control how frequently the host sends advertisements
when it is the master for a virtual host, and pass which is used to
authenticate carp advertisements.

ifconfig(8)

www.freebsd.org

The following parameters are used to configure carp(4) protocol on an
interface:

vhid ....

advbase ....

advskew ....

pass phrase
Set the authentication key to phrase.

So, again, where is this authentication advertisement actually happening?

SirDice · Feb 25, 2019

asv0 said:
I don't think I've wrote that anywhere but as it was ported from the OpenBSD one I would assume that has the same encryption algo.

You're taking information from the OpenBSD man pages and are assuming it's the same for FreeBSD. Don't make that assumption.

asv0 · Feb 25, 2019

OK, let's cut this off.

I was wrong making the assumption that because OpenBSD uses SHA1-HMAC then the ported FreeBSD version would also do that.

Now, do you have something more substantial to add to this thread?

SirDice · Feb 25, 2019

Yes, your CARP is working, one interface is MASTER, the other BACKUP. Which means it works. If authentication was a problem both nodes wouldn't be able to figure out who's what.

asv0 · Feb 25, 2019

Dear SirDice, I understand my post is long but that doesn't mean shouldn't be read prior to post a reply.

asv0 said:
As I did't enable the preemption I'm testing the failover manually and it works well just as connectivity to the service IP .130 does.
Everything works very well BUT authentication!

phoenix · Feb 27, 2019

What, exactly, is the issue you think you're seeing?

Authentication is obviously working, otherwise you'd have two MASTER devices and no BACKUP devices.

asv0 · Feb 28, 2019

Hi phoenix, thanks for you reply.
According to my understanding, authentication is not mandatory and it happens only if the "pass" parameter followed by an actual "password" is present.
And what I see is that PFsense cluster is authenticating via "simple text" and actually using a password which is not the one set in the configuration

Code:

23:14:19.556562 IPv4 (0x0800), length 60: XX.XX.XX.XX > 224.0.0.18: VRRPv2, Advertisement, vrid 101, prio 105, authtype simple, intvl 3s, length 20
        0x0000:  45 c0 00 28 00 00 00 00 ff [70] 56 20 [-- -- -- --]  E..(.....pV.....
        0x0010:  [e0 00 00 12] [21] 65 69 01 [01] 03 ce ff -- -- -- --  ....!ei.........
        0x0020:  [6e 4f 63 6e 4f 63 00 00] 00 00 00 00 00 00       nOcnOc........

while the FreeBSD cluster is not authenticating at all.

Code:

22:57:50.622956 IP 10.2.2.128 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36
        0x0000:  45 10 00 38 00 00 40 00 ff [70] 8e b1 [0a 02 02 80]  E..8..@..p......
        0x0010:  [e0 00 00 12] [21] 01 64 07 [00] 01 8c 06 -- -- -- --  ....!.d.........
        0x0020:  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  ................
        0x0030:  -- -- -- -- -- -- -- --                      ........

Which doesn't mean that the nodes cannot communicate with each other successfully exchanging CARP packets. There is just no authentication happening where I would expect one (of any kind) to happen.

CARP auth not working on 11.2-RELEASE-p9

asv0

ifconfig(8)

carp

SirDice

Administrator

asv0

SirDice

Administrator

asv0

carp

ifconfig(8)

SirDice

Administrator

asv0

SirDice

Administrator

asv0

phoenix

asv0