Solved vnet MAC address when using jails - how to avoid leaking?

codetrotter · May 8, 2023

Hi, I am using jails with vnet on FreeBSD 13.1-RELEASE

Host /etc/rc.conf looks like:

Code:

hostname="de1"

ifconfig_em0_name="extif"
ifconfig_extif="DHCP"
ifconfig_extif_ipv6="inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64"
ipv6_defaultrouter="fe80::1%extif"

ipv6_gateway_enable="YES"

cloned_interfaces="epair0 epair1 epair2 bridge0 bridge1"

ifconfig_epair0a_name="e0a_ext_jpub"
ifconfig_epair0b_name="e0b_ext_jpub"

ifconfig_epair1a_name="e1a_jail1"
ifconfig_epair1b_name="e1b_jail1"

ifconfig_epair2a_name="e2a_jail2"
ifconfig_epair2b_name="e2b_jail2"

ifconfig_bridge0_name="jbrext"
ifconfig_jbrext="addm extif addm e0a_ext_jpub"

ifconfig_bridge1_name="jbrpub"
ifconfig_jbrpub="addm e0b_ext_jpub addm e1a_jail1 addm e2a_jail2"

#ifconfig_jbrext_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64"

ifconfig_e0a_ext_jpub_ipv6="inet6 2a01:xxxx:xxxx:xxxx::4 prefixlen 64"
ifconfig_e0b_ext_jpub_ipv6="inet6 2a01:xxxx:xxxx:xxxx::5 prefixlen 64"

#ifconfig_jbrpub_ipv6="inet6 2a01:xxxx:xxxx:xxxx::6 prefixlen 64"

ifconfig_e1a_jail1_ipv6="inet6 2a01:xxxx:xxxx:xxxx::7 prefixlen 64"

ifconfig_e2a_jail2_ipv6="inet6 2a01:xxxx:xxxx:xxxx::b prefixlen 64"

sshd_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

zfs_enable="YES"

jail_enable="YES"

And then I have an /etc/jail.conf file that looks like:

Code:

$j = "/jail";

path = "$j/$name";

exec.clean;
exec.start = "sh /etc/rc";
exec.stop = "sh /etc/rc.shutdown";
mount.devfs;
exec.prestart = "logger starting jail $name";
exec.poststart = "logger jail $name has started";
exec.prestop = "logger shutting down jail $name";
exec.poststop = "logger jail $name has shut down";
exec.consolelog = "/var/tmp/jail-consolelog-$name";

jail1 {
  host.hostname = "jail1";

  vnet;
  vnet.interface = "e1b_jail1";

  sysvsem = "new";
  sysvmsg = "new";
  sysvshm = "new";
}

ipv4-proxy {
  host.hostname = "ipv4-proxy.example.com";

  ip4 = inherit;
  ip6 = inherit;

  allow.raw_sockets = true;
}

jail2 {
  host.hostname = "jail2";

  vnet;
  vnet.interface = "e2b_jail2";
}

And the /etc/rc.conf files of the two jails look like follows.

The /etc/rc.conf file of jail1 looks like

Code:

ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx::4"

ifconfig_e1b_jail1_ipv6="inet6 2a01:xxxx:xxxx:xxxx::8 prefixlen 64"

postgresql_enable="YES"
redis_enable="YES"
nginx_enable="YES"

The /etc/rc.conf file of jail2 looks like

Code:

ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx::4"

ifconfig_e2b_jail2_ipv6="inet6 2a01:xxxx:xxxx:xxxx::c prefixlen 64"

nginx_enable="YES"

This gives me globally routable IPv6 for each of the jails. That is awesome and desirable.

Unfortunately, the vnet interfaces are using different MAC addresses and these virtual MAC addresses end up hitting the external network interface.

Because of this, my server hosting provider sent me an email complaining that I am source-MAC not permitted by them.

They have a resource page about the problem at https://docs.hetzner.com/robot/dedicated-server/faq/error-faq/#mac-errors

How can I achieve the same as above, giving my jails globally routable IPv6 addresses from the /64 subnet assigned to me, without sending packets with bad source-MAC?

SirDice · May 8, 2023

codetrotter said:
Unfortunately, the vnet interfaces are using different MAC addresses and these virtual MAC addresses end up hitting the external network interface.

Do they have to be VNET jails? Without VNET the jails would be bound to the external interface. The IP addresses of the jails would simply be aliases on the interface, thus only showing that MAC address.

codetrotter · May 8, 2023

SirDice said:
Do they have to be VNET jails? Without VNET the jails would be bound to the external interface. The IP addresses of the jails would simply be aliases on the interface, thus only showing that MAC address.

Yes, I want them to be vnet jails

SirDice · May 8, 2023

codetrotter said:
I want them to be vnet jails

want or need? Those are two different things. The first is simple to overcome, the second isn't. The simplest solution is to use non-VNET jails.

codetrotter · May 8, 2023

SirDice said:
want or need? Those are two different things. The first is simple to overcome, the second isn't. The simplest solution is to use non-VNET jails.

Want and need.

So anyway, does anyone know how I can solve the problem?

SirDice · May 8, 2023

They give two possible 'solutions'.

1) bridged. This would require using standard jails, similar set up as you have but without the VNET.
2) routed. Not exactly possible as you can't segment a /64 any further.

codetrotter · May 8, 2023

That’s unfortunate but thank you

codetrotter · May 8, 2023

Is there any other way, that it could be achieved, with vnet still, that anyone can think of?

SirDice · May 8, 2023

One other possible solution I just though of, but it's horrendous. Bind the jails to lo1, use site-local addresses on the jails and NAT on the external interface.

codetrotter · May 8, 2023

Yeah, not ideal

I’ll try to ask on the mailing lists later. Maybe someone there has some other additional ideas

codetrotter · May 8, 2023

What would happen if I set the "mac" property for each vnet interface in the jail configuration file to the same MAC address as what the external network interface on my FreeBSD host has?
ifconfig extif | grep ether

This will output the MAC address of the external interface (in the format "ether XX:XX:XX:XX:XX:XX").

For example, for jail1, I would add the following line to the jail configuration:

vnet.interface = "e1b_jail1,mac=XX:XX:XX:XX:XX:XX";

Replacing "XX:XX:XX:XX:XX:XX" with the MAC address of my external network interface.

After making these changes, restart the jails and verify that the vnet interfaces now have the same MAC address as my external network interface.

I’d try it myself to see what happens, but I am asking first because with Hetzner there is no automatic KVM available, meaning that if I screw things up I may have to schedule a specific time at which I get to use KVM. And that would really screw things up for me.

Is it possible to do like this using the same MAC address, and would it work?

zirias@ · May 8, 2023

I can't help here, but ... someone enlighten me please, is their really any valid concern (like e.g. security) about extra MACs from your host?

Reading these docs, my impression is you can get allowed "virtual" MACs for up to 6 addresses (IOW, interfaces?). So, 6. For a /64 network. What the hell?

zirias@ · May 8, 2023

SirDice said:
2) routed. Not exactly possible as you can't segment a /64 any further.

Wait. Sure you can do that. It's "invalid" only for a single reason: local parts must be 64bit for SLAAC which is mandatory for IPv6 implementations. But nevertheless, it works (BTDT) as long as you don't use SLAAC. So, might indeed be an option to solve that ...

Here, something simple like having a separate bridge with all the jails in network <prefix>:0::/80 and the host itself in <prefix>:1::/80 might just work fine, setting up the required routes of course...

codetrotter · May 8, 2023

zirias@ said:
I can't help here, but ... someone enlighten me please, is their really any valid concern (like e.g. security) about extra MACs from your host?

Reading these docs, my impression is you can get allowed "virtual" MACs for up to 6 addresses (IOW, interfaces?). So, 6. For a /64 network. What the hell?

The 6 MAC addresses are for the case where some orders up to 6 individual additional IPv4 addresses I think.

But yeah I agree, I don’t understand why Hetzner has this rule. Surely their network is set up so that one customer is not able to impersonate another customers servers by copying their MAC addresses or anything like that.

codetrotter · May 8, 2023

zirias@ said:
Wait. Sure you can do that. It's "invalid" only for a single reason: local parts must be 64bit for SLAAC which is mandatory for IPv6 implementations. But nevertheless, it works (BTDT) as long as you don't use SLAAC. So, might indeed be an option to solve that ...

Awesome! I am indeed not using SLAAC for the jails anyways. Any more details on how I can subnet my /64 and use it for the jails?

SirDice · May 8, 2023

zirias@ said:
Wait. Sure you can do that. It's "invalid" only for a single reason: local parts must be 64bit for SLAAC which is mandatory for IPv6 implementations.

Not just for SLAAC. Global unicast addresses have a 64 bit interface ID.

zirias@ · May 9, 2023

SirDice said:
Not just for SLAAC. Global unicast addresses have a 64 bit interface ID.

Uhm ... which is a definition. And it's technically required to ensure SLAAC can always be used ...

Of course, routers could refuse to route on anything longer than /64, because it's not required. Still, it's not an issue with FreeBSD (test it

)

zirias@ · May 9, 2023

codetrotter said:
Any more details on how I can subnet my /64 and use it for the jails?

I used /80 subnets at home for a while, but my setup was a bit different, so this is just what I think should work:

Create a bridge, assign it e.g. xxxx:xxxx:xxxx:xxxx:1::1/80, add all your jail epair ends to it, but not your real NIC.
Configure the epair ends inside the jails to xxxx:xxxx:xxxx:xxxx:1::n/80 (with n > 1 of course) and set xxxx:xxxx:xxxx:xxxx:1::1 as the default gateway.
Configure your real NIC for xxxx:xxxx:xxxx:xxxx::2/80 and set a route to xxxx:xxxx:xxxx:xxxx:1::/80 via xxxx:xxxx:xxxx:xxxx:1::1.

The only route known outside your FreeBSD box will still be the /64, so I don't see any issues there. Not entirely sure this setup will work though, but at least something similar should...

codetrotter · May 9, 2023

zirias@ said:
I used /80 subnets at home for a while, but my setup was a bit different, so this is just what I think should work:

Create a bridge, assign it e.g. xxxx:xxxx:xxxx:xxxx:1::1/80, add all your jail epair ends to it, but not your real NIC.
Configure the epair ends inside the jails to xxxx:xxxx:xxxx:xxxx:1::n/80 (with n > 1 of course) and set xxxx:xxxx:xxxx:xxxx:1::1 as the default gateway.
Configure your real NIC for xxxx:xxxx:xxxx:xxxx::2/80 and set a route to xxxx:xxxx:xxxx:xxxx:1::/80 via xxxx:xxxx:xxxx:xxxx:1::1.

The only route known outside your FreeBSD box will still be the /64, so I don't see any issues there. Not entirely sure this setup will work though, but at least something similar should...

Thank you zirias, I will try this

zirias@ · May 9, 2023

codetrotter said:
Surely their network is set up so that one customer is not able to impersonate another customers servers by copying their MAC addresses or anything like that.

Eh ... avoiding ARP poisoning attacks might indeed be the background of their rules, still I think there has to be a better way than that

Anyways, good luck with this slightly weird routing workaround

covacat · May 9, 2023

im not sure gifs work in jails but if they do can't you create a gif tunnel over the epair and put the real ip on gif inner and something else as gif outer (even ipv4)?

codetrotter · May 11, 2023

Looks like your suggestion works very well zirias. Thanks again. Marking thread as solved.

For anyone coming here from Google wanting to subnet IPv6 in a similar situation with Hetzner or other hosting providers, here is what my /etc/rc.conf ended up looking like for the host machine:

Code:

hostname="de1"

ifconfig_em0_name="extif"
ifconfig_extif="DHCP"
ifconfig_extif_ipv6="inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 80"
ipv6_defaultrouter="fe80::1%extif"

ipv6_gateway_enable="YES"

cloned_interfaces="epair1 epair2 bridge1"

ifconfig_epair1a_name="e1a_jail1"
ifconfig_epair1b_name="e1b_jail1"

ifconfig_epair2a_name="e2a_jail2"
ifconfig_epair2b_name="e2b_jail2"

ifconfig_bridge1_name="jbrpub"
ifconfig_jbrpub="addm e1a_jail1 addm e2a_jail2"

ifconfig_jbrpub_ipv6="inet6 2a01:xxxx:xxxx:xxxx:1::6 prefixlen 80"

ifconfig_e1a_jail1_ipv6="inet6 2a01:xxxx:xxxx:xxxx:1::7 prefixlen 80"

ifconfig_e2a_jail2_ipv6="inet6 2a01:xxxx:xxxx:xxxx:1::b prefixlen 80"

sshd_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

zfs_enable="YES"

jail_enable="YES"

And the /etc/jail.conf on my host machine

Code:

$j = "/jail";

path = "$j/$name";

exec.clean;
exec.start = "sh /etc/rc";
exec.stop = "sh /etc/rc.shutdown";
mount.devfs;
exec.prestart = "logger starting jail $name";
exec.poststart = "logger jail $name has started";
exec.prestop = "logger shutting down jail $name";
exec.poststop = "logger jail $name has shut down";
exec.consolelog = "/var/tmp/jail-consolelog-$name";

jail1 {
  host.hostname = "jail1";

  vnet;
  vnet.interface = "e1b_jail1";

  sysvsem = "new";
  sysvmsg = "new";
  sysvshm = "new";
}

ipv4-proxy {
  host.hostname = "ipv4-proxy.example.com";

  ip4 = inherit;
  ip6 = inherit;

  allow.raw_sockets = true;
}

jail2 {
  host.hostname = "jail2";

  vnet;
  vnet.interface = "e2b_jail2";
}

And the /etc/rc.conf of jail1

Code:

ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx:1::6"

ifconfig_e1b_jail1_ipv6="inet6 2a01:xxxx:xxxx:xxxx:1::8 prefixlen 80"

postgresql_enable="YES"
redis_enable="YES"
nginx_enable="YES"

And the /etc/rc.conf of jail2

Code:

ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx:1::6"

ifconfig_e2b_jail2_ipv6="inet6 2a01:xxxx:xxxx:xxxx:1::c prefixlen 80"

nginx_enable="YES"

zirias@ · May 11, 2023

Nice you got it working. Well, indeed, routing on prefixes longer than /64 is out of IPv6 specs, but FreeBSD (and probably a few other systems?) still has no issues doing it anyways

codetrotter · Jun 4, 2023

I just updated my server and the jails to FreeBSD 13.2 and now I am not able to run the jails this way.

Everything was working fine for weeks, across many reboots, while using FreeBSD 13.1.

But somehow when I updated to FreeBSD 13.2 I am not able to run the jails this way.

Not sure why. Been racking my brain and fiddling around for hours, and the only way I am able to run my jails now is by out-commenting the

Code:

ifconfig_e1b_jail1_ipv6="inet6 2a01:xxxx:xxxx:xxxx:1::8 prefixlen 80"

and the

Code:

ifconfig_e2b_jail2_ipv6="inet6 2a01:xxxx:xxxx:xxxx:1::c prefixlen 80"

from the /etc/rc.conf files of the two jails respectively

codetrotter · Jun 5, 2023

Any idea what I could do zirias@

Solved vnet MAC address when using jails - how to avoid leaking?

codetrotter

SirDice

Administrator

codetrotter

SirDice

Administrator

codetrotter

SirDice

Administrator

codetrotter

codetrotter

SirDice

Administrator

codetrotter

codetrotter

zirias@

zirias@

codetrotter

codetrotter

SirDice

Administrator

zirias@

zirias@

codetrotter

zirias@

covacat

codetrotter

zirias@

codetrotter

codetrotter