FreeBSD 15 Bridges, VLANs and Jails - Nice!

[codeedog]

With version 15's new VLAN aware bridges, we only need one bridge at the host level and one epair per jail to handle networking for every jail's needs.

Prior to version 15, when I was trying to build a vnet jailed gateway with a handful of VLANs I had to create an octopus of bridges (one per VLAN) and lots of dangling epairs. Version 15's new tech changed all that. Not only can all of the external VLANs collapse into one bridge, but now we can fold all private bridges into the one host bridge (add internal-only VLAN IDs to isolate communications).

With the VLAN aware bridges you only need one bridge on the host and one (possibly trunked) epair dropped into the jail. Make an epair VLAN aware epair and configure each epair/bridge connection to the subset of VLANs used by the jail. For any private jail-to-jail internal networks, use a new VLAN ID to connect a subgroup of jails via their epairs through the bridge.

The key is to get the ifconfig settings correct. Here's an example for the set up (without the jail configuration files) for a host, a physical interface (igb0 with VLANs 10,20) and two subjails with three VLANs (10, 20, 2001):

ifconfig bridge0 create up
ifconfig bridge0 vlanfilter addm igb0 tagged 10,20 up
ifconfig epair0  create -vlanhwfilter up
ifconfig epair0b -vlanhwfilter up
ifconfig bridge0 vlanfilter addm epair0a tagged 10,2001 up
ifconfig epair1  create -vlanhwfilter up
ifconfig epair1b -vlanhwfilter up
ifconfig bridge0 vlanfilter addm epair1a tagged 20,2001 up

# Pass epair0b into jail0 (vnet.interface = epair0b)
# Pass epair1b into jail1 (vnet.interface = epair1b)

With the above configuration, external packets arriving on VLAN 10 go to jail0 and those arriving on VLAN 20 go to jail1. Meanwhile, for jail0 and jail1 to communicate, they can use VLAN 2001.

Inside the jail, the epair VLANs can be splintered off:

ifconfig -j jail0 epair0b.10   create inet 192.168.10.55/24 up
ifconfig -j jail0 epair0b.2001 create inet 10.20.01.0/31 up
ifconfig -j jail1 epair1b.20   create inet 192.168.20.55/24 up
ifconfig -j jail0 epair1b.2001 create inet 10.20.01.1/31 up

jail0 communicates with the external network via epair0b.10.
jail1 communicates with the external network via epair1b.20.
Both jails communicate with each other using epair0b.2001 and epair1b.2001.

I built a complex test platform and watched ICMP packets zip around. I can share the code.
Hat tip to reddit user u/-iwantmy2dollars- for their post, which prompted my testing and this post.

 

[toukkas]

I think this looks very nifty and and clean. I've looked at both the reddit thread and this one, but apparenty I fail to understand the concept?

If I want to set an IP on the host, in the best of my understanding it should not be on the vlan nor the bridge interface?
I try to set up a simpler case with a bridge containing tagged vlans, and the plan would be to create vnet epairs for jails and bhyve guests, but I do not grasp where the host IP could be set (epair also for the host?) for this setup to be functional.

 

[toukkas]

Allright, my issue were twofold. At first I had an attempt with a lagg that I failed to setup, but (as I have this host running another system...) I switched to another port on the switch instead of reconfiguring the aggregation, but forgot to add the jail vlan to that port.
Then, the other failure I had was that when a jail started (during the creation of epair), the host lost its connectivity. At a glance, it looks like I've managed to solve that issue as well, with a vlan interface on the host.

cloned_interfaces="bridge0 vlan999"
ifconfig_ixl3="-vlanhwfilter up"
ifconfig_ixl3_description="Trunk"
ifconfig_bridge0="vlanfilter addm ixl3 tagged 24,442,900-999"
create_args_vlan999="vlan 999 vlandev bridge0"
ifconfig_vlan999="inet 172.28.12.4/28"

 

[toukkas]

It was a temporary success. During jail start up (creation) of epair, the host connectivity stops - about 30 packets lost during ping (tcpdump shows absence of icmp replies during 34 seconds). The same happens during destroy of the epair.

 

[toukkas]

It's only the first (and/or only) epair that breaks the connection. Subsequent jails does not seem to affect the host connectivity. So a placeholder jail might be a workaround.

 

[toukkas]

Okay, everything seem to work as expected.. The machine was previously setup with several virtual guests using dedicated interfaces (pass through devices), but I haven't gotten that far yet to exclude the interfaces from the host and it seem that those interfaces now caused a spanning tree loop (and the host got blocked from the switches while negotiating RTSP).