FreeBSD 15 Bridges, VLANs and Jails - Nice!

[codeedog]

With version 15's new VLAN aware bridges, we only need one bridge at the host level and one epair per jail to handle networking for every jail's needs.

Prior to version 15, when I was trying to build a vnet jailed gateway with a handful of VLANs I had to create an octopus of bridges (one per VLAN) and lots of dangling epairs. Version 15's new tech changed all that. Not only can all of the external VLANs collapse into one bridge, but now we can fold all private bridges into the one host bridge (add internal-only VLAN IDs to isolate communications).

With the VLAN aware bridges you only need one bridge on the host and one (possibly trunked) epair dropped into the jail. Make an epair VLAN aware epair and configure each epair/bridge connection to the subset of VLANs used by the jail. For any private jail-to-jail internal networks, use a new VLAN ID to connect a subgroup of jails via their epairs through the bridge.

The key is to get the ifconfig settings correct. Here's an example for the set up (without the jail configuration files) for a host, a physical interface (igb0 with VLANs 10,20) and two subjails with three VLANs (10, 20, 2001):

ifconfig epair0  create up
ifconfig epair0b up
ifconfig epair1  create up
ifconfig epair1b up
ifconfig bridge0 create up
ifconfig bridge0 vlanfilter addm igb0 tagged 10,20 up
ifconfig bridge0 addm epair0a tagged 10,2001
ifconfig bridge0 addm epair1a tagged 20,2001

# Pass epair0b into jail0 (vnet.interface = epair0b)
# Pass epair1b into jail1 (vnet.interface = epair1b)

With the above configuration, external packets arriving on VLAN 10 go to jail0 and those arriving on VLAN 20 go to jail1. Meanwhile, for jail0 and jail1 to communicate, they can use VLAN 2001.

Inside the jail, the epair VLANs can be splintered off:

ifconfig -j jail0 epair0b.10   create inet 192.168.10.55/24 up
ifconfig -j jail0 epair0b.2001 create inet 10.20.01.0/31 up
ifconfig -j jail1 epair1b.20   create inet 192.168.20.55/24 up
ifconfig -j jail0 epair1b.2001 create inet 10.20.01.1/31 up

jail0 communicates with the external network via epair0b.10.
jail1 communicates with the external network via epair1b.20.
Both jails communicate with each other using epair0b.2001 and epair1b.2001.

I built a complex test platform and watched ICMP packets zip around. I can share the code.
Hat tip to reddit user u/-iwantmy2dollars- for their post, which prompted my testing and this post.

 

[toukkas]

I think this looks very nifty and and clean. I've looked at both the reddit thread and this one, but apparenty I fail to understand the concept?

If I want to set an IP on the host, in the best of my understanding it should not be on the vlan nor the bridge interface?
I try to set up a simpler case with a bridge containing tagged vlans, and the plan would be to create vnet epairs for jails and bhyve guests, but I do not grasp where the host IP could be set (epair also for the host?) for this setup to be functional.

 

[toukkas]

Allright, my issue were twofold. At first I had an attempt with a lagg that I failed to setup, but (as I have this host running another system...) I switched to another port on the switch instead of reconfiguring the aggregation, but forgot to add the jail vlan to that port.
Then, the other failure I had was that when a jail started (during the creation of epair), the host lost its connectivity. At a glance, it looks like I've managed to solve that issue as well, with a vlan interface on the host.

cloned_interfaces="bridge0 vlan999"
ifconfig_ixl3="-vlanhwfilter up"
ifconfig_ixl3_description="Trunk"
ifconfig_bridge0="vlanfilter addm ixl3 tagged 24,442,900-999"
create_args_vlan999="vlan 999 vlandev bridge0"
ifconfig_vlan999="inet 172.28.12.4/28"

 

[toukkas]

It was a temporary success. During jail start up (creation) of epair, the host connectivity stops - about 30 packets lost during ping (tcpdump shows absence of icmp replies during 34 seconds). The same happens during destroy of the epair.

 

[toukkas]

It's only the first (and/or only) epair that breaks the connection. Subsequent jails does not seem to affect the host connectivity. So a placeholder jail might be a workaround.

 

[toukkas]

Okay, everything seem to work as expected.. The machine was previously setup with several virtual guests using dedicated interfaces (pass through devices), but I haven't gotten that far yet to exclude the interfaces from the host and it seem that those interfaces now caused a spanning tree loop (and the host got blocked from the switches while negotiating RTSP).

 

[codeedog]

You have a complex network you're integrating into with quite a bit going on. I don't know if I have the chops to advise you. My recommendation would be to isolate these things, work on them separately and then recombine them once you're convinced they work as expected.

For your very first question if you haven't already solved it, in your case for adding host level IP addresses on the VLANs coming in from the NIC, they'd hang off of ixl3. They'd look like ixl3.24, ixl3.442, etc. For internal IP addresses, I haven't tested those in v15, yet. You used to be able to assign them to a bridge, but I believe that's no longer possible. My hack (until I think of something better) would be to create and attach an epair to the bridge and leave it in the host (say, epair0a gets attached to the bridge). Then, you can assign an IP address to the other end of the epair. If it's a trunked epair, then you'd splinter it off and assign the IP address to the splinter (epair0b.900 gets 10.90.0.1, epair0b,901 gets 10.90.1.1, etc).

Hope that helps.

 

[bakul]

Adding https://blog.feld.me/posts/2026/02/using-new-bridges-freebsd-15/ for reference.

 

[toukkas]

You have a complex network you're integrating into with quite a bit going on. I don't know if I have the chops to advise you. My recommendation would be to isolate these things, work on them separately and then recombine them once you're convinced they work as expected.

For your very first question if you haven't already solved it, in your case for adding host level IP addresses on the VLANs coming in from the NIC, they'd hang off of ixl3. They'd look like ixl3.24, ixl3.442, etc. For internal IP addresses, I haven't tested those in v15, yet. You used to be able to assign them to a bridge, but I believe that's no longer possible. My hack (until I think of something better) would be to create and attach an epair to the bridge and leave it in the host (say, epair0a gets attached to the bridge). Then, you can assign an IP address to the other end of the epair. If it's a trunked epair, then you'd splinter it off and assign the IP address to the splinter (epair0b.900 gets 10.90.0.1, epair0b,901 gets 10.90.1.1, etc).

Hope that helps.

Thanks, I'm still a bit baffled on how I ended up in my situation, but when I first had issues setting up the lagg, I believe that the interfaces that I so far did not set up as pass through devides were causing me my first issues (which I then thought were due to the lagg)... and I thought about that epair and while it worked to set an IP to that, I still had the loops that were bugging me (I did not yet know about them at that time). I don't know what kind of device this "vlan" device is and if it has any performance penalties, but it seem to behave at least.

Adding the pass through devices in loader.conf solved my headache:

tail -2 /boot/loader.conf
vmm_load="YES"
pptdevs="102/0/0 102/0/1 102/0/2 102/0/3 183/0/0 183/0/1"

Then in the /etc/rc.conf (my ixl2 and ixl3 became ixl0 and ixl1 respectivelly, due to pptdevs):

cloned_interfaces="bridge0 lagg0 vlan999"
ifconfig_ixl0="-vlanhwfilter up"
ifconfig_ixl1="-vlanhwfilter up"
ifconfig_lagg0="laggproto lacp laggport ixl0 laggport ixl1 up"
ifconfig_lagg0_description="VM Trunk"
ifconfig_bridge0="vlanfilter addm lagg0 tagged 24,442,900-999"
create_args_vlan999="vlan 999 vlandev bridge0"
ifconfig_vlan999="inet 172.28.12.4/28"

then in my /etc/jails.conf.d/jail.conf I have this:

jail {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 5;

  # PATH/HOSTNAME
  path = "/jails/containers/${name}";
  host.hostname = "${name}";

  # VNET/VIMAGE
  vnet;
  vnet.interface = "${epair}b";

  # NETWORKS/INTERFACES
  $id = "32";
  $ip = "10.24.0.${id}/24";
  $gateway = "10.24.0.1";
  $bridge = "bridge0";
  $epair = "epair${id}";

  # ADD TO bridge INTERFACE
  exec.prestart += "ifconfig ${epair} create -vlanhwfilter up";
  exec.prestart += "ifconfig ${bridge} vlanfilter addm ${epair}a untagged 24 up";
  exec.start    += "ifconfig ${epair}b -vlanhwfilter ${ip} up";
  exec.poststop = "ifconfig bridge0 deletem ${epair}a";
  exec.poststop += "ifconfig ${epair}a destroy";
}

My next stop will be bhyve and jailed bhyve, but so far I like the new implementation of the bridge as it feels a lot cleaner. I have yet to figure out if there is some kind of spoof protection available to block a jail from setting up a not allowed IP, but this is my home network so it's not that critical (but could be a nice bonus).

 

[codeedog]

Adding https://blog.feld.me/posts/2026/02/using-new-bridges-freebsd-15/ for reference.

I saw this on the r/freebsd. Nice write up!

BTW, I've recently found some trouble with epairs not applying checksums, but advertising they do so packets are dropped when they're received on the physical network by other machines and those packets are not NAT'd. That's a mouthful, so to be clear for packets originating from an epair:

Source​	Route​	Destination​	Result​	Notes​
epair	> bridge >	host or another jail	OK	no checksum, all virtual
epair	> bridge > host(NAT) > NIC >	LAN device	OK	no checksum, host computes checksum at NAT
epair	> bridge > NIC > edge router(NAT) >	WAN device	OK	no checksum, edge router computes checksum at NAT
epair	> bridge > NIC >	LAN device	Dropped	no checksum, arrives empty, packet dropped

So, the problem is as far as I can tell epairs lie about their ability to checksum. To solve this, use ifconfig <if> -txcsum -rxcsum

This killed my network for 14 hours because DNS was broken. It's always DNS...