FreeBSD 15 / Bridge

iRobbery · 2026-04-16T07:55:23+0100

Hi,

I have a working setup on quite a few machines.

The machine has two bridges

bridge0 -> public internet as member (ice0, has the ip), and vnet interfaces from jails using internet routed ip addresses
bridge9 -> isolated bridge, on which jails have an internal netwerk link (10.10.2.x/24 for example)

On the host i NAT 10.10.2.0/24 to ice0; this works fine on FreeBSD 14.x; it is a quite straight forward setup if you ask me.

However if i copy this setup, rc.conf/ipfw.rules/sysctl.conf on the same machine but then with a fresh 15.0-RELEASE install, i cant get the NAT to work at all.

I'm told on IRC #freebsd / libera that 'it should work still the same on 15.0', but it doesnt. I've read the following.

Using The New Bridges of FreeBSD 15

FreeBSD 15 comes with a new bridging implementation which has native support for VLANs. They have also soft-deprecated the ability to have any layer 3 addresses on member interfaces which makes it behave like a real hardware switch. The net.link.bridge.member_ifaddrs sysctl controls this...

blog.feld.me

https://people.freebsd.org/~ivy/bridge_vlan_filtering.txt

I dont use any vlan, just simple bridging and ipfw NAT.

VladiBG · 2026-04-16T07:58:53+0100

Show us your config

sko · 2026-04-16T08:06:37+0100

iRobbery said:
bridge0 -> public internet as member (ice0, has the ip), and vnet interfaces from jails using internet routed ip addresses

Usually the physical interface does not hold any address, otherwise it will reject other traffic before it reaches the bridge. I.e. you only configure IPs on the downstream interfaces (vnet/epair), not on the uplink. (unless you configure *all* ips on that interface and then RDR to the jails - that's the classic, pre-vnet setup for jail networking)

Please show the config for that and corresponding parts of ifconfig output (/w redacted public IPs)

iRobbery · 2026-04-16T08:10:46+0100

rc.conf

hostname="myhostname.tld"
ifconfig_ice0="DHCP"
cloned_interfaces="bridge0 bridge9"

ifconfig_bridge0="addm ice0"
ifconfig_bridge9="inet 10.10.2.1 netmask 255.255.255.0 up"

sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"

firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_logging="YES"
firewall_logif="YES"
#
ipfw_nat_load="YES"
gateway_enable="YES"
iocage_enable="YES"

ipfw.rules

#!/bin/sh
# Flush out the list before we begin.
ipfw -q -f flush

# Set rules command prefix
cmd="ipfw -q add"

wanif="ice0"
lanif="bridge9"

lannet="10.10.2.0/24"

ipfw table badhosts create type addr
ipfw table goodhosts create type addr

ipfw table goodhosts add 1.2.3.4

# No restrictions on Loopback Interface
$cmd 00001 allow all from any to any via lo0

# No restrictions on LAN Interface
$cmd 00008 allow all from any to any via $lanif

# Configure NAT-WAN interface
ipfw -q nat 1 config if $wanif log reset same_ports

# Check inbound traffic for redirections
$cmd 00006 nat 1 ip from any to me in via $wanif

$cmd 50 deny all from "table(badhosts)" to any

# Allow dynamic rules table connections
$cmd 00101 check-state

# NAT Lan traffic:
$cmd 00104 skipto 1000 tcp from $lannet to any out via $wanif keep-state
$cmd 00105 skipto 1000 tcp from $lannet to any out via $wanif setup keep-state
$cmd 00106 skipto 1000 udp from $lannet to any out via $wanif keep-state
$cmd 00107 skipto 1000 icmp from $lannet to any out via $wanif keep-state

# -- Host Traffic --
# Allow access to public DNS
# DNS TCP
$cmd 00201 allow tcp from any to any 53 out via $wanif setup keep-state
$cmd 00202 allow tcp from any to any 853 out via $wanif keep-state

# DNS UDP
$cmd 00203 allow udp from any to any 53 out via $wanif keep-state

# Allow outbound ANY connections
$cmd 00299 allow tcp from me to any out via $wanif setup keep-state

# Allow outbound HTTP and HTTPS connections
$cmd 00300 allow tcp from any 80 to any out
$cmd 00301 allow tcp from any 443 to any out

# Allow inbound HTTP and HTTPS connections
$cmd 00302 allow tcp from any to any 80 out via $wanif keep-state
$cmd 00303 allow tcp from any to any 443 out via $wanif keep-state

# Allow outbound email connections
$cmd 00400 allow tcp from me to any 25 out via $wanif setup keep-state
$cmd 00401 allow tcp from me to any 110 out via $wanif setup keep-state
$cmd 00402 allow tcp from any to any 993 out via $wanif keep-state

# Allow outbound ping
$cmd 00500 allow icmp from me to any out via $wanif keep-state

# Allow outbound NTP
$cmd 00501 allow udp from me to any 123 out via $wanif keep-state

# Allow inbound public pings
$cmd 00700 allow icmp from any to me in via $wanif

$cmd 800 allow all from "table(goodhosts)" to any
$cmd 810 allow all from any to "table(goodhosts)"

# Deny and log all other outbound connections
$cmd 00900 deny log all from any to any out

# NAT outbound traffic
$cmd 01000 nat 1 ip from any to any out via $wanif keep-state
$cmd 01003 allow ip from any to any

$cmd 12900 deny log all from any to any out

loader.conf

tcp_bbr_load="YES"
ipfw_nat_load="YES"

iocage jail create

iocage create -r 15.0-RELEASE -n test-jail
iocage set vnet=on test-jail
iocage set bpf=on test-jail
iocage set boot=on test-jail
iocage set defaultrouter=10.10.2.1 test-jail
iocage set dhcp=none test-jail
iocage set interfaces="vnet0:bridge9" test-jail
iocage set ip4_addr="vnet0|10.10.2.101/24" test-jail

iRobbery · 2026-04-16T08:12:31+0100

sko said:
Usually the physical interface does not hold any address, otherwise it will reject other traffic before it reaches the bridge. I.e. you only configure IPs on the downstream interfaces (vnet/epair), not on the uplink. (unless you configure *all* ips on that interface and then RDR to the jails - that's the classic, pre-vnet setup for jail networking)

Please show the config for that and corresponding parts of ifconfig output (/w redacted public IPs)

moving the DHCP to the bridge0 interface (as warned per /var/log/messages, that it will be deprecated, but its not yet. And doesnt make any difference afaics, nat still fails

("they have also soft-deprecated the ability to have any layer 3 addresses on member interfaces which makes it behave like a real hardware switch. The net.link.bridge.member_ifaddrs sysctl controls this behavior and it will be removed in FreeBSD 16.0-RELEASE, same as if set to zero.")

iRobbery · 2026-04-16T08:15:28+0100

sko said:
Usually the physical interface does not hold any address, otherwise it will reject other traffic before it reaches the bridge. I.e. you only configure IPs on the downstream interfaces (vnet/epair), not on the uplink. (unless you configure *all* ips on that interface and then RDR to the jails - that's the classic, pre-vnet setup for jail networking)

Please show the config for that and corresponding parts of ifconfig output (/w redacted public IPs)

the requested ifconfig output

root@myhostname.tld:~ # ifconfig
ice0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4e10038<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 04:7c:16:f1:43:72
inet 66.249.1.1 netmask 0xffffffff broadcast 255.255.255.255
media: Ethernet autoselect (25G-AUI <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ice1: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4e10438<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,LRO,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 04:7c:16:f1:43:73
media: Ethernet autoselect (25G-AUI <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=10<VLAN_HWTAGGING>
ether 58:9c:fc:10:3e:7f
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge flags=0<>
member: ice0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 1 priority 128 path cost 55 vlan protocol 802.1q
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
bridge9: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=10<VLAN_HWTAGGING>
ether 06:7c:16:7e:48:06
inet 10.10.2.1 netmask 0xffffff00 broadcast 10.10.2.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge flags=0<>
member: vnet1.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 8 priority 128 path cost 2000 vlan protocol 802.1q
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
ipfw0: flags=1008801<UP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 65536
options=0
groups: ipfw
vnet1.1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: associated with jail: ns4 as nic: epair1b
options=60000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6,TXCSUM_IPV6>
ether 06:7c:16:7e:48:06
hwaddr 58:9c:fc:10:f7:c2
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

sko · 2026-04-16T09:13:27+0100

Thanks, please use the 'code' tags in the future; quotes completely destroy any helpful formatting.

As said: the ice0 interface shouldn't have any address as it only serves as the upstream for all interfaces attached to the bridge. The address for the host belongs on the bridge.
With an address set on the physical upstream interface, it is very likely that it will just drop traffic intended for other IPs - i.e. to the jails attached to the bridge. Especially if hardware offloading is involved the behaviour is completely unpredictable.

beware: the rc.conf syntax for bridges has become extremely strict/pesky/broken - any deviation from the specific order for the options and the interface will be fully or partially unconfigured and no IP assigned to the bridge!

It might be also worth checking all net.link.bridge. sysctls - especially everything that involves filtering.

Apart from the bridge/member configuration another thing caught my attention:
Since the NAT address is assigned via DHCP, it might not be set yet when ipfw is loaded, so it doesn't know about that IP. I never used ipfw, but from a quick glance at the manpage ipfw nat show config should give you the running configuration including the NAT IP it uses (maybe someone can confirm that this is the ipfw-equivalent to pfctl -s nat)
You can try to use 'SYNCDHCP' in rc.conf and/or to quickly check if this is the culprit, just manually reload ipfw after the host is up and an address assigned to the bridge.

Other than that: why aren't you using a jail for egress/edge routing (and firewalling)? That's the more common way to do it, as it confines and isolates the entry point from the internet into the system/network into a jail and adds quite a bit of additional security. You can completely lock down that jail and disable pretty much all services that might add an attack surface - sshd, ntpd etc...
You can then use the internal interface(s) of that gateway jail as the default router. This makes the network setup on the host much simpler, and when it comes to routing, simple is good. It also makes it dead simple to upgrade or migrate the gateway or to add redundancy.[/code]

iRobbery · 2026-04-16T09:20:54+0100

ah i used the quotes button, sorry about that. Thought that was suppose to work properly.

i've tried both, ice0 doing the DHCP and the bridge0, doesnt make a difference. Also reloaded ipfw after booting manually didnt change anything. Though one thing i do notice, one rule counting creazy packets/bytes.

`
00106 180221417030 9191292268530 Thu Apr 16 08:17:39 2026 skipto 1000 udp from 10.10.2.0/24 to any out via ice0 keep-state :default
`

But tcpdump doesnt show any traffic or stp storm.

The reason i use the ice0/dhcp for nat (pkg updates etc for the internal jails), is just so i dont need another jail with public ip just for running updates. I use the interface for mgmt/updates for internal jails. The other jails typically have a leg in each bridge and their own public ip & private ip.

Emrion · 2026-04-16T09:21:38+0100

sko said:
It might be also worth checking all net.link.bridge. sysctls - especially everything that involves filtering.

This recalls me a problem I got. don't know if it is related or not.

PF Post in thread 'A simple pf rule?'

Apr 14, 2025

sysctl net.link.bridge.pfil_member=1 did the trick.

Without that, there is no pfil hook in the interface members of the bridge. I wonder why it's the default, but it is.

I'm surprised that no one answered to this question which was actually simple.

iRobbery · 2026-04-16T09:34:25+0100

Emrion said:
This recalls me a problem I got. don't know if it is related or not.

PF Post in thread 'A simple pf rule?'

Apr 14, 2025

sysctl net.link.bridge.pfil_member=1 did the trick.

Without that, there is no pfil hook in the interface members of the bridge. I wonder why it's the default, but it is.

I'm surprised that no one answered to this question which was actually simple.

Emrion

Unfortunately no effect.

VladiBG · 2026-04-16T09:47:11+0100

Why you need br0?

sko · 2026-04-16T09:52:49+0100

moving the ip to the bridge very likely interfered with that nat ruleset - although I don't see any ' nat config ip' line where that address is actually set. If ipfw deducts the ip from the $wanif in the ' $cmd 01000 nat 1 ip from any to any out via $wanif keep-state' line, it most likely will fail now as the address is on the bridge now.
I guess you have to use bridge0 now as the $wanif, but as I said: I never used ipfw, so I may be completely off here.

Also IIUC, the net.link.bridge.pfil_bridge and .ipfw sysctls needs to be set to 1, and maybe even 'pfil_local_phys' (although that shouldn't be necessary if the ip no longer is on the physical interface)

iRobbery said:
The reason i use the ice0/dhcp for nat (pkg updates etc for the internal jails), is just so i dont need another jail with public ip just for running updates. I use the interface for mgmt/updates for internal jails. The other jails typically have a leg in each bridge and their own public ip & private ip.

The host doesn't need a public addres anymore, that's the point. Access to local/physical hosts goes through a dedicated jumphost/'jumpjail' (or the gateway). Only jails running actual internet-facing services like e.g. a reverse proxy need to have public addresses and have an interface in that wan-facing vlan. But physical/jailhosts are strictly only connected to the management network via a dedicated interface.

iRobbery · 2026-04-16T09:58:53+0100

sko said:
moving the ip to the bridge very likely interfered with that nat ruleset - although I don't see any ' nat config ip' line where that address is actually set. If ipfw deducts the ip from the $wanif in the ' $cmd 01000 nat 1 ip from any to any out via $wanif keep-state' line, it most likely will fail now as the address is on the bridge now.
I guess you have to use bridge0 now as the $wanif, but as I said: I never used ipfw, so I may be completely off here.

Also IIUC, the net.link.bridge.pfil_bridge and .ipfw sysctls needs to be set to 1, and maybe even 'pfil_local_phys' (although that shouldn't be necessary if the ip no longer is on the physical interface)

The host doesn't need a public addres anymore, that's the point. Access to local/physical hosts goes through a dedicated jumphost/'jumpjail' (or the gateway). Only jails running actual internet-facing services like e.g. a reverse proxy need to have public addresses and have an interface in that wan-facing vlan. But physical/jailhosts are strictly only connected to the management network via a dedicated interface.

yes in that case i do update the script to use bridge0 as wanif.

I've been pondering about what you said regarding using a jail for that nat traffic. And i don't mind changing the setup either. Indeed host can use that 'fw' jail then too. And i'll probably go this way.

But what bothers me a bit is that, this worked fine for me since 13.x or even earlier, up to and including 14.4. And now with 15.0 it doesn't anymore, and cannot see anything that explains it. But betting the horse on it being a recent change to bridge/ipfw. And i like to know what

Also this ipfw rule with the many packets indicates something. (because i don't see this traffic with tcpdump) but at some point i now notice, it prevents me from stopping the jail, connectivity drops, and in worst case the whole host machine locks up and requires a powercycle. This is quite different behaviour compared to pre 15.0-release

sko · 2026-04-16T10:05:37+0100

iRobbery said:
Also this ipfw rule with the many packets indicates something. (because i don't see this traffic with tcpdump) but at some point i now notice, it prevents me from stopping the jail, connectivity drops, and in worst case the whole host machine locks up and requires a powercycle. This is quite different behaviour compared to pre 15.0-release

Is ipfw by any chance picking up the traffic that it already NATed and should go out the physical interfare and instead re-routes it to the bridge again, i.e. creating a loop?

iRobbery · 2026-04-16T10:06:22+0100

sko said:
Is ipfw by any chance picking up the traffic that it already NATed and should go out the physical interfare and instead re-routes it to the bridge again, i.e. creating a loop?

that's what i suspect yes.

sko · 2026-04-16T10:11:21+0100

iRobbery said:
that's what i suspect yes.

make that rule more specific, e.g.

Code:

-$cmd 01000 nat 1 ip from any to any out via $wanif keep-state
+$cmd 01000 nat 1 ip from table(localnets) to !table(localnets) out via $wanif keep state

(not sure if ipfw supports negation with '!' as PF does)

iRobbery · 2026-04-16T10:20:02+0100

00107 136708203 11483489052 Thu Apr 16 09:16:54 2026 skipto 1000 icmp from 10.10.2.0/24 to not 10.10.2.0/24 out via ice0 keep-state :default

The 'not' is proper syntax, but as you can see the numbers still insane high (loop probably). This happens after i try a ping 1.1.1.1 from the jail. At this point i want to comment the line again, reload ipfw on the host, and the machine just freezes, and requires a power cycle.

(keep in mind i have 3 other 14.3/14.4 machines where the exact same ruleset/configuration works just fine, this breaks my brain)