Hello,
I've just updated my RockPro64 home NAS thingie from FreeBSD 14.3-RELEASE to 15.0-RELEASE, and I'm having issues with my VNET jails. I'm able to access machines on the bridge itself from the jail, e.g. ssh from one jail to the other, but I'm not able to access any external machine, and my external machines are not able to access the jails.
Here's the networking part of my rc.conf:
Here's my pf.conf:
Here's my jail.conf:
Here's a demonstration of the problem, SSHing into "the neighboring jail" (192.168.100.110 - works) and to my workstation (192.168.100.101 - doesn't work from jail):
I've tried disabling pf both inside and outside of the jail, but to no avail.
Here's the network configuration status from outside and within the jail:
And the console log of the jail itself:
There are some warnings in the console log about already existing route rules, but from the output of
Does anyone have any suggestions on what could've changed in FreeBSD 15.0-RELEASE? Perhaps some sysctl defaults? I've looked at other forum posts, and saw suggestions about
I'm using the GENERIC aarch64 kernel:
Thanks,
Dalibor
I've just updated my RockPro64 home NAS thingie from FreeBSD 14.3-RELEASE to 15.0-RELEASE, and I'm having issues with my VNET jails. I'm able to access machines on the bridge itself from the jail, e.g. ssh from one jail to the other, but I'm not able to access any external machine, and my external machines are not able to access the jails.
Here's the networking part of my rc.conf:
Code:
hostname="central-rock.home"
defaultrouter="192.168.100.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.100.100/24 addm dwc0 up"
ifconfig_dwc0="up"Here's my pf.conf:
Code:
scrub in
ext_if="bridge0"
anchor "blocklistd/*" in on $ext_if
block all
pass proto tcp from 127.0.0.1 to 127.0.0.1 port { rndc } keep state
pass out proto tcp to any port { ssh, domain, kerberos-sec, nfsd, sunrpc, http, https } keep state
pass in proto tcp to any port { ssh, domain, kerberos-sec, nfsd, sunrpc } keep state
pass proto udp to any port { domain, ntp, kerberos-sec, nfsd, sunrpc } keep stateHere's my jail.conf:
Code:
www {
# STARTUP/LOGGING
exec.consolelog = "/var/log/jail_console_${name}.log";
# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;
devfs_ruleset = 5;
# PATH/HOSTNAME
path = "/usr/local/jails/containers/${name}";
host.hostname = "${name}.central-rock.home";
# VNET/VIMAGE
vnet;
vnet.interface = "${epair}b";
# NETWORKS/INTERFACES
$id = "110";
$ip = "192.168.100.${id}/24";
$gateway = "192.168.100.1";
$bridge = "bridge0";
$epair = "epair${id}";
# ADD TO bridge INTERFACE
exec.prestart = "ifconfig ${epair} create up";
exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
exec.start = "ifconfig ${epair}b ${ip} up";
exec.start += "route add default ${gateway}";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "ifconfig ${epair}a destroy";
}
irc {
# STARTUP/LOGGING
exec.consolelog = "/var/log/jail_console_${name}.log";
# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;
devfs_ruleset = 5;
# PATH/HOSTNAME
path = "/usr/local/jails/containers/${name}";
host.hostname = "${name}.central-rock.home";
# VNET/VIMAGE
vnet;
vnet.interface = "${epair}b";
# NETWORKS/INTERFACES
$id = "111";
$ip = "192.168.100.${id}/24";
$gateway = "192.168.100.1";
$bridge = "bridge0";
$epair = "epair${id}";
# ADD TO bridge INTERFACE
exec.prestart = "ifconfig ${epair} create up";
exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
exec.start = "ifconfig ${epair}b ${ip} up";
exec.start += "route add default ${gateway}";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "ifconfig ${epair}a destroy";
}Here's a demonstration of the problem, SSHing into "the neighboring jail" (192.168.100.110 - works) and to my workstation (192.168.100.101 - doesn't work from jail):
Code:
root@central-rock:~ # jexec -u root irc ssh www.central-rock.home
The authenticity of host 'www.central-rock.home (192.168.100.110)' can't be established.
ED25519 key fingerprint is SHA256:Kj0ycnTB3wghkdU+fs6NpXp0nLs91oQw0StpHcgv4Xc.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^C
root@central-rock:~ # ssh sapphire-scalpel.home
root@sapphire-scalpel.home: Permission denied (publickey).
root@central-rock:~ # jexec -u root irc ssh -vvv sapphire-scalpel.home
debug1: OpenSSH_10.0p2, OpenSSL 3.5.4 30 Sep 2025
debug3: Running on FreeBSD 15.0-RELEASE FreeBSD 15.0-RELEASE releng/15.0-n280995-7aedc8de6446 GENERIC arm64
debug3: Started with: ssh -vvv sapphire-scalpel.home
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug2: resolving "sapphire-scalpel.home" port 22
debug3: resolve_host: lookup sapphire-scalpel.home:22
debug3: Fssh_channel_clear_timeouts: clearing
debug3: ssh_connect_direct: entering
debug1: Connecting to sapphire-scalpel.home [192.168.100.101] port 22.
debug3: Fssh_set_sock_tos: set socket 3 IP_TOS 0x48
<hang>
^CI've tried disabling pf both inside and outside of the jail, but to no avail.
Here's the network configuration status from outside and within the jail:
Code:
root@central-rock:~ # ifconfig -a
dwc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
ether 82:16:d0:7b:f5:0b
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=10<VLAN_HWTAGGING>
ether 58:9c:fc:10:77:67
inet 192.168.100.100 netmask 0xffffff00 broadcast 192.168.100.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge flags=0<>
member: epair111a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 9 priority 128 path cost 2000 vlan protocol 802.1q
member: epair110a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 5 priority 128 path cost 2000 vlan protocol 802.1q
member: dwc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 1 priority 128 path cost 55 vlan protocol 802.1q
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
options=0
groups: pflog
epair110a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: jail:www
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:10:24:0b
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair111a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: jail:irc
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:10:bf:1d
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@central-rock:~ # netstat -r
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.100.1 UGS bridge0
localhost link#2 UH lo0
192.168.100.0/24 link#3 U bridge0
central-rock link#2 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 link#2 URS lo0
localhost link#2 UHS lo0
::ffff:0.0.0.0/96 link#2 URS lo0
fe80::%lo0/10 link#2 URS lo0
fe80::%lo0/64 link#2 U lo0
fe80::1%lo0 link#2 UHS lo0
ff02::/16 link#2 URS lo0
root@central-rock:~ # jexec -u root irc ifconfig -a
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0 metric 0 mtu 33152
options=0
groups: pflog
epair111b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:10:98:82
inet 192.168.100.111 netmask 0xffffff00 broadcast 192.168.100.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@central-rock:~ # jexec -u root irc netstat -r
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.100.1 UGS epair111b
localhost link#11 UH lo0
192.168.100.0/24 link#10 U epair111b
irc link#11 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 link#11 URS lo0
localhost link#11 UHS lo0
::ffff:0.0.0.0/96 link#11 URS lo0
fe80::%lo0/10 link#11 URS lo0
fe80::%lo0/64 link#11 U lo0
fe80::1%lo0 link#11 UHS lo0
ff02::/16 link#11 URS lo0And the console log of the jail itself:
Code:
add net default: gateway 192.168.100.1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/perl5/5.42/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
Starting Network: lo0 epair111b.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair111b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:10:98:82
inet 192.168.100.111 netmask 0xffffff00 broadcast 192.168.100.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
route: message indicates error: File exists
add host 127.0.0.1: gateway lo0 fib 0: route already in table
route: message indicates error: File exists
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Enabling pf.
Starting blocklistd.
Updating motd:.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating /var/run/os-release done.
Starting syslogd.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Wed Dec 10 19:57:26 CET 2025There are some warnings in the console log about already existing route rules, but from the output of
netstat -r, the routing tables seem to be configured correctly. DNS also works, since I have my own DNS server running on the host system.Does anyone have any suggestions on what could've changed in FreeBSD 15.0-RELEASE? Perhaps some sysctl defaults? I've looked at other forum posts, and saw suggestions about
sysctl net.inet.ip.forwarding and sysctl net.pf.default_to_drop, but enabling the prior doesn't change anything, even if I change the jails to use the host machine as the gateway, and the latter reports 0, so it's presumably disabled.I'm using the GENERIC aarch64 kernel:
Code:
root@central-rock:~ # uname -a
FreeBSD central-rock.home 15.0-RELEASE FreeBSD 15.0-RELEASE releng/15.0-n280995-7aedc8de6446 GENERIC arm64Thanks,
Dalibor