Hello FreeBSD community,
I’m encountering a persistent issue with VNET jails on my FreeBSD 14.2 system acting as a network router. The setup involves a VLAN interface (vlan1) on ix1 with the IP 192.168.98.5, serving the 192.168.98.0/24 subnet. I have a jail (local_services) configured with VNET, using epair0b with IP 192.168.98.7, and the host can ping the jail successfully. However, pings from the jail to the host (192.168.98.5) fail with “Network is unreachable,” and I don’t see the ICMP traffic on tcpdump -i vlan1 -n.
System Details:
local_services.conf
pf.conf
rc.conf (snipped)
Is this a VNET configuration issue, a VLAN tagging mismatch, or a routing problem? Why aren’t the jail’s pings reaching vlan1? Any suggestions for debugging or config adjustments would be greatly appreciated!
I’m encountering a persistent issue with VNET jails on my FreeBSD 14.2 system acting as a network router. The setup involves a VLAN interface (vlan1) on ix1 with the IP 192.168.98.5, serving the 192.168.98.0/24 subnet. I have a jail (local_services) configured with VNET, using epair0b with IP 192.168.98.7, and the host can ping the jail successfully. However, pings from the jail to the host (192.168.98.5) fail with “Network is unreachable,” and I don’t see the ICMP traffic on tcpdump -i vlan1 -n.
System Details:
- OS: FreeBSD 14.2
- Hardware: Multi-NIC system with ix0 (WAN) and ix1 (parent of vlan1).
- Network: vlan1 is tagged with VLAN ID 1, matching the switch configuration (ports 1-24, 27, 28 untagged, 25 tagged for VLAN 1). The Asus router (192.168.98.1) and *few* others (insignificant) are on the switch, with untagged ports 25 and 26.
- Jail Config: VNET with epair0b, default route 192.168.98.1, and a direct route to 192.168.98.5 via epair0b.
- pf: Enabled with rules to allow traffic, including pass on vlan1 all, but disabling pf didn’t resolve the issue.
- Host-to-jail ping (192.168.98.5 to 192.168.98.7) works, with 192.168.98.7 in arp -a.
- Jail-to-host ping fails, with no ICMP or ARP requests visible on tcpdump -i vlan1 -n.
- tcpdump -i vlan1 -n shows other subnet traffic (e.g., 10.177.9.0/24 from iLo), suggesting untagged traffic might be interfering.
- Added ifconfig epair0a vlan 1 vlandev vlan1 to ensure tagged traffic.
- Set net.inet.ip.forwarding=1 on host.
- Added a direct route in the jail to 192.168.98.5 via epair0b.
- Disabled pf temporarily, but no change (except loss of NAT, expected).
- Captured traffic on epair0a, but pings still don’t reach vlan1.
local_services.conf
Code:
local_services {
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";
allow.raw_sockets;
sysvshm = "new";
exec.clean;
mount.devfs;
devfs_ruleset = "4";
exec.jail_user = "root";
host.hostname = "${name}";
path = "/storage/jails/containers/${name}";
vnet;
vnet.interface = "epair0b";
exec.prestart = "ifconfig epair0 create || exit 1";
exec.prestart += "ifconfig epair0a up || exit 1";
exec.prestart += "ifconfig epair0a vlan 1 vlandev vlan1 || exit 1";
exec.prestart += "ifconfig epair0a inet 192.168.98.7/24 alias || exit 1";
exec.prestart += "sysctl net.inet.ip.forwarding=1";
exec.poststart = "jexec -U root ${name} sysctl net.inet.ip.forwarding=1 2>/dev/null || true";
exec.poststart += "jexec -U root ${name} route add -host 192.168.98.5 -interface epair0b 2>/dev/null || true";
exec.poststop = "sysctl net.inet.ip.forwarding=0 2>/dev/null || true";
exec.poststop += "jexec -U root ${name} sysctl net.inet.ip.forwarding=0 2>/dev/null || true";
exec.poststop += "jexec -U root ${name} route delete -host 192.168.98.5 2>/dev/null || true";
exec.poststop += "ifconfig epair0a -alias 192.168.98.7 2>/dev/null || true";
exec.poststop += "ifconfig epair0 destroy 2>/dev/null || true";
}pf.conf
Code:
# Interface definitions
ext_if="ix0"
int_if="ix1"
lan_if="vlan1"
wan_net="192.168.2.0/24"
lan_net="192.168.98.0/24"
ring_net="192.168.99.0/24"
# Tables for tracking malicious IPs
table <port_scanners> persist
table <bruteforce> persist
# NAT rules
nat on $ext_if inet proto udp from any to ($ext_if) port 51820 -> 192.168.2.2 port 51820
nat on $ext_if from 192.168.2.2 to any -> ($ext_if)
nat on $ext_if from $wan_net to any -> ($ext_if)
nat on $ext_if from $lan_net to any -> ($ext_if)
nat on $ext_if from $ring_net to any -> ($ext_if)
# Block known malicious IPs
block in quick on $ext_if from <port_scanners> to any
block in quick on $ext_if from <bruteforce> to any
# Default deny policy
block in log on $ext_if
# Allow Ring UDP traffic (video streaming)
pass in on $ext_if proto udp from 3.138.237.19 port 30595 to $ring_net keep state
pass out on $ext_if proto udp from $ring_net to 3.138.237.19 port 30595 keep state
# Allow Ring TCP traffic (HTTPS to AWS servers)
pass in on $ext_if proto tcp from {15.197.190.1, 3.129.199.65, 3.131.145.165, 3.231.47.171, 52.0.171.160, 18.233.69.115, 52.3.90.173} port 443 to $ring_net flags S/SA keep state
pass out on $ext_if proto tcp from $ring_net to {15.197.190.1, 3.129.199.65, 3.131.145.165, 3.231.47.171, 52.0.171.160, 18.233.69.115, 52.3.90.173} port 443 flags S/SA keep state
# Allow gaming traffic
pass in on $ext_if proto udp from any to $lan_net port {3074, 27015:27030} keep state
pass out on $ext_if proto udp from $lan_net to any port {3074, 27015:27030} keep state
# Allow camera traffic
pass in on $ext_if proto {tcp, udp} from any to $ring_net port {80, 554, 443} keep state
pass out on $ext_if proto {tcp, udp} from $ring_net to any port {80, 554, 443} keep state
# Allow inbound TCP (port scans to be handled manually)
pass in log on $ext_if proto tcp from any to ($ext_if) port 1:65535 flags S/SA keep state
# Allow inbound TCP/UDP
pass in log on $ext_if proto { tcp, udp } from any to ($ext_if) keep state
# Allow WireGuard traffic to the router
pass in log on $ext_if proto udp from any to 192.168.2.2 port 51820 keep state
pass out log on $int_if proto udp from any to 192.168.2.2 port 51820 keep state
# Allow outbound traffic from internal networks
pass out on $ext_if from ($ext_if) to any keep state
# Allow inbound replies to established connections
pass in on $ext_if from any to ($ext_if) keep state
# Allow all traffic on the internal interface
pass on $int_if keep state
# Allow all traffic on the LAN interface (jails) with explicit protocols
pass on $lan_if proto icmp all keep state
pass on $lan_if proto { tcp, udp } all keep staterc.conf (snipped)
Code:
# Network configuration
ifconfig_ix0="DHCP"
ifconfig_ix1="up"
ifconfig_vlan1="inet 192.168.98.5 netmask 255.255.255.0 vlan 1 vlandev ix1"
gateway_enable="YES"
pf_enable="YES"
pflog_enable="YES"
pf_rules="/etc/pf.conf"
Code:
root@vector_edge:/etc # ifconfig vlan1
vlan1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG>
ether 38:ea:a7:32:1a:8d
inet 192.168.98.5 netmask 0xffffff00 broadcast 192.168.98.255
groups: vlan
vlan: 1 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@vector_edge:/etc # jexec 1 netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.98.1 UGS epair0b
127.0.0.1 link#8 UH lo0
192.168.98.0/24 link#7 U epair0b
192.168.98.7 link#8 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 link#8 URS lo0
::1 link#8 UHS lo0
::ffff:0.0.0.0/96 link#8 URS lo0
fe80::%lo0/10 link#8 URS lo0
fe80::%lo0/64 link#8 U lo0
fe80::1%lo0 link#8 UHS lo0
ff02::/16 link#8 URS lo0
root@vector_edge:/etc #
