Solved VNET Jail: pfctl: pfi_get_ifaces: Operation not supported by device

martinrame · Jun 21, 2021

Hi, I'm trying to create a VNET jail with pf, but when I try to run pf from inside the jail I get pfctl: pfi_get_ifaces: Operation not supported by device.

Host and jail, both are running 13.0-RELEASE.

Host: /etc/jail.conf

Code:

honeypot {                                                                                 
  $id   = "52";                                                                            
  $addr   = "192.168.100.200";                                                             
  $mask   = "255.255.255.0";                                                               
  $gw   = "192.168.100.1";                                                                 
  vnet;                                                                                    
  vnet.interface  = "epair${id}b";                                                         
                                                                                           
  exec.prestart = "ifconfig epair${id} create up";                                         
  exec.prestart += "ifconfig epair${id}a up descr vnet-${name}";                           
  exec.prestart += "ifconfig bridge0 addm epair${id}a up";                                 
                                                                                           
  exec.start  = "/sbin/ifconfig lo0 127.0.0.1 up";                                         
  exec.start  += "/sbin/ifconfig epair${id}b ${addr} netmask ${mask} up";                  
  exec.start  += "/sbin/route add default ${gw}";                                          
                                                                                           
  exec.poststop   = "ifconfig bridge0 deletem epair${id}a";                                
  exec.poststop  += "ifconfig epair${id}a destroy";                                        
                                                                                           
  host.hostname = "${name}.bsd.am";                                                        
  exec.consolelog = "/var/log/jail-${name}.log";                                           
  persist;                                                                                 
                                                                                           
  allow.raw_sockets;                                                                       
  allow.set_hostname;                                                                      
  securelevel = 2;                                                                         
                                                                                           
  enforce_statfs = 2;                                                                      
  devfs_ruleset="5";                                                                       
  mount.devfs;                                                                             
}

Host: /etc/devfs.conf

Code:

[devfsrules_jail_with_vpn=5]                                                               
add include $devfsrules_hide_all                                                           
add include $devfsrules_unhide_basic                                                       
add include $devfsrules_unhide_login                                                       
add include $devfsrules_jail                                                               
add path 'tun*' unhide                                                                     
add path "bpf*" unhide                                                                     
add path "pf*" unhide

Host: /etc/rc.conf

Code:

hostname="ws1.local.domain"
ifconfig_re0="inet 192.168.100.111 netmask 0xffffff00"
defaultrouter="192.168.100.1"
sshd_enable="YES"
kld_list="linux vmm nmdm nvidia nvidia-modeset fusefs"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
dbus_enable="YES"
linux_load="YES"
linux_enable="YES"
vmm_load="YES"
nmdm_load="YES"
cupsd_enable="YES"
slim_enable="YES"
vboxnet_enable="NO"
vm_enable="YES"
vm_dir="zfs:zroot/vms"
vm_list="devel senaite"
vm_delay="5"
cloned_interfaces="bridge0 lo1"
ifconfig_bridge0="addm re0 lo1"
gateway_enable="YES"
pf_enable="yes"
pf_rules="/etc/pf.conf"
pflog_enable="NO"
#pflog_logfile="/var/log/pflog"
pflog0_enable="NO"
jupyter_enable="YES"
jail_enable="YES"
jail_list="unbound postgresql jupyterlab honeypot"
webcamd_enable="YES"
ntpd_program="/usr/local/sbin/ntpd"
ntpdate_program="/usr/local/sbin/ntpdate"

Host: /etc/sysctl.conf

Code:

# $FreeBSD$
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.zfs.min_auto_ashift=12
vfs.usermount=1
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1       # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0  # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0  # Packet filter on the bridge interface
net.link.bridge.pfil_member=0  # Packet filter on the member interface
hw.snd.default_unit=5 # se usa pcm5 (webcam e-view con micrófono incorporado

Host: ifconfig

Code:

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether b4:2e:99:ea:d3:6c
    inet 192.168.100.111 netmask 0xffffff00 broadcast 192.168.100.255
    inet 192.168.100.205 netmask 0xffffffff broadcast 192.168.100.205
    inet 192.168.100.203 netmask 0xffffffff broadcast 192.168.100.203
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-public
    ether 58:9c:fc:10:8c:25
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair52a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
    groups: bridge vm-switch viid-4c918@
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.1.5 netmask 0xffffffff
    inet 127.0.1.4 netmask 0xffffffff
    inet 127.0.1.1 netmask 0xffffffff
    inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-devel-0-public
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:de:73
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1659
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-senaite-0-public
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:fd
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1983
epair52a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vnet-honeypot
    options=8<VLAN_MTU>
    ether 02:74:1b:ea:1e:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Jail: ifconfig

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair52b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:98:23:22:35:0b
    inet 192.168.100.200 netmask 0xffffff00 broadcast 192.168.100.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

martinrame · Jun 21, 2021

Apart from this, I noticed when the host boots then I start the jail I can ping 192.168.100.200 from the host (192.168.100.111), but if I restart the jail sudo service jail restart honeypot I no longer can ping to it.

rootbert · Jun 21, 2021

could you double-check your FreeBSD version running on host and jail? I encountered the same error when I tried upgrading a jail on a host where I missed the final "freebsd-update install"

martinrame · Jun 21, 2021

Yes:

Jail:

Code:

root@honeypot:/ # freebsd-version
13.0-RELEASE-p2

Host:

Code:

[leonardo@ws1 ~] $ freebsd-version
13.0-RELEASE

martinrame · Jun 21, 2021

Upgrading host and rebooting...

martinrame · Jun 21, 2021

Now both are 13.0-RELEASE P2, but the error still shows.

martinrame · Jun 22, 2021

rootbert, you were right, I was upgrading the jail and that showed `13.0-RELEASE p2`, but that wasn't enough.

I created a new 13.0-RELEASE jail (after fighting with bsdinstall 12.1 see here) and that was it, configured pf just like in a physical machine.

jhfoo · Nov 6, 2021

martinrame did you solve this? I'm getting the same error and it's been frustrating not getting similar reports other than your post.

jhfoo · Nov 6, 2021

SOLVED: I upgraded my jail to match the same release and patch (13.0-RELEASE p5) as the host and now fctl works as expected.