Solved Jail / vnet problem

Hello,

I'm playing with jails with the help of sysutils/iocage. I have no problem with shared ip but I can't make VIMAGE/VNET networking work.

Some informations:
- Jails host machine: VirtualBox VM under Windows (network / attached to: bridged adapter), IP: 192.168.5.61/24, OS: FreeBSD 12.1-STABLE r356602 generic amd64, iocage: py37-iocage-1.2_4
- Jail: host_hostname: plex, 12.1-RELEASE, defaultrouter:192.168.5.1, devfs_ruleset: 1000 (cloned from devfs_ruleset 10), ip4_addr:vnet0|192.168.5.4/24, vnet:1

I used this guide to configure the jails host: https://iocage.readthedocs.io/en/latest/networking.html#vimage-vnet.

Jails host machine:/etc/rc.conf:
Code:
zfs_enable="YES"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up"
ifconfig_em0="DHCP up"
Jails host machine:/etc/sysctl.conf:
Code:
net.inet.ip.forwarding=1
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0
Jails host machine ifconfig:
Code:
# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 08:00:27:45:ac:f3
        inet 192.168.5.61 netmask 0xffffff00 broadcast 192.168.5.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:79:be:95:75:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: plex as nic: epair0b
        options=8<VLAN_MTU>
        ether 08:00:27:14:fa:09
        hwaddr 02:0e:2e:b7:5a:0a
        inet6 fe80::a00:27ff:fe14:fa09%vnet0.1 prefixlen 64 scopeid 0x4
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Plex jail ifconfig:
Code:
# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 08:00:27:14:fa:0a
        hwaddr 02:0e:2e:b7:5a:0b
        inet 192.168.5.4 netmask 0xffffff00 broadcast 192.168.5.255
        inet6 fe80::a00:27ff:fe14:fa0a%epair0b prefixlen 64 scopeid 0x2
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The problem:
I can ping the jails host from the plex jail and vice-versa but, inside the jail, I can't ping nothing except the jails host, not even the router. And I can't connect to any service that runs in the jail from my local network.

I think this is a route problem but all my trials have failed so far. For example, from another FreeBSD machine, I tried : route add 192.168.5.4/32 192.168.5.61 and no effect. I still can't ping or connect the plex jail from this PC. I also tried to add the same static route on my router (pfSense) but to no avail.

Anyone can help me?
 
I have working VNET jails system, though it's working on the bare metal server, but anyway, on my server I have this in sysctls:
Code:
net.link.bridge.pfil_member: 1
net.link.bridge.pfil_bridge: 1
net.link.bridge.pfil_onlyip: 1
As you can see, they're all different from yours. Maybe you should try to change yours.

Also check if you have ipfw running on host or in jail or both.

Also it will be good to see your /etc/jail.conf.

But there's also could be a problem with your VirtualBox network! It's also could be helpful to see VirtualBox network config.
 
Hi!
How about checking if "Promiscuous Mode" is allowed on your host VM's Bridged Adapter in VirtualBox VM Settings?
When it's set to "Deny" (default), packets from your local network to the jail's MAC address (not host's) are denied by the host VM's bridged adapter, if I remember correctly.
VirtualBox_Netif_Promiscuous.png
 
genneko, you found the trick! Thank you very much. It was that my VM was set to "deny" for the promiscuous mode.

This is why I gave all the details of my test. The cause could be anywhere, including the fact I test it in a VirtualBox VM.

Thank to you, I have now a plexmediaserver that perfectly works in a jail.
 
Back
Top