problems with vnet in jail

Hello All,
I would like too set up an iocage jail with vnet but can't make my way forward with it. I have followed the iocage documentation but have the following problem.
The jail in on the host with IP 192.168.1.1 which faces LAN. There is another physical interface on the host which faces WAN.

I have put this into /etc/sysctl.conf
Code:
net.inet.ip.forwarding=1 
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0

and this into /etc/rc.conf

Code:
ifconfig_igb0="inet 192.168.1.1 netmask 255.255.255.0"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"

This is the ifconfig output

Code:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 90:1b:0e:89:41:e9
        inet 172.20.1.0 netmask 0xfffffff8 broadcast 172.20.1.7
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 90:1b:0e:6b:c6:d4
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.102 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492
        options=80000<LINKSTATE>
        inet X.X.X.X--> Y.Y.Y.Y netmask 0xffffffff
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 705
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:e3:8d:a8:06:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

The 192.168.1.100-103 are other non-vnet jails

Then I set up an iocage jail with the following adjustments
Code:
sudo iocage set vnet=on JAIL
sudo iocage set defaultrouter=192.168.1.1 JAIL
sudo iocage set ip4_addr="vnet0|191.168.1.104/24" JAIL

But when I try to start the jail I get the following error
Code:
No default gateway found for ipv6.
* Starting JAIL
+ Started OK
+ Using devfs_ruleset: 1004 (iocage generated default)
+ Configuring VNET FAILED
route: writing to routing socket: Network is unreachable
add net default: gateway 192.168.1.1 fib 0: Network is unreachable

Stopped JAIL due to VNET failure

Why is this happening? I have other jails on the same host without VNET which use 192.168.1.1 gateway without a a problem as other computers on the LAN which do the same?

Best regards,
T
 
Hello All,
I have deleted the jail and recreated it without making any changes. For some reason now the error above is gone and the jail starts.

It is still not functioning as intended though.
The jail can't be pinged and also from inside of the jail nothing can be pinged.


These are some commands ran from inside of the jail

Code:
root@emby2:~ # ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
ping: sendto: Permission denied

Code:
root@emby2:~ # pkg upgrade
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly/Latest/pkg.txz: No address record
Address resolution failed for http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly.
Consider changing PACKAGESITE.

Code:
root@emby2:~ # drill google.com
Error: error sending query: Error creating socket

The /etc/resolv.conf inside the jail is the same as on the host and other non-vnet jails.



This is ifconfig inside the jail

Code:
        lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:60:77:9d:a0
        hwaddr 02:59:28:5f:8f:0b
        inet 192.168.1.104 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::ff:60ff:fe77:9da0%epair0b prefixlen 64 scopeid 0x2
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>



This is ifconfig on the host

Code:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 90:1b:0e:89:41:e9
        inet 172.20.1.0 netmask 0xfffffff8 broadcast 172.20.1.7
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether 90:1b:0e:6b:c6:d4
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.102 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:e3:8d:a8:06:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492
        options=80000<LINKSTATE>
        inet X.X.X.X --> Y.Y.Y.Y netmask 0xffffffff
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 790
vnet0.9: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: emby2 as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:77:9d:9f
        hwaddr 02:59:28:5f:8f:0a
        inet6 fe80::ff:60ff:fe77:9d9f%vnet0.9 prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Can you help me to understand what is going on?

Best regards,
T
 
sorry I cannot help ... all I can say I have also had many problems with iocage and network/vnet setup and highly recommend staying away from iocage. The tools delivered by the base system are really great and managing jails with those and maybe some scripts is no more difficult than handling iocage
 
Concerning the ping, it's because raw sockets aren't allowed by default in jail.
See iocage(8)

allow_raw_sockets=[1 | 0]
The prison root is allowed to create raw sockets. Setting
this parameter allows utilities like ping(8) and
traceroute(8) to operate inside the prison. If set, the
source IP addresses are enforced to comply with the IP ad-
dress bound to the jail, regardless of whether the
IP_HDRINCL flag has been set on the socket. Since raw
sockets can be used to configure and interact with various
network subsystems, extra caution should be used where
privileged access to jails is given out to untrusted par-
ties.

But this is not your main concern. You should have access to the network otherwise.
I would need to access to my jails to help you, but I can't for the moment.
I think that someone else will help you in the meantime.
 
So, where's the epair0a interface? Should be there on the host…
(and then you'd probably want to connect it to your bridge for a "virtual switch")
 
So, where's the epair0a interface? Should be there on the host…
(and then you'd probably want to connect it to your bridge for a "virtual switch")
Hello. I am just a newbie and don't quite understand what you mean.
epair0b is mentioned under vnet0.9 though.

Emrion
I have turned on the raw sockets on the jail, I forgot to mention.

rootbert
Yes, I have seen quite a few posts recommending using the base tools but it seems to be a bit over my head. Until this Vnet problem iocage was working well for me and I see other reports where they have vnet set up. It may be some other settings in my system that are causing the mischief.
 
You're mixing tagged and untagged vlan(4) traffic on your bridge. vnet0.9 is a vlan(4) interface causing the traffic on bridge0 to be tagged (VLAN ID 9). It's however bridged to igb0 which is untagged. So you now have tagged and untagged VLAN traffic on igb0.
 
SirDice, thank you for your input. When searching forums to solve the problem I have seen you saying something similar in a post to another user 3or so weeks ago but he didn't engage further.

I haven't made any of those arrangements you are describing, I just followed the iocage docs and also the howto here where the user had no such issue
https://forums.freebsd.org/threads/jailed-plex-server-with-iocage.73794/

How can I correct this situation?
 
One of my jails is precisely a plex server using iocage and vnet. I didn't follow this howto which seems somewhat overkill concerning the kernel tunables.
 
One of my jails is precisely a plex server using iocage and vnet. I didn't follow this howto which seems somewhat overkill concerning the kernel tunables.
I haven't done any of the kernel tunables.
The setup of the vnet closely follows the iocage docs.
 
I don't see any obvious mistake. My config seems close to yours.
With VNET, pings are allowed anyway.

What are the freebsd-version of the host and this jail, by the way?

Could you post the (whole) content of /etc/rc.conf and /etc/sysctl.conf of the host and the result of iocage get all JAIL?

Also, have you some firewall rules on the host?
 
Thank you Emrion.
Both host and jail are running 12.2-p3

Here is the requested output, a lot of text



here is /etc/rc.conf
Code:
$ cat /etc/rc.conf
clear_tmp_enable="YES"
hostname="blow"
keymap="uk"

# Network configuration
ifconfig_em0="172.20.1.0/29 up"
ifconfig_igb0="inet 192.168.1.1 netmask 255.255.255.0"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"

# Start ssh and time sync
sshd_enable="YES"
ntpd_enable="YES"

# Needed for firewall and NAT
gateway_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

# Enable ZFS
zfs_enable="YES"

# Disable sendmail
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Connect to internet
ppp_enable="YES"
ppp_nat="NO"
ppp_profile="net"
ppp_mode="ddial"

# Start firewall
firewall_enable="YES"
firewall_logging="YES"
firewall_nat_enable="YES"
firewall_script="/etc/ipfw.rules"

#start iocage
iocage_enable="YES"

# Start fail2ban
fail2ban_enable="YES"

# Start DNS server
dnsmasq_enable="YES"

# XOrg needs this
dbus_enable="YES"

# To allow automount usb sticks
autofs_enable="YES"

Here is /etc/sysctl.conf
[CODE
]$ cat /etc/sysctl.conf
# $FreeBSD: releng/12.2/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $
#
# This file is read when going to multi-user and its contents piped thru
# ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0

# Allow more than one pass through FW
net.inet.ip.fw.one_pass=0

######this was added for vnet support as per iocage readthedocs
net.inet.ip.forwarding=1 # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0 # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0 # Packet filter on the bridge interface
net.link.bridge.pfil_member=0 # Packet filter on the member interface
[/CODE]


Now all the switches on the jail

Code:
$ iocage get all emby2
CONFIG_VERSION:27
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:0
bpf:0
children_max:0
cloned_release:12.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.1.1
defaultrouter6:auto
depends:none
devfs_ruleset:1004
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:emby2
host_hostuuid:emby2
host_time:1
hostid:ca2ad479-05c4-11e7-ad91-901b0e8941e9
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:192.168.1.104/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/emby2/data
jail_zfs_mountpoint:none
last_started:2021-02-19 19:53:46
localhost_ip:none
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:12.2-RELEASE-p3
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:02ff60779d9f 02ff60779da0
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:auto
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off
 
Last edited:
Is it possible to deactivate ipfw a few seconds, just to test if your plex jail is working better?
Thanks.
It doesn't make any difference.
The DNS names don't resolve from within the jail.
Also, vnet jail can't be pinged from the host, while the other non-vnet jails can.
Are there any other suggestions pls?
 
It appears that in the property ip4_addr, you forgot to mention the interface.

From iocage(8)
ip4_addr="interface|ip-address/netmask"
The IPv4 address for VNET and shared IP jails.

You should have: ip4_addr:vnet0|192.168.1.104/24
Please, modify this property and retry.
 
Thanks, Emrion.
The iocage doc recommend to specify the interface which was my starting point. It wasn't working, the problem was the same.
Then I noticed that in the plex howto linked above they don't specify the interface so I tried that, no change. The setting stayed on.
 
After testing all differences we have between our respective iocage settings, I came to the conclusion that they aren't the problem. The main differences between our two machines are: I have only one physical interface (which brings internet data) and no firewall.

When I first set up these jails, I did it in a VM (Virtual Box). The problem I encountered was that the VM, by default, refuses to set its network interface into promiscuous mode. All jails worked except the one with VNET.

This brings a question: what does return this command dmesg | grep promiscuous?
Have you rebooted recently your machine?
 
Hello Emrion,
Thanks.
There is no output from that command. What does it mean?

I have recently rebooted the machine, yes. Why?
 
igb0 and vnet0.x (x is variable) should be in promiscuous mode. I think this is why the VNET jail doesn't work. That being said, I don't know why they aren't in promiscuous mode.

There is no more igb man page since FreeBSD-12.0. It seems that the igb code has been more or less merged with the one of em(4). Could it be a bug or a drawback that comes from ipfw?
 
Hello Emrion,
I have spent countless hours trying to sort it out but I can't break through.

The output of dmesg | grep promiscuous is now
Code:
igb0: promiscuous mode enabled
vnet0.2: promiscuous mode enabled

However, there is also a strange error in the dmesg
Code:
epair0a: Ethernet address: 02:1d:6c:53:95:0a
epair0b: Ethernet address: 02:1d:6c:53:95:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: changing name to 'vnet0.2'
ng_ether_ifnet_arrival_event: can't re-name node epair0b
igb0: link state changed to DOWN
vnet0.2: promiscuous mode enabled
lo0: link state changed to UP
igb0: link state changed to UP

What does it mean?
ng_ether_ifnet_arrival_event: can't re-name node epair0b

I don't know what the significance of that is, but epair0a on the host is renamed to vnet0.2. Does a similar even need to take place int the jail for these to match?

Really desperate now.....
 
I can just tell you what dmesg writes for me:
* [I|O|C] starting jails...
epair0a: Ethernet address: 02:89:c4:d7:11:0a
epair0b: Ethernet address: 02:89:c4:d7:11:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: changing name to 'vnet0.1'
vnet0.1: promiscuous mode enabled
I haven't any error message concerning epair0b.
Have you tried to reboot without ipfw enabled?
 
The firewall is the first thing to look when you have network troubles. It's a basic rule, but it doesn't mean the firewall is responsible of all network problems.

I'm curious to know what you have done to bring these interfaces in promiscuous mode. Maybe there is here the answer of this mysterious message concerning the renaming of epair0b.
 
My firewall has no restrictions on the communication through the LAN interface though.

To get the promiscuous mode on I manually added a vlan and then deleted it. Ever since then everytime an iocage vnet jail starts the promiscuous mode gets activated.

I don't know what iocage actually does when configuring the vnet, before the vnet jail starts and how it structures its vnets

When I add another vnet jail, its internal epair is renamed to the epair of the first jail.
E.g, first vnet jail created got epair0, the second jail to be created got epair1.
When starting the second jail I get the following in the dmesg

Code:
epair1a: Ethernet address: 02:59:28:5f:8f:0a
epair1b: Ethernet address: 02:59:28:5f:8f:0b
epair1a: link state changed to UP
epair1b: link state changed to UP
epair1a: changing name to 'vnet0.9'
ng_ether_ifnet_arrival_event: can't re-name node epair1b
epair1b: changing name to 'epair0b'
vnet0.9: promiscuous mode enabled
lo0: link state changed to UP

Why does the epair1b get renamed to epair0b (which is then duplicate of internal epair of the first jail?

I am very close to give up on this now as it was just a little project on the side which has not taken too much time and I don't know where to go next.
 
I found the solution for this.

Emrion, you were right about your pointer to suspect the firewall to be the culprit.
The above error was actually a red herring.

When ipfw runs on the host, open ipfw firewall needs to be run in the jail. Putting this into the /etc/rc.conf inside the jail sorts the problem out.

Code:
firewall_enable="YES"
firewall_type="open"

Thanks for your time.
T
 
Back
Top