Solved iocage jail with vnet cannot ping

Hi, I'm trying to configre vnet on an iocage jail.

I want to use a different ip range than the host. The host is 192.168.0.111 and the jail should be 192.168.1.102.

The relevant parts of /etc/rc.conf are:

Code:
hostname="ws1.local.domain"
ifconfig_re0="inet 192.168.0.111 netmask 0xffffff00"
defaultrouter="192.168.0.1"
zfs_enable="YES"
iohyve_enable="YES"
cloned_interfaces="bridge0 tap0 tap1 lo1"
ifconfig_bridge0="addm re0 addm tap0 addm tap1"
gateway_enable="YES"
pf_enable="yes"
pf_rules="/etc/pf.conf"
pflog_enable="yes"
pflog_logfile="/var/log/pflog"
iocage_enable="YES

Note: tap0 and tap1 are used by two Bhyve VMs.

And /etc/sysctl.conf:

Code:
# $FreeBSD: releng/12.1/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.zfs.min_auto_ashift=12
vfs.usermount=1
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1       # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0  # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0  # Packet filter on the bridge interface
net.link.bridge.pfil_member=0  # Packet filter on the member interface

I created the jail using:

Code:
iocage create -n jupyterlab \
    ip4_addr="vnet1|192.168.1.102" \
    interfaces="vnet1:bridge0" \
    -r 12.1-release \
    vnet="on" \
    allow_raw_sockets="1" \
    boot="on" \
    defaultrouter="192.168.1.101"

ifconfig on host:

Code:
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether b4:2e:99:ea:d3:6c
    inet 192.168.0.111 netmask 0xffffff00 broadcast 192.168.0.255
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:bb:b1:a4:5b:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet1.23 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether 00:bd:84:ff:f6:00
    groups: tap
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 2193
tap1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether 00:bd:86:ff:f6:01
    groups: tap
    media: Ethernet autoselect
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 192.168.1.101 netmask 0xffffff00
    groups: lo
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
vnet1.23: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: jupyterlab as nic: epair1b
    options=8<VLAN_MTU>
    ether b4:2e:99:f3:fb:28
    hwaddr 02:2c:55:7c:30:0a
    inet6 fe80::b62e:99ff:fef3:fb28%vnet1.23 prefixlen 64 scopeid 0x8
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

and netstat -rn on the jail:

Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.101      UGS     epair1b
127.0.0.1          link#1             UH          lo0
192.168.1.0/24     link#3             U       epair1b
192.168.1.102      link#3             UHS         lo0

Now I cannot ping anywhere, not 192.168.1.101, nor 192.168.0.111.
 
iocage has cost me quite some time to debug and nerves with errors like you mention. I have moved hundreds of jails away from iocage to the base tools and have never looked back, so I am sorry that I cannot help you. But I can highly recommend you to just build the jail with the tools from the base system - it is quite easy. Additionally, if python/iocage/whatever component needed breaks after an update you are not left with non-functioning jails...
 
The vnet1.23 is a vlan(4) interface. So the traffic on bridge0 is VLAN tagged traffic. Traffic on re0 is untagged, the rest of your network probably is too.
 
The vnet1.23 is a vlan(4) interface. So the traffic on bridge0 is VLAN tagged traffic. Traffic on re0 is untagged, the rest of your network probably is too.
Thanks, I just want to create a simple vnet jail for testing. What is the less complicated method of creating such jail?
 
just create a zfs dataset and install with bsdinstall jail /your/dataset, and then you can use that snippet in your jail.conf:
Code:
jailname {
    host.hostname = jailname;
    vnet;
    vnet.interface = "epair11b";
    path ="/path/to/jail";                     # Path to the jail
    exec.prestart += "ifconfig epair11 create";
    exec.prestart += "ifconfig epair11a up";
    exec.prestart += "ifconfig bridge0 addm epair11a";
    exec.prestop += "ifconfig epair11b -vnet $name";
    exec.poststop += "ifconfig epair11a destroy";
}

and in your jail in rc.conf you configure your epair11b + defaultrouter. (epair11 is just one of my configs)
 
Well I was able to create the jail, but still cannot ping to the host and from host to the jail.

Host IP is 192.168.0.111, jail IP is 192.168.1.102.

Here's my /etc/jail.conf

Code:
# Global settings applied to all jails
host.hostname = "$name.domain.local";
path = "/datos/jails/$name";
exec.consolelog = "/var/log/jail.$name.console.log";

vnet;
vnet.interface = ${epair}b;
exec.prestart += "ifconfig $epair create up";
exec.prestart += "ifconfig bridge0 addm ${epair}a";
exec.created = "ifconfig ${epair}b";
exec.clean;
exec.start = "ifconfig ${epair}b inet 192.168.1.$ip/16";
exec.start += "route add default 192.168.0.111";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig bridge0 deletem ${epair}a";
exec.poststop += "sleep 2";
exec.poststop += "ifconfig ${epair}a destroy";

jupyterlab {
    $ip = 102;
    $epair = "epair$ip";
}

/etc/rc.conf

Code:
hostname="ws1.local.domain"
ifconfig_re0="inet 192.168.0.111 netmask 0xffffff00"
defaultrouter="192.168.0.1"
sshd_enable="YES"
kld_list="linux vmm nmdm nvidia nvidia-modeset fuse"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
dbus_enable="YES"
linux_load="YES"
nvidia_load="YES"
nvidia_name="nvidia"
nvidia_modeset_load="YES"
nvidia_modeset_name="nvidia-modeset"
nvidia_modeset_enable="YES"
vmm_load="YES"
nmdm_load="YES"
iohyve_enable="YES"
slim_enable="YES"
vboxnet_enable="YES"
vm_enable="YES"
vm_dir="zfs:datos/vms"
vm_list=""
vm_delay="5"
cloned_interfaces="bridge0 tap0 tap1 lo1"
ifconfig_bridge0="addm re0 addm tap0 addm tap1"
gateway_enable="YES"
pf_enable="yes"
pf_rules="/etc/pf.conf"
pflog_enable="yes"
pflog_logfile="/var/log/pflog"
jupyter_enable="YES"
iocage_enable="YES"
jail_enable="YES"
jail_list="jupyterlab

/etc/sysctl.conf

Code:
# $FreeBSD: releng/12.1/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.zfs.min_auto_ashift=12
vfs.usermount=1
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1       # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0  # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0  # Packet filter on the bridge interface
net.link.bridge.pfil_member=0  # Packet filter on the member interface

/etc/pf.conf
Code:
# vim: set ft=pf
# /etc/pf.conf

#Declare the interfaces, Public IP, private subnet,
EXT_IF0 = "re0"
EXT_IF1 = "lo1"

IP_PUB="192.168.0.111"
NET_JAIL="192.168.1.0/24"
LAN_IP="192.168.0.1"
PSQL_JAIL_IP="192.168.1.101"
JUPYTERLAB_JAIL_IP="192.168.1.102"
nat pass on $EXT_IF0 from $NET_JAIL to any -> $IP_PUB
nat pass on $EXT_IF1 from $NET_JAIL to any -> $LAN_IP
# PostgreSql Jail
rdr on $EXT_IF0 proto tcp from any to $IP_PUB port 5432 -> $PSQL_JAIL_IP
rdr on lo0 proto tcp from any to 127.0.0.1 port 5432 -> $PSQL_JAIL_IP
# JupyterLab Jail
rdr on $EXT_IF0 proto tcp from any to $IP_PUB port 8888 -> $JUPYTERLAB_JAIL_IP
rdr on lo0 proto tcp from any to 127.0.0.1 port 8888 -> $JUPYTERLAB_JAIL_IP
 
You appear to be missing allow.raw_sockets in your jail.
Code:
             allow.raw_sockets
                     The jail root is allowed to create raw sockets.  Setting
                     this parameter allows utilities like ping(8) and
                     traceroute(8) to operate inside the jail.  If this is
                     set, the source IP addresses are enforced to comply with
                     the IP address bound to the jail, regardless of whether
                     or not the IP_HDRINCL flag has been set on the socket.
                     Since raw sockets can be used to configure and interact
                     with various network subsystems, extra caution should be
                     used where privileged access to jails is given out to
                     untrusted parties.

Not related to your issue, just something I noticed in your /etc/rc.conf:
Code:
nvidia_load="YES"
nvidia_name="nvidia"
nvidia_modeset_load="YES"
nvidia_modeset_name="nvidia-modeset"
nvidia_modeset_enable="YES"
Remove these, they do absolutely nothing.
 
Thanks, applied those changes, but still ping commands doesn't reach the host, nor the other jails and vms.
 
Hi!
In the last example, I noticed that the jail has /16 netmask while the host has /24.
My guess is the jail can reach the host directly but the host forwards packets destined for the jail to its default gateway(192.168.0.1), which might not know the jail's network.

Here's my /etc/jail.conf
Code:
exec.start = "ifconfig ${epair}b inet 192.168.1.$ip/16";
exec.start += "route add default 192.168.0.111";

/etc/rc.conf
Code:
ifconfig_re0="inet 192.168.0.111 netmask 0xffffff00"
defaultrouter="192.168.0.1"
 
Great!, I changed both jail and host to /23 and the router to 192.168.0.1, now I can ping both ways. What I still cannot do is to reach 8.8.8.8 or 1.1.1.1 from the jail.
 
I changed the gateway from 192.168.0.1 (the router) to 192.168.0.111 (host) and now I can reach the Internet.

I'll mark this as solved.
 
martinrame So I have tried to emulate this with no luck. I am running FreeBSD 12.2 Stable in a virtual box. My host ip is 10.0.2.15 my gateway is 10.0.2.2. I can't even get an ip for the jail. My goal is to have VPN running on the host and the jail not use the VPN interface.

My hosts /etc/rc.conf

Code:
hostname="FreeBSD12-2-STABLE"
ifconfig_em0="inet 10.0.2.15 netmask 0xfffffe00"
sshd_enable="YES"

# configs for a jail friendly system
syslogd_flags="-ss"
sendmail_enable="NONE"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
vboxguest_enable="YES"
vboxservice_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/privatvpn.conf"
#openvpn_enable="YES"

# jail options
jail_enable=YES
jail_list=""
jail_reverse_stop=YES
#jail_parallel_start=NO

# networking for jails
cloned_interfaces="bridge0 tun0 lo0"
ifconfig_bridge0="addm em0 addm tun0"
gateway_enable="YES"
defaultrouter="10.0.2.2"

# pf settings
pf_enable="YES"
pf_flags=""
pf_rules="/usr/local/etc/pf/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

My hosts /etc/jail.conf

Code:
# Global settings applied to all jails
allow.raw_sockets="1";
host.hostname = "$name.domain.local";
path = "/jail/$name";
exec.consolelog = "/var/log/jail.$name.console.log";

vnet;
vnet.interface = ${epair}b;
exec.prestart += "ifconfig $epair create up";
exec.prestart += "ifconfig bridge0 addm ${epair}a";
exec.created = "ifconfig ${epair}b";
exec.clean;
exec.start = "ifconfig ${epair}b inet 10.0.3.${ip}/23";
exec.start += "route add default 10.0.2.15";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig bridge0 delete ${epair}a";
exec.poststop += "sleep 2";
exec.poststop += "ifconfig ${epair}a destroy";

sickrage {
    $ip = 100;
    $epair = "epair$ip";
    interface = "${epair}b";
}

My hosts pf config file/usr/local/etc/pf/pf.conf

Code:
ext_if="em0"
lo="lo0"
host="10.0.2.15"
net_jail="10.0.3.0/24"
lan_ip="10.0.2.2"
sickrage_jail_ip="10.0.3.100"

nat pass on $ext_if from $net_jail to any -> $host
nat pass on $lo from $net_jail to any -> $lan_ip

host ifconfig output:


root@FreeBSD12-2-STABLE:~ # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
ether 08:00:27:41:bd:63
inet 10.0.2.15 netmask 0xfffffe00 broadcast 10.0.3.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
ether 08:00:27:e5:c8:bd
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
groups: lo
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:38:35:c3:09:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair100a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
tun0: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
groups: tun
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair100a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:0a:25:11:17:0a
inet6 fe80::a:25ff:fe11:170a%epair100a prefixlen 64 scopeid 0x8
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


host netstat -rn output


root@FreeBSD12-2-STABLE:~ # netstat -rn
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 10.0.2.2 UGS em0
10.0.2.0/23 link#1 U em0
10.0.2.15 link#1 UHS lo0
127.0.0.1 lo0 UHS lo0

Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 lo0 UHS lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#3 U lo0
fe80::1%lo0 link#3 UHS lo0
fe80::%epair100a/64 link#8 U epair100
fe80::a:25ff:fe11:170a%epair100a link#8 UHS lo0
ff02::/16 ::1 UGRS lo0



root@FreeBSD12-2-STABLE:~ # jls
JID IP Address Hostname Path
10 sickrage.domain.local /jail/sickrage


ifconfig and netstat output in the jail:


root@sickrage:/ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog
epair100b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:0a:25:11:17:0b
inet 10.0.3.100 netmask 0xfffffe00 broadcast 10.0.3.255
inet6 fe80::a:25ff:fe11:170b%epair100b prefixlen 64 tentative scopeid 0x3
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>



root@sickrage:/ # netstat -rn
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 10.0.2.15 UGS epair100
10.0.2.0/23 link#3 U epair100
10.0.3.100 link#3 UHS lo0
127.0.0.1 link#1 UH lo0

Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#1 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#1 U lo0
fe80::1%lo0 link#1 UHS lo0
fe80::%epair100b/64 link#3 U epair100
fe80::a:25ff:fe11:170b%epair100b link#3 UHS lo0
ff02::/16 ::1 UGRS lo0


Any help would be greatly appreciated.
 
Back
Top