jails VNET Jail fails to ping outside default network.

[Ferocious1881]

1. Description of the problem​

When running ping against 1921.68.1.1 or another address outside of the FreeBSD internal kernel network or the virt-manager default network inside a VNET jail inside a libvirt virtual machine drops all packages. On a linux host.

2. Replication instructions​

2.1 Setup and start of the libvirt virtual machine.​

On a KVM enabled linux host setup the virtual machine, where virt-manager default network is up and running, continue as follows:

1. Download FreeBSD-15.1-RELEASE-amd64-dvd1.iso and the checksum file CHECKSUM.SHA512-FreeBSD-15.1-RELEASE-amd64.
2. Check the checksum with sha512sum --ignore-missing -c CHECKSUM.SHA512-FreeBSD-15.1-RELEASE-amd64.
3. Run mount -o loop FreeBSD-15.1-RELEASE-amd64-dvd1.iso $AMYP, where $AMYP is a mountpoint of your preference.
4. Run cp -ar $AMYP $SWP , where $AMYP is the mountpoint where $SWP is a writable place where you can work.
5. Now run
   

   cat >> $SWP/$(dirname $AMYP )/boot/loader.conf <<EOF
   boot_multicons="YES"
   boot_serial="YES"
   comconsole_speed="115200"
   console="comconsole,vidconsole"
   EOF

   which should configure the console as in Chapter 30. Serial Communications 0.6.3. Setting a Faster Serial Port Speed.
6. Finally build the new disk with
   

   CUSTOM_ISO_TITLE=$(isoinfo -d -i ${ISO_DIR}/${ISO} | grep "Volume id" | awk '{print $3}')
   sudo mkisofs -J -R -no-emul-boot -V "${CUSTOM_ISO_TITLE}" -p "Joyent" -b boot/cdboot -o FreeBSD-15.1-RELEASE-SERIAL-amd64-dvd1.iso.iso $SWP/$(dirname $AMYP )

7. At last: start the VM, press enter or wait to start of the install wizard.
   

   virt-install --connect qemu:///system -n replication_base --description "Replication Base" \
       --os-variant=freebsd15.0 \
       --ram=512 --vcpus=1 \
       --graphics vnc --console pty \
       --disk path=/var/lib/libvirt/images/rb.qcow2,bus=virtio,format=qcow2,size=10 \
       --cdrom "$(pwd)/FreeBSD-15.1-RELEASE-SERIAL-amd64-dvd1.iso.iso" \
       --network network:default

2.2 Installation of FreeBSD​

Now is turn to install FreeBSD in the Virtual Machine, thus Following Handbook Chapter 2. Installing FreeBSD.

1. Now we enter the Virtual Machine installation console with virsh --connect qemu:///system console replication_base.
2. Following 2.4.1. FreeBSD Boot Loader Menu is trivial and needs no commentary.
3. Given that we are in a console continuing with the default keymap is the proper choice for 2.5.1. Selecting the Keymap Menu.
4. Set hostname to " example" on 2.5.2. Setting the Hostname.
5. On 2.5.3. Selecting Installation Type select Distributions Sets.
6. On 2.5.4. Selecting Components to Install only select lib32.
7. On 2.6. Allocating Disk Space select Auto (ZFS).
8. On 2.6.4. Guided Partitioning Using Root-on-ZFS set Swap Size to 0 and then "proceed with Installation".
9. On Figure 21. ZFS Pool Type, select stripe.
10. On Figure 22. Disk Selection select vtbd0 with space and the continue with enter.
11. On Figure 28. Last Chance select YES.
12. On 2.8.1. Setting the root Password skip.
13. On 2.8.2. Configuring Network Interfaces select manual, Yes for configuration of IPv4, Yes for DHCP of the IPv4, No for IPv6.
14. On Figure 45. DNS Configuration just go with OK.
15. On 2.8.3. Setting the Time Zone select 0 UTC, and then confirm on Figure 50. Confirm Time Zone.
16. Now skip Figure 51. Select Date and Figure 52. Select Time.
17. Enable all services on 2.8.4. Enabling Services.
18. Enable all except hide_uids, hide_gids, and random_pids on 2.8.5. Enabling Hardening Security Options.
19. On 2.8.6. Add Users select No.
20. On 2.8.7. Final Configuration select finish.
21. On Figure 59. Manual Configuration select Yes.
22. Login as root (there is no password, so only press enter).
23. Now add
    

    boot_multicons="YES"
    boot_serial="YES"
    comconsole_speed="115200"
    console="comconsole,vidconsole"

    to /mnt/boot/loader.conf as recommened by Chapter 30. Serial Communications 0.6.3. Setting a Faster Serial Port Speed.
24. Finally run poweroff.

Now FreeBSD should be already installed in the Virtual Machine, and reconnecting with the console should give access to the system.

2.3 First Start and Configuration​

1. Before finally starting with the replication, go to /etc/resolv.conf and uncomment the nameserver, as otherwise you will be getting Transient resolver failure errors.
2. Run pkg for installing pkg, and accept with y the installation prompt.
3. Run pkg update -f for updating the package list as seen in Cannot install pakcages in fresh installation FreeBSD 15.1-RELEASE-dvd1.

2.3 Create the VNET Jail​

Create a file named jail.conf with the following content

example {
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown jail";
exec.consolelog = "/var/log/jail_console_${name}.log";

allow.raw_sockets;
exec.clean;
mount.devfs;
securelevel = 1;
enforce_statfs = 2;
devfs_ruleset = 25;

path = "/jails/${name}";
host.hostname = "${name}";

$unterid = "101";
$id = "${unterid}";
$ip = "192.168.122.${unterid}/24";
$gateway = "192.168.122.100";
$bridge = "bridge0";
$epair = "epair${id}";

vnet;
vnet.interface = "${epair}b";

exec.prestart  = "/sbin/ifconfig ${epair} create up";
exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.prestart += "/sbin/ifconfig ${epair}a ether 00:00:00:00:00:0a";
exec.prestart += "/sbin/ifconfig ${epair}b ether 00:00:00:00:00:0b";
exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
exec.start    += "/sbin/route add default ${gateway}";
exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}

then procced the following:

1. Run zfs create zroot/jails.
2. Run zfs create zroot/jails/example.
3. Run zfs set mountpoint=/jails zroot/jails.
4. Install bsd on /jails/example[cmd] with [cmd]bsdinstall jail /jails/example without installing any service or additional user.
5. Run zfs snapshot zroot/jails/example@base.
6. Create bridge0 with ifconfig bridge0 create addm vtnet0 up.
7. Run dhclient bridge0.
8. Create the jail jail -crm -f jail.conf.

3. Replication of the problem​

Run jexec example ping 1921.68.1.1 or replacing 1921.68.1.1 to any other ip address outside your FreeBSD contained IP or default virt-manager network. To get nothing and when exiting ping get a message of all packages drooped.