Issues assigning epair interface to vnet jail

[thecodechemist]

Hi,
I'm having issues with VNET on my FreeBSD 13.0-RELEASE-p11 host running on a cloud VPS.

After some other problems previously discussed in this thread, I'm now stuck with the following issue:
I have a very simple VNET jail setup (config see below) and tried to manually assign it an epair interface with the following steps:

~ # ifconfig bridge create
bridge0
~ # ifconfig epair create
epair0a
~ # ifconfig bridge0 addm jailether
~ # ifconfig epair0a mtu 1450
~ # ifconfig bridge0 addm epair0a
~ # ifconfig
jailether: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1450
    options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
    ether 96:00:01:37:d7:61
    inet ###.###.###.### netmask 0xffffffff broadcast ###.###.###.###
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1450
    ether 58:9c:fc:10:38:5e
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000
    member: jailether flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
epair0a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1450
    options=8<VLAN_MTU>
    ether 02:0a:db:3f:dc:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:0a:db:3f:dc:0b
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
~ # ifconfig epair0b mtu 1450
~ # ifconfig epair0b vnet webserver

This is where I get stuck now. The console just hangs up, the epair0b interface disappears from the host but doesn't appear in the jail. I'm also not able to destroy epair0a (console hangs up again) and when stopping the jail via service jail stop webserver or even jail -R webserver it's stuck as dying.
This even keeps me from rebooting the host, so the only way to clean this up right now is to power off the host and reboot.

Here's my configuration:
/etc/rc.conf

## Basic network settings
hostname="xxx"
ifconfig_vtnet0_name="jailether"
ifconfig_jailether="inet a.b.c.d/32 mtu 1450 up"
defaultrouter="e.f.g.h"
static_routes="defgw"
route_defgw="-host e.f.g.h -iface jailether"

## Enable system services
pf_enable="NO"

## Enable user services
jail_enable="YES"
jail_list=""

/etc/jail.conf

exec.timeout=90;
stop.timeout=30;

host.hostname="$name";
path="/jail/jails/$name/root";
mount.fstab="/jail/jails/$name/fstab";
mount.devfs;
exec.clean;
exec.start="sh /etc/rc";
exec.stop="sh /etc/rc.shutdown";
exec.consolelog="/var/tmp/jail_$name.log";


webserver {
  vnet;
  devfs_ruleset=5;
}

 

[patmaddox]

What cloud provider are you using, and what image did you use?

I know when I added the main interface as a bridge member, the network connection stopped and I had to restart (on GCP - I think DigitalOcean did not have that problem). Though you clearly got past that point.

Have you tried the configuration I shared in the other thread? It works for me on GCP, DO, and hardware.

If it works, you can then begin to migrate it step-by-step to your desired config (e.g. adding jailether as a bridge member, running ifconfig vnet in host vs using jail.conf vnet.interface, etc) and see where the breakdown is.

 

[thecodechemist]

What cloud provider are you using, and what image did you use?

I’m using Hetzner and the “FreeBSD 13.0 (amd64/netinstall)
FreeBSD-13.0-RELEASE-amd64-bootonly.iso” ISO Image they provide.

Have you tried the configuration I shared in the other thread? It works for me on GCP, DO, and hardware.

Yes, I did. Unfortunately it didn’t work either, that’s why I went full manual to try every single step one by one, to isolate the issue.

If it works, you can then begin to migrate it step-by-step to your desired config (e.g. adding jailether as a bridge member, running ifconfig vnet in host vs using jail.conf vnet.interface, etc) and see where the breakdown is.

That’s what I’ve done, only that it didn’t work either .
I honestly have no Idea how I could simplify my current test setup even more. And still have no idea what might lead to this issue.
Actually, one idea, but not enough knowledge about it: May there be an issue with the changed MTU on the vnet side? Is there maybe a setting I have to adjust in the vnet config I haven’t found yet?
And is it actually necessary to adjust the MTU on both epair interfaces?

 

[BS:D]

This may not be the exact solution you need but it might help.

I also had issues with VNET, epairs, bridges and jails in the past so I wrote a few scripts to restart jails and stop jails.

Example setup with a public bridge for external traffic (bridge1), a private bridge for internal traffic between jails (bridge2) & 3 jails:

(I made sure that bridges were created on boot & not destroyed or recreated. Also made sure jails did not start on boot.) -

/etc/rc.conf:

#############
## Bridges ##
#############
cloned_interfaces="bridge1 bridge2"
# Bridge1 is for external traffic
# Bridge2 is for internal traffic 
ifconfig_bridge1="addm em0"
ifconfig_em0="up"

/etc/jail.conf:

path=/zpool/jails/$name;
host.hostname = $name.freebsd;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
vnet = "new";
mount.devfs;
persist;

asterisk {
vnet.interface = "epair1b";
vnet.interface += "epair101b";
}

nginx {
vnet.interface = "epair2b";
vnet.interface += "epair102b";
}

mysql {
vnet.interface = "epair3b";
vnet.interface += "epair103b";
}

Bash script to start/restart all jails:

#!/usr/local/bin/bash
####################
## Destroy epairs ##
####################
ifconfig epair1a destroy;
ifconfig epair2a destroy;
ifconfig epair3a destroy;
ifconfig epair101a destroy;
ifconfig epair102a destroy;
ifconfig epair103a destroy;
################
## Stop jails ##
################
service jail onestop;
########################
## Create jail epairs ##
########################
ifconfig epair1 create;
ifconfig epair2 create;
ifconfig epair3 create;
ifconfig epair101 create;
ifconfig epair102 create;
ifconfig epair103 create;
#################################
## Add jails to Public bridge1 ##
#################################
ifconfig bridge1 addm epair1a;
ifconfig bridge1 addm epair2a;
ifconfig bridge1 addm epair3a;
ifconfig epair1a up;
ifconfig epair2a up;
ifconfig epair3a up;
ifconfig bridge1 up;
##################################
## Add jails to Private bridge2 ##
##################################
ifconfig bridge2 addm epair101a;
ifconfig bridge2 addm epair102a;
ifconfig bridge2 addm epair103a;
ifconfig epair101a up;
ifconfig epair102a up;
ifconfig epair103a up;
ifconfig bridge2 up;
#################
## Start jails ##
#################
service jail onestart;

Bash script to stop all jails:

#!/usr/local/bin/bash
####################
## Destroy epairs ##
####################
ifconfig epair1a destroy;
ifconfig epair2a destroy;
ifconfig epair3a destroy;
ifconfig epair101a destroy;
ifconfig epair102a destroy;
ifconfig epair103a destroy;
################
## Stop jails ##
################
service jail onestop;

 

[joggx]

Who the heck deleted my post? I was following up with a problem of the solution but a new pops out my post 'hijack the unrelated topic'. Are these admin suckers trying to stop ppl from getting help? May this STUPID FORUM doomed all the way down the road.

 

[SirDice]

I did. You didn't "follow up", you posted a completely unrelated question in this thread, thereby hijacking it. Now, settle down, go take a walk or something and post your question in its own thread.