Solved Weird Network Bug Jail 14.0-Release compare 13.2-Release

GoNeFast_01

GoNeFast_01

I created a jail with a configuration structure that I developed for all my internal freebsd jails, I decided to install 14.0-Release without CBSD/Churchers Bhyve, essentially good old FreeBSD Jails.
The jail 14.0-Release does not create an ipv4; only the ipv6 option is in the ifconfig... I created a jail with 13.2-Release with the same config and everything works like it should.


Jail.14-Release ifconfig inside jail looks something like this for example:

Code: 
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        lladdr
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0 metric 0 mtu 33160
        lladdr
        groups: pflog
epair4b: flags=1008863<UP,BROADCAST,NEEDSEPOCH,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        ether 02:a9:28:1c:e9:0b
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Jail.13.2-Release ifconfig inside jail looks something like this for example:


Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.4 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
epair3b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:4e:d5:0b:ef:0b
        inet 10.0.175.2 netmask 0xffffff00 broadcast 10.0.175.255
        inet 192.168.2.250 netmask 0xffffff00 broadcast 192.168.2.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Anyone facing similar problem? Am I missing something that needs to be done in 14.0-Release that cannot be done in 13.2.... Host OS is a 13.2p3.
Is this an ongoing bug with 14.0 ?

Anyone that has some debug suggestion welcome.
 
First, are you saying that you have a physical machine running 13.2p3, that is hosting two jails, one running 13.2 and the other running 14.0? If so, that might be your problem: you've got a FreeBSD 14.0 userland installed in a jail with a FreeBSD 13.2 kernel. According to the FreeBSD Handbook at Section 17.3, a jail can't be on a newer release than its host.

Second, I got tripped up on something like this recently, only to find I had a typo in one of the stanzas in /etc/jail.conf. While I appreciate that you're using the same config for both jails, can you post the contents of that file for both jails?
 
samjenk appreciate the response....
samjenk said:
According to the FreeBSD Handbook at Section 17.3, a jail can't be on a newer release than its host.
Click to expand...

samjenk said:
First, are you saying that you have a physical machine running 13.2p3, that is hosting two jails, one running 13.2 and the other running 14.0? If so, that might be your problem: you've got a FreeBSD 14.0 userland installed in a jail with a FreeBSD 13.2 kernel.
Click to expand...
Yes Correct....

If this is the case based on the tip shown in Handbook 17.3 than I will assume this is the issue... So in other words, if I want to test FBSD 14.0 RELEASE and/or 15.0 Latest STABLE kernel/userland for certain packages it will be best to do a bhyve installation instead of a jail ✔️


samjenk said:
Second, I got tripped up on something like this recently, only to find I had a typo in one of the stanzas in /etc/jail.conf. While I appreciate that you're using the same config for both jails, can you post the contents of that file for both jails?
Click to expand...

Below is the 13.2
Code: 
####################################################################################
#### GLOBAL JAIL SETTINGS                                                         ##
####################################################################################
####
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  exec.consolelog = "/var/log/jail_${name}_console.log";
#
  exec.timeout=90;
  stop.timeout=30;
#
  host.hostname = ${name}.[domain].com;
  $j="/usr/jails/jails-data";
  path="$j/${name}-data";
#
####
####################################################################################
#### JAILS Gaming_Steam                                                           ##
####################################################################################
####
Gaming_Steam2 {
    allow.raw_sockets;
    allow.mount;
#
    allow.mount.devfs;
    mount.devfs;
    devfs_ruleset=4;
#
    allow.sysvipc;
#
    enforce_statfs=1;
    children.max=5;
    vnet;
    vnet.interface="epair3b";
#
    exec.prestart="echo Start jail $name has started.";
    exec.prestart+="ifconfig epair3 create";
    exec.prestart+="ifconfig bridge1 addm epair3a";
    exec.prestart+="ifconfig epair3a inet 10.0.175.1/24 up";
#
   exec.start+="ifconfig lo0 127.0.0.4 up";
   exec.start+="ifconfig epair3b inet 10.0.175.2/24 up";
   exec.start+="ifconfig epair3b 192.168.2.250/24 alias";
   exec.start+="route add default 192.168.2.254";
#
   exec.poststart="echo Start jail $name is complete.";
#
   exec.prestop="echo Stop jail $name has started.";
#
   exec.poststop="Stop jail $name has Finished...";
   exec.poststop+="ifconfig epair3a destroy";
}

Below is 14.0
Code: 
####################################################################################
#### GLOBAL JAIL SETTINGS                                                         ##
####################################################################################
####
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  exec.consolelog = "/var/log/jail_${name}_console.log";
#
  exec.timeout=90;
  stop.timeout=30;
#
  host.hostname = ${name}.[domain].com;
  $j="/usr/jails/jails-data";
  path="$j/${name}-data";
#
####
####################################################################################
#### JAILS Gaming_Steam                                                           ##
####################################################################################
####
#
Gaming_Steam {
    allow.raw_sockets;
    allow.mount;
#
    allow.mount.devfs;
    mount.devfs;
    devfs_ruleset=4;
#
    allow.sysvipc;
#
    enforce_statfs=1;
    children.max=5;
    vnet;
    vnet.interface="epair4b";
#
    exec.prestart="echo Start jail $name has started.";
    exec.prestart+="ifconfig epair4 create";
    exec.prestart+="ifconfig bridge1 addm epair4a";
    exec.prestart+="ifconfig epair4a inet 10.0.176.1/24 up";
#
   exec.start+="ifconfig lo0 127.0.0.5 up";
   exec.start+="ifconfig epair4b inet 10.0.176.2/24 up";
   exec.start+="ifconfig epair4b 192.168.2.251/24 alias";
#
   exec.poststart="echo Start jail $name is complete.";
#
   exec.prestop="echo Stop jail $name has started.";
#
   exec.poststop="Stop jail $name has Finished...";
   exec.poststop+="ifconfig epair4a destroy";
}
####
 
Yeah, everything I've read says that you can't run a newer version of FreeBSD inside a jail. I'd imagine it's because a newer userland might be expecting a different interface than the one being provided by its kernel.

I copied out the two configurations you posted into two files ("file1" for the 13.2 jail, "file2" for the 14.0 jail) and noticed the 13.2 jail has an extra exec.start stanza declaring a default route:

Code: 
$ diff file1 file2
22c22,23
< Gaming_Steam2 {
---
> #
> Gaming_Steam {
35c36
<     vnet.interface=epair3b;
---
>     vnet.interface=epair4b;
38,45c39,45
<     exec.prestart+=ifconfig epair3 create;
<     exec.prestart+=ifconfig bridge1 addm epair3a;
<     exec.prestart+=ifconfig epair3a inet 10.0.175.1/24 up;
< #
<    exec.start+=ifconfig lo0 127.0.0.4 up;
<    exec.start+=ifconfig epair3b inet 10.0.175.2/24 up;
<    exec.start+=ifconfig epair3b 192.168.2.250/24 alias;
<    exec.start+=route add default 192.168.2.254;
---
>     exec.prestart+=ifconfig epair4 create;
>     exec.prestart+=ifconfig bridge1 addm epair4a;
>     exec.prestart+=ifconfig epair4a inet 10.0.176.1/24 up;
> #
>    exec.start+=ifconfig lo0 127.0.0.5 up;
>    exec.start+=ifconfig epair4b inet 10.0.176.2/24 up;
>    exec.start+=ifconfig epair4b 192.168.2.251/24 alias;
52c52
<    exec.poststop+=ifconfig epair3a destroy;
---
>    exec.poststop+=ifconfig epair4a destroy;
53a54
> ####

Out of curiosity, if you add an similar stanza to the 14.0 jail's definition, does it solve your problem? Whether it's a good idea is another question!
 
samjenk said:
Out of curiosity, if you add an similar stanza to the 14.0 jail's definition, does it solve your problem? Whether it's a good idea is another question!
Click to expand...
No difference at all.... I comment that out because the default route stanza if I leave it in the jail 14.0 doesn't execute causes failed error...
Code: 
Start jail Gaming_Steam has started.
epair4a
30
jail: Gaming_Steam: route add default 192.168.2.254: failed

This is routing in jail14 without the route stanza and doesn't give a failed error:

Code: 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1/32       link#1             US          lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              link#1             URS         lo0
localhost          link#1             UHS         lo0
::ffff:0.0.0.0/96  link#1             URS         lo0
fe80::%lo0/10      link#1             URS         lo0
fe80::%lo0/64      link#1             U           lo0
fe80::1%lo0        link#1             UHS         lo0
ff02::/16          link#1             URS         lo0

I will close this as solve and assume that the issue is what you suggested.
 
You must log in or register to reply here.
Back
Top