VNET netgraph jail(s) fail to jng bridge to LAGG interface

[DemoNIck]

On a FreeBSD 12.2 server I am running several jails.
The FreeBSD server has two (2) physical eth interfaces (igb0, igb1).
These 2 eth interfaces (igb0, igb1) "combine" a LAGG interface on that server.

/etc/rc.conf

ifconfig_igb0="up"
ifconfig_igb1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 SYNCDHCP"

So far, I have been using IP aliases for the jail's networking without any problems.

/etc/jail.conf

$parentdir = "/var/jail";
path = "$parentdir/$name";
$eth = "lagg0";
ip4.addr = "${eth}|${ipv4}";
exec.clean;
exec.start     = "/bin/sh /etc/rc";
exec.stop      = "/bin/sh /etc/rc.shutdown";
exec.system_user = "root";
exec.jail_user = "root";


jail1 {
       host.hostname = ${name}.local.lan;
       ipv4 = "10.0.0.10/24";  
}

Yet, I am trying to move to VNET networking. So I changed my /etc/jail.conf to

$parentdir = "/var/jail";
path = "$parentdir/$name";
$eth = "lagg0";
exec.clean;
exec.start     = "/bin/sh /etc/rc";
exec.stop      = "/bin/sh /etc/rc.shutdown";
exec.system_user = "root";
exec.jail_user = "root";

vnet;
vnet.interface = "ng0_${name}";                 # vnet interface(s)

exec.start += "ifconfig ng0_${name} ${ipv4}";
exec.start += "route add default ${gw4}";

exec.prestart += "jng bridge ${name} ${eth}";   # bridge interface(s)
exec.poststop += "jng shutdown ${name}";        # destroy interface(s)

jail1 {
       host.hostname = ${name}.local.lan;
       ipv4 = "10.0.0.10/24";
       $gw4 = "10.0.0.1";
}

Now when I try to start the jail(s), I get the following error messages. No jail(s) are created

[root@server:~]# service jail start
Starting jails:ngctl: send msg: No such file or directory
jail: jail1: jng bridge jail1 lagg0: failed

Any help would be much appreciated.

Thank you in advance for your time.

 

[PMc]

Question is, what does jng do?

I never used lagg. What I have is a netgraph bridge that joins all my physifs, so that the machine actually works as a bridge/switch. Then I attach my vnet jails onto that bridge.

Looks like this:

This one goes into rc.d and creates the bridge (should run before jails come up):

netgraphs_lan_ifs="fxp1 fxp3 re0"
netgraphs_lan_if1_name="nge_lan_1u"
netgraphs_lan_if1_mac="03:1a:91:01:02:02"
netgraphs_lan_if1_addr="*************/**"

netgraphs_lan_start()
{
    if ngctl info lanswitch: > /dev/null 2>&1; then
        netgraphs_lan_stop
    fi
   
    echo "Creating LAN Switch"
   
    # link0 of the bridge goes to the base system (do not kill that
    # when recreating the bridge, as it carries a required addr)
   
    if ngctl info ${netgraphs_lan_if1_name}: > /dev/null 2>&1; then
        ngctl -f - <<EOF
            mkpeer bridge crhook link16
            name .:crhook lanswitch
            connect lanswitch: ${netgraphs_lan_if1_name}: link0 ether
EOF
    else
        ngctl -f - <<EOF
            mkpeer bridge crhook link16
            name .:crhook lanswitch
            mkpeer lanswitch: eiface link0 ether
            name lanswitch:link0 $netgraphs_lan_if1_name
EOF
    fi

    # put a name, mac and addr onto lan_if1
   
    _ifname=`ngctl msg ${netgraphs_lan_if1_name}: getifname | \
                awk '$1 == "Args:" { print substr($2, 2, length($2)-2)}'`
    ifconfig $_ifname name $netgraphs_lan_if1_name
    ifconfig $netgraphs_lan_if1_name link $netgraphs_lan_if1_mac
    ifconfig $netgraphs_lan_if1_name inet $netgraphs_lan_if1_addr

    # now attach all the physicals to the bridge
   
    link=1
    for if in $netgraphs_lan_ifs; do
        ngctl connect lanswitch: ${if}: link${link} lower
        ngctl msg ${if}: setpromisc 1
        ngctl msg ${if}: setautosrc 0
        link=`expr ${link} + 1`
    done
}
netgraphs_lan_stop()
{
    echo "Shutting down LAN switch"
    ngctl shutdown lanswitch:
    # lan_if1 not removed!
}

And this is how the respective vnet jails get configured in jail.conf:

admn {
        jid = 8;
        devfs_ruleset = 11;
        host.hostname = "admn.my.domain";
        vnet = "new";
        $ifname1l = nge_${name}_1l;
        $ifname1l_mac = 03:1a:91:01:01:07;
        vnet.interface = "$ifname1l";
        exec.prestart = "
            echo -e \"mkpeer eiface crhook ether\nname .:crhook $ifname1l\" \
                | /usr/sbin/ngctl -f -
            /usr/sbin/ngctl connect ${ifname1l}: lanswitch: ether link17
            ifname=`/usr/sbin/ngctl msg ${ifname1l}: getifname | \
                awk '$1 == \"Args:\" { print substr($2, 2, length($2)-2)}'`
            /sbin/ifconfig \$ifname name $ifname1l
            /sbin/ifconfig $ifname1l link $ifname1l_mac
        ";
        exec.poststop = "
            sleep 6 ;
            /usr/sbin/ngctl shutdown ${ifname1l}:
        ";
}

And finally this is in rc.conf inside the jail:

### Network routing options: ###
network_interfaces="lo0 nge_admn_1l"
ifconfig_lo0="inet 127.0.0.1"
ifconfig_nge_admn_1l="inet ************* netmask **********"
defaultrouter="*************"   # This is the link0 addr from above!

 

[DemoNIck]

Question is, what does jng do?

I think what I am trying to do, is bridge the jail's vnet.interface ng0_${name} to my host's LAGG ${eth} interface. Am I not? ;-p

The same configuration, on another FreeBSD server without a LAGG interface but with a physical interface re0 worked.

I think, that on the host, I might have to create another one vnet.interface (i.e. ng_host), then bridge it to the host's LAGG interface (lagg0) and then bring the lagg0 up. Then on the jail I will have to bridge the jail's vnet.interface to the ng_host and not the lagg0 as I am doing now. I will have to test.

On the other hand I might be completely wrong and I should RTFM again.

Whatever the case is, thank you very much for your answer.

PS

ifconfig_lo0="inet 127.0.0.1"

Why do you configure the jail's loopback interface ?

 

[PMc]

I think what I am trying to do, is bridge the jail's vnet.interface ng0_${name} to my host's LAGG ${eth} interface. Am I not? ;-p

It appeared to me, and it should be possible that way. But, creating the bridge in one step, and then creating the jail in a second step would separate the concerns and probably make it easier to debug.

The same configuration, on another FreeBSD server without a LAGG interface but with a physical interface re0 worked.

Ah. Then, figuring out in which way creating a plain bridge (without a jail) with a lagg is different to creating a bridge with a physical if, would appear to be the next logical step.

From Your wording I get the impression that you think of "bridging one thing to another". From my understanding this is not how bridges work - I understand a bridge as a separate entity (usually hardware, but in this case software) that just has a couple of network plugs and connects them together. A bridge can exist with only one plug connected (it is useless then, but nevertheless works). This is how ngctl shows the bridge:

  Name: lanswitch       Type: bridge          ID: 0000000a   Num hooks: 5
  Local hook      Peer name       Peer type    Peer ID         Peer hook     
  ----------      ---------       ---------    -------         ---------     
  link16          nge_admn_1l     eiface       0000002b        ether         
  link3           re0             ether        00000001        lower         
  link2           fxp3            ether        00000005        lower         
  link1           fxp1            ether        00000003        lower         
  link0           nge_lan_1u      eiface       0000000b        ether

PS

Why do you configure the jail's loopback interface ?

Just copied that from my other machines; I always did it that way. My site is running for a couple of decades already, so there may be lots of things that have simplified over time.

 

[Deviant0ne]

I have an almost identitical setup, the only difference is I am using a static IP on my LAGG.

In my configuration, I called jng from inside the jail stanza.

tm1 {
    host.hostname  = "TM";
    vnet;
    vnet.interface = "ng0_tm0";
    exec.prestart += "jng bridge tm0 lagg0";
    exec.start    += "ifconfig ng0_tm0 10.0.1.11/24";
    exec.start    += "route add default 10.0.1.1";
    exec.prestop  += "ifconfig ng0_tm0 -vnet $name";
    exec.poststop += "jng shutdown tm0";
    mount.fstab    = "/Jails/fstab/tm1.fstab";
}

Also, don’t forget to add ng_ether_load="YES" to /boot/loader.conf