jails Passthrough SR-IOV VF Network Interface to Jail: Which jail managers can do the job?

C

cb000

I am currently experimenting with jails. My testing machine has a Mellanox ConnectX-4 NIC. Following the Nvidia procedure and forum posts, I upgraded the firmware and successfully enabled and created VFs. Now, I have several mceX interfaces (X=0, 1 are the PFs, and 2 – 3 are the VFs).
Code: 
root@freebsd0:~ # cat /etc/iovctl.conf
PF {
        device : "mlx5_core0";
        num_vfs : 2,
}

DEFAULT {
        passthrough : false;
}

VF-0 {
        mac-addr : "aa:88:44:00:02:01";
}

VF-1 {
        mac-addr : "aa:88:44:00:02:02";
}

Following the handbook, I created a native thick jail with the mce2 interface, and it worked as expected. The jail does not have an IP by default, and the VF is controlled by the jail, so it does not appear on the host anymore.

Code: 
root@freebsd0:~ # cat /etc/jail.conf
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

classic {

        # STARTUP/LOGGING
        exec.consolelog = "/var/log/jail_console_${name}.log";

        host.hostname = "${name}";
        path = "/usr/local/jails/containers/${name}";
        vnet;
        #vnet.interface = "mce2.160";
        vnet.interface = "mce2";
        devfs_ruleset="7";
        allow.raw_sockets;
}

root@freebsd0:~ # jexec -u root classic

root@classic:/ # cat /etc/rc.conf
ifconfig_mce2="mtu 9000 UP"
vlans_mce2="160"
ifconfig_mce2_160="SYNCDHCP"

root@classic:/ # ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mce2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
        options=7eef07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,NV,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWRXTSTMP,MEXTPG,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO,RXTLS4,RXTLS6>
        ether aa:88:44:00:02:01
        media: Ethernet 10GBase-CR1 <full-duplex,rxpause,txpause>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
mce2.160: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
        options=1c680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG,TXTLS4,TXTLS6>
        ether aa:88:44:00:02:01
        inet xxx.xxx.xxx.xxx netmask 0xffffff00 broadcast xxx.xxx.xxx.255
        groups: vlan
        vlan: 123 vlanproto: 802.1q vlanpcp: 0 parent interface: mce2
        media: Ethernet 10GBase-CR1 <full-duplex,rxpause,txpause>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

However, since I need to type multiple commands to create a jail, I am looking for a suitable jail manager that can do the same job using a formatted command or config file. Unfortunately, I haven’t found a way to do this with BastilleBSD or CBSD.

For BastilleBSD, I used the following command to create a jail. There is no option to passthrough the NIC to the jail, and I must set an IP or use DHCP to create the jail.
Code: 
root@freebsd0:~ # bastille create -T bjail 14.1-RELEASE DHCP mce3

For CBSD, I tried the jconstruct-tui method to create a jail. It allows me not to set an IP, but it seems that I must create a vnet epair bridge in Networking.

Any suggestions are welcome.
 
Today I tried Appjail. It can create a thin jail using appjail quick test vnet=mce3 start, but I don’t know how to create a thick jail with only the passthrough interface visible and the bpf unhidden for DHCP.
 
I added support for vnet interfaces to sysutils/iocell, but just like several other PRs they have been sitting for over half a year now.

I have already created a (local at my buildhosts) port 'iocell-devel' which uses my repo as upstream and has several other fixes/improvements/additions. I wanted to try getting it added to the ports tree - maybe under another name, since it basically is a fork by now - but haven't found the time yet... (iocell by design doesn't use the (modifyable) default values if an option is unset for a jail - I still want to fix this behavior before I present it as a 'new version'/fork/whatever...)

One just adds the interface(s) to the vnet_interfaces jail property via iocell set vnet_interfaces="mce2.8,mce3.5" jailname and those are simply handed over as 'vnet.interface' parameter to the jail command.
 
cb000 said:
For CBSD, I tried the jconstruct-tui method to create a jail. It allows me not to set an IP, but it seems that I must create a vnet epair bridge in Networking.
Click to expand...
If you want to assign an interface, then in CBSD it looks like this:
Code: 
ifconfig igb0.160 create
ifconfig igb0.160 up
cbsd jcreate jname=test vnet=1 interface=igb0.160 allow_raw_sockets=1 ip4_addr=REALDHCP devfs_ruleset=5

or via CBSDfile:
Code: 
jail_test()
{
   vnet=1
   interface="igb0.160"
   allow_raw_sockets=1
   ip4_addr=REALDHCP
   devfs_ruleset=5
}
then: `cbsd up`.

As for TUI, you need to go in 'jailnic` options from `cbsd jconfig` -> 'nic1' -> 'nic_parent'
 
Ole said:
If you want to assign an interface, then in CBSD it looks like this:
Code: 
ifconfig igb0.160 create
ifconfig igb0.160 up
cbsd jcreate jname=test vnet=1 interface=igb0.160 allow_raw_sockets=1 ip4_addr=REALDHCP devfs_ruleset=5
Click to expand...
Thank you for your reply, Ole.

I call cbsd jcreate jname=test vnet=1 interface=mce3 allow_raw_sockets=1 devfs_ruleset=7
but it is not what I want. I see it creates an epair bridge. In the jail, I see eth0 instead of mce3. What I want is to avoid the usage of epair, and just passthrough the VF interface to the jail to reduce the CPU resource on the virtual network.
 
cb000 said:
Today I tried Appjail. It can create a thin jail using appjail quick test vnet=mce3 start, but I don’t know how to create a thick jail with only the passthrough interface visible and the bpf unhidden for DHCP.
Click to expand...
* DHCP: https://appjail.readthedocs.io/en/latest/networking/DHCP-and-SLAAC/ [1]
* For a thickjail, simply set the option `type=thick`.

If the feature you mention will create an interface on your host, simply pass the `vnet=interface` option like any other interface [2].

[1] The documentation refers to the use of devfs.rules(5), but appjail(1) can dynamically manage your devices: https://appjail.readthedocs.io/en/latest/DEVFS/
[2] Of course, remember that using VNET will make your interface disappear from your host.
 
I do not use any manager, just been using variables in /etc/jail.conf:

/etc/jail.conf
Code: 
# interface name on the host ( the vf interfaces are named iavfX in my case)
$if       = "iavf$ifnum";
$jail_if = "$if";

exec.prestart      = "ifconfig $if up mtu $mtu -tso4 -tso6 -lro -vlanhwtso";

exec.start = "dhclient $jail_if";
exec.start += "/bin/sh /etc/rc";

# configuration of the VMs (setting interface num)
jenkins {
  $ifnum                 = "1";
}

git {
$ifnum = "2";
}

etc...

Reading this years later, I think that $if and $jail_if are redundant and test residues.
To summarize, $ifnum is set at jail level and used at global level.

I hope this was clear.
 
You must log in or register to reply here.
Back
Top