mlx5en attempt to use SR-IOV results in kernel panic

While trying to setup SR-IOV VFs on a Mellanox ConnectX-4 100G NIC I'm getting a kernel panic. I've run the appropriate commands to enable VFs and the desired quantity via the mellanox firmware tool, as documented in their user guide. Then I setup the /etc/iov/mce1.conf file to contain the following:

Code:
PF {
    device: "mlx5_core1";
    num_vfs: 8;
}

DEFAULT {
    passthrough: false;
}

I've also tried the NIC's interface name as such, but the same kernel panic arises.
Code:
PF {
    device: "mce1";
    num_vfs: 8;
}

DEFAULT {
    passthrough: false;
}

Here's the kernel panic output.
Code:
mlx5_core2: <mlx5_core> at device 0.10 numa-domain 2 on pci19                                                                                                                                                                                                 
panic: Assertion dev != NULL failed at /usr/src/sys/compat/linuxkpi/common/src/linux_pci.c:492                                                                                                                                                                                                                                                                                                                                                              
version = FreeBSD 13.0-CURRENT #0 : Sat Dec 26 23:08:09 UTC 2020                                                                                                                                                                                         

KDB: stack backtrace:                                                                                                                                                                                                                                         
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0357e014f0                                                                                                                                                                               
vpanic() at vpanic+0x186/frame 0xfffffe0357e01540                                                                                                                                                                                                             
panic() at panic+0x43/frame 0xfffffe0357e015a0                                                                                                                                                                                                               
pci_resource_start() at pci_resource_start+0x137/frame 0xfffffe0357e015e0                                                                                                                                                                                     
init_one() at init_one+0x1148/frame 0xfffffe0357e01650                                                                                                                                                                                                       
linux_pci_attach_device() at linux_pci_attach_device+0x5a2/frame 0xfffffe0357e016b0                                                                                                                                                                           
device_attach() at device_attach+0x3d2/frame 0xfffffe0357e016f0                                                                                                                                                                                               
device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xfffffe0357e01720                                                                                                                                                                           
bus_generic_attach() at bus_generic_attach+0x18/frame 0xfffffe0357e01740                                                                                                                                                                                     
pci_iov_ioctl() at pci_iov_ioctl+0x13f4/frame 0xfffffe0357e01800                                                                                                                                                                                             
devfs_ioctl() at devfs_ioctl+0xbb/frame 0xfffffe0357e01850                                                                                                                                                                                                   
vn_ioctl() at vn_ioctl+0x131/frame 0xfffffe0357e01960                                                                                                                                                                                                         
devfs_ioctl_f() at devfs_ioctl_f+0x24/frame 0xfffffe0357e01980                                                                                                                                                                                               
kern_ioctl() at kern_ioctl+0x28b/frame 0xfffffe0357e019f0                                                                                                                                                                                                     
sys_ioctl() at sys_ioctl+0x12a/frame 0xfffffe0357e01ac0                                                                                                                                                                                                       
amd64_syscall() at amd64_syscall+0x13a/frame 0xfffffe0357e01bf0                                                                                                                                                                                               
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0357e01bf0                                                                                                                                                                                   
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x3faff90095a, rsp = 0x6b590ffc4f38, rbp = 0x6b590ffc5020 ---                                                                                                                                               
KDB: enter: panic                                                                                                                                                                                                                                             
[ thread pid 43416 tid 101371 ]                                                                                                                                                                                                                               
Stopped at      kdb_enter+0x37: movq    $0,0x1288546(%rip)                                                                     
db> reboot

My /boot/loader.conf file has mlx5en_load="YES"' set and the NIC works fine as long as I'm not trying to enable VFs. I've verified that the NIC is running the latest firmware and that VFs work in Linux, so hopefully there's just some syntax that I'm missing or setting incorrectly.

The /etc/rc.conf file has this entry to load the VFs.
Code:
iovctl_files="/etc/iov/mce1.conf"

Any ideas about how to resolve this would be much appreciated.
 
Last edited by a moderator:
Hello. I had the exact same problem. I just solved it by using mlx5_core0 instead of mlx5_core1.
I do not know if this is a bug (I think it should at least return an error instead of crashing) or if we are supposed to use the first port only for this.
Note that I am running FreeBSD 12.2, not 13.
 
While trying to setup SR-IOV VFs on a Mellanox ConnectX-4 100G NIC I'm getting a kernel panic. I've run the appropriate commands to enable VFs and the desired quantity via the mellanox firmware tool, as documented in their user guide.
what did you do ? can it be done directly in freebsd?
 
Back
Top