"No IRQs left" Mellanox MT26448 (ConnectX-2 10Gb enet) on Supermicro X9SRL (FreeBSD 11.0-RELEASE)

Howdy! I'm having a heck of a time getting a pair of ConnectX-2 adapters working on my system. I have tried loadable modules as well as a custom kernel with the modules/options built-in with the same result. The system is generally very well behaved except for this problem. I am simply out of ideas at this point in time (please excuse my first post).

When loading the kernel modules, it appears they load and the cards are recognized, but I get this "No IRQs left, device can't be started." madness. This is using the Mellanox 2.1.6 drivers which come with 11.0-RELEASE and GENERIC kernel.

I have included an example of what happens with the module load, the lscpi, vmstat -ai showing interrupts, and attached kldstat showing all compiled-in and loaded modules and dmesg. I am happy to post more, but I don't know what else would be helpful.

Code:
root@monstro:/ # kldload mlxen
root@monstro:/ # dmesg | tail
[IMG]https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif[/IMG]

mlx4_core0: <mlx4_core> mem 0xfb700000-0xfb7fffff,0xf3800000-0xf3ffffff irq 40 at device 0.0 on pci6
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Feb  3 2017)

mlx4_core0: No IRQs left, device can't be started.
device_attach: mlx4_core0 attach returned 28
mlx4_core0: <mlx4_core> mem 0xfb300000-0xfb3fffff,0xf3000000-0xf37fffff irq 16 at device 0.0 on pci10
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Feb  3 2017)

mlx4_core0: No IRQs left, device can't be started.
device_attach: mlx4_core0 attach returned 28


Code:
root@monstro:/ # lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
00:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04)
00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
00:1c.6 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 7 (rev b6)
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b6)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
01:00.0 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (A-Segment Bridge) (rev 09)
01:00.2 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (B-Segment Bridge) (rev 09)
02:04.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5706 Gigabit Ethernet (rev 02)
03:05.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5706 Gigabit Ethernet (rev 02)
05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
08:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
0a:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
0b:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0c:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0d:04.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)


Code:
root@monstro:/ # vmstat -ia
interrupt                          total       rate
???                                    0          0
irq1:                                  0          0
stray irq1                             0          0
irq0: attimer0                         0          0
stray irq0                             0          0
irq3:                                  0          0
stray irq3                             0          0
irq4:                                  0          0
stray irq4                             0          0
irq5:                                  0          0
stray irq5                             0          0
irq6:                                  0          0
stray irq6                             0          0
irq7:                                  0          0
stray irq7                             0          0
irq8: atrtc0                           0          0
stray irq8                             0          0
irq9: acpi0                            2          0
stray irq9                             0          0
irq10: uart2                           0          0
stray irq10                            0          0
irq11:                                 0          0
stray irq11                            0          0
irq12:                                 0          0
stray irq12                            0          0
irq13:                                 0          0
stray irq13                            0          0
irq14:                                 0          0
stray irq14                            0          0
irq15:                                 0          0
stray irq15                            0          0
irq16: ehci0                     9526835         97
stray irq16                            0          0
irq17:                                 0          0
stray irq17                            0          0
irq18:                                 0          0
stray irq18                            0          0
irq19:                                 0          0
stray irq19                            0          0
irq20:                                 0          0
stray irq20                            0          0
irq21:                                 0          0
stray irq21                            0          0
irq22:                                 0          0
stray irq22                            0          0
irq23: ehci1                      201906          2
stray irq23                            0          0
irq24:                                 0          0
stray irq24                            0          0
irq25:                                 0          0
stray irq25                            0          0
irq26: bce0                      9032712         92
stray irq26                            0          0
irq27:                                 0          0
stray irq27                            0          0
irq28: bce1                     10612876        108
stray irq28                            0          0
irq29:                                 0          0
stray irq29                            0          0
irq30:                                 0          0
stray irq30                            0          0
irq31:                                 0          0
stray irq31                            0          0
irq32:                                 0          0
stray irq32                            0          0
irq33:                                 0          0
stray irq33                            0          0
irq34:                                 0          0
stray irq34                            0          0
irq35:                                 0          0
stray irq35                            0          0
irq36:                                 0          0
stray irq36                            0          0
irq37:                                 0          0
stray irq37                            0          0
irq38:                                 0          0
stray irq38                            0          0
irq39:                                 0          0
stray irq39                            0          0
irq40:                                 0          0
stray irq40                            0          0
irq41:                                 0          0
stray irq41                            0          0
irq42:                                 0          0
stray irq42                            0          0
irq43:                                 0          0
stray irq43                            0          0
irq44:                                 0          0
stray irq44                            0          0
irq45:                                 0          0
stray irq45                            0          0
irq46:                                 0          0
stray irq46                            0          0
irq47:                                 0          0
stray irq47                            0          0
cpu0:timer                      33848633        343
irq256: hpet0:t0                       0          0
stray irq256                           0          0
irq257: hpet0:t1                       0          0
stray irq257                           0          0
irq258: hpet0:t2                       0          0
stray irq258                           0          0
irq259: hpet0:t3                       0          0
stray irq259                           0          0
irq260: hpet0:t4                       0          0
stray irq260                           0          0
irq261: hpet0:t5                       0          0
stray irq261                           0          0
irq262: hpet0:t6                       0          0
stray irq262                           0          0
irq263: hpet0:t7                       0          0
stray irq263                           0          0
irq264: xhci0                          0          0
stray irq264                           0          0
irq265: mpr0                     8147997         83
stray irq265                           0          0
irq266: em0                            0          0
stray irq266                           0          0
irq267: em1                            0          0
stray irq267                           0          0
irq268: ahci0                         27          0
stray irq268                           0          0
cpu1:timer                      11680909        118
cpu4:timer                      12066595        122
cpu7:timer                      11196384        113
cpu6:timer                      13054931        132
cpu5:timer                      11188758        113
cpu2:timer                      11319839        115
cpu11:timer                     11077046        112
cpu9:timer                      11335830        115
cpu8:timer                      11665891        118
cpu3:timer                      11244335        114
cpu10:timer                     13033996        132
Total                          200235502       2030

Any help or suggestions would be appreciated.

Thanks,
Ted
 

Attachments

  • dmesg.txt
    35.2 KB · Views: 380
  • kldstat.txt
    11.4 KB · Views: 347
Tried to compile the 2.1 and 2.1.5 drivers with FBSD 11 kernel source and there appear to be a number of conflicts with kernel headers, etc. so this doesn't seem incredibly viable. Friend has same card working in 10.3 with 2.1 drivers.
 
Moved from (old) Mellanox to (old) Chelsio and far fewer problems. There seems to be better support for old Chelsio cards.
 
Back
Top