Howdy! I'm having a heck of a time getting a pair of ConnectX-2 adapters working on my system. I have tried loadable modules as well as a custom kernel with the modules/options built-in with the same result. The system is generally very well behaved except for this problem. I am simply out of ideas at this point in time (please excuse my first post).
When loading the kernel modules, it appears they load and the cards are recognized, but I get this "No IRQs left, device can't be started." madness. This is using the Mellanox 2.1.6 drivers which come with 11.0-RELEASE and GENERIC kernel.
I have included an example of what happens with the module load, the lscpi, vmstat -ai showing interrupts, and attached kldstat showing all compiled-in and loaded modules and dmesg. I am happy to post more, but I don't know what else would be helpful.
Any help or suggestions would be appreciated.
Thanks,
Ted
When loading the kernel modules, it appears they load and the cards are recognized, but I get this "No IRQs left, device can't be started." madness. This is using the Mellanox 2.1.6 drivers which come with 11.0-RELEASE and GENERIC kernel.
I have included an example of what happens with the module load, the lscpi, vmstat -ai showing interrupts, and attached kldstat showing all compiled-in and loaded modules and dmesg. I am happy to post more, but I don't know what else would be helpful.
Code:
root@monstro:/ # kldload mlxen
root@monstro:/ # dmesg | tail
[IMG]https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif[/IMG]
mlx4_core0: <mlx4_core> mem 0xfb700000-0xfb7fffff,0xf3800000-0xf3ffffff irq 40 at device 0.0 on pci6
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Feb 3 2017)
mlx4_core0: No IRQs left, device can't be started.
device_attach: mlx4_core0 attach returned 28
mlx4_core0: <mlx4_core> mem 0xfb300000-0xfb3fffff,0xf3000000-0xf37fffff irq 16 at device 0.0 on pci10
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Feb 3 2017)
mlx4_core0: No IRQs left, device can't be started.
device_attach: mlx4_core0 attach returned 28
Code:
root@monstro:/ # lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
00:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04)
00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
00:1c.6 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 7 (rev b6)
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b6)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
01:00.0 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (A-Segment Bridge) (rev 09)
01:00.2 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (B-Segment Bridge) (rev 09)
02:04.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5706 Gigabit Ethernet (rev 02)
03:05.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5706 Gigabit Ethernet (rev 02)
05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
08:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
0a:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
0b:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0c:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0d:04.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
Code:
root@monstro:/ # vmstat -ia
interrupt total rate
??? 0 0
irq1: 0 0
stray irq1 0 0
irq0: attimer0 0 0
stray irq0 0 0
irq3: 0 0
stray irq3 0 0
irq4: 0 0
stray irq4 0 0
irq5: 0 0
stray irq5 0 0
irq6: 0 0
stray irq6 0 0
irq7: 0 0
stray irq7 0 0
irq8: atrtc0 0 0
stray irq8 0 0
irq9: acpi0 2 0
stray irq9 0 0
irq10: uart2 0 0
stray irq10 0 0
irq11: 0 0
stray irq11 0 0
irq12: 0 0
stray irq12 0 0
irq13: 0 0
stray irq13 0 0
irq14: 0 0
stray irq14 0 0
irq15: 0 0
stray irq15 0 0
irq16: ehci0 9526835 97
stray irq16 0 0
irq17: 0 0
stray irq17 0 0
irq18: 0 0
stray irq18 0 0
irq19: 0 0
stray irq19 0 0
irq20: 0 0
stray irq20 0 0
irq21: 0 0
stray irq21 0 0
irq22: 0 0
stray irq22 0 0
irq23: ehci1 201906 2
stray irq23 0 0
irq24: 0 0
stray irq24 0 0
irq25: 0 0
stray irq25 0 0
irq26: bce0 9032712 92
stray irq26 0 0
irq27: 0 0
stray irq27 0 0
irq28: bce1 10612876 108
stray irq28 0 0
irq29: 0 0
stray irq29 0 0
irq30: 0 0
stray irq30 0 0
irq31: 0 0
stray irq31 0 0
irq32: 0 0
stray irq32 0 0
irq33: 0 0
stray irq33 0 0
irq34: 0 0
stray irq34 0 0
irq35: 0 0
stray irq35 0 0
irq36: 0 0
stray irq36 0 0
irq37: 0 0
stray irq37 0 0
irq38: 0 0
stray irq38 0 0
irq39: 0 0
stray irq39 0 0
irq40: 0 0
stray irq40 0 0
irq41: 0 0
stray irq41 0 0
irq42: 0 0
stray irq42 0 0
irq43: 0 0
stray irq43 0 0
irq44: 0 0
stray irq44 0 0
irq45: 0 0
stray irq45 0 0
irq46: 0 0
stray irq46 0 0
irq47: 0 0
stray irq47 0 0
cpu0:timer 33848633 343
irq256: hpet0:t0 0 0
stray irq256 0 0
irq257: hpet0:t1 0 0
stray irq257 0 0
irq258: hpet0:t2 0 0
stray irq258 0 0
irq259: hpet0:t3 0 0
stray irq259 0 0
irq260: hpet0:t4 0 0
stray irq260 0 0
irq261: hpet0:t5 0 0
stray irq261 0 0
irq262: hpet0:t6 0 0
stray irq262 0 0
irq263: hpet0:t7 0 0
stray irq263 0 0
irq264: xhci0 0 0
stray irq264 0 0
irq265: mpr0 8147997 83
stray irq265 0 0
irq266: em0 0 0
stray irq266 0 0
irq267: em1 0 0
stray irq267 0 0
irq268: ahci0 27 0
stray irq268 0 0
cpu1:timer 11680909 118
cpu4:timer 12066595 122
cpu7:timer 11196384 113
cpu6:timer 13054931 132
cpu5:timer 11188758 113
cpu2:timer 11319839 115
cpu11:timer 11077046 112
cpu9:timer 11335830 115
cpu8:timer 11665891 118
cpu3:timer 11244335 114
cpu10:timer 13033996 132
Total 200235502 2030Any help or suggestions would be appreciated.
Thanks,
Ted