XL 710 and FreeBSD 13/12

C

calebk

Hi everyone:

I have a strange problem with an Intel(R) Ethernet Controller XL710 for 40GbE card which I have. This card is connected via QSFP+ from a Supermicro server to a QSFP port on a Mellanox switch.

When I first installed FreeBSD 13.2, it shows this:

Code: 
Jul 10 13:28:50 mercury02 kernel: ixl2: <Intel(R) Ethernet Controller XL710 for 40GbE QSFP+ - 2.3.3-k> mem 0xc5000000-0xc57fffff,0xc5808000-0xc580ffff irq 48 at device 0.0 numa-domain 0 on pci12
Jul 10 13:28:50 mercury02 kernel: ixl2: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d923 oem 1.268.0
Jul 10 13:28:50 mercury02 kernel: ixl2: The driver for the device detected a newer version of the NVM image than expected.
Jul 10 13:28:50 mercury02 kernel: ixl2: Please install the most recent version of the network driver.
Jul 10 13:28:50 mercury02 kernel: ixl2: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
Jul 10 13:28:50 mercury02 kernel: ixl2: Using 1024 TX descriptors and 1024 RX descriptors
Jul 10 13:28:50 mercury02 kernel: ixl2: Using 8 RX queues 8 TX queues
Jul 10 13:28:50 mercury02 kernel: ixl2: Using MSI-X interrupts with 9 vectors
Jul 10 13:28:50 mercury02 kernel: ixl2: Ethernet address: xx:xx:xx:xx:xx:xx
Jul 10 13:28:50 mercury02 kernel: ixl2: Allocating 8 queues for PF LAN VSI; 8 queues active
Jul 10 13:28:50 mercury02 kernel: ixl2: PCI Express Bus: Speed 8.0GT/s Width x8
Jul 10 13:28:50 mercury02 kernel: ixl2: SR-IOV ready
Jul 10 13:28:50 mercury02 kernel: ixl2: netmap queues/slots: TX 8/1024, RX 8/1024
Jul 10 13:28:50 mercury02 kernel: ixl2: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Jul 10 13:28:50 mercury02 kernel: ixl2: link state changed to UP
Jul 10 13:35:44 mercury02 kernel: ixl2: link state changed to DOWN
Jul 10 13:35:48 mercury02 kernel: ixl2: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Jul 10 13:35:48 mercury02 kernel: ixl2: link state changed to UP

I tried the following:
  • setting a static ip address on that interface with a defaultrouter
    ifconfig ixl2 inet a.b.c.d netmask 255.255.252.0 up
  • getting a dhcp ip address on that interface with dhclient
    dhclient ixl2
  • as the same, but synchronously.
The issues were this:
  • The static IP could be set but nothing could be pinged.
  • dhcpclient times out and there were no leases offered.
Other things that I tried but having the same problems as I've described above:

1. Using intel-ilx-kmod
I noted that

Code: 
Jul 10 13:28:50 mercury02 kernel: ixl2: The driver for the device detected a newer version of the NVM image than expected.
Jul 10 13:28:50 mercury02 kernel: ixl2: Please install the most recent version of the network driver.
was mentioned, so I did just that, I downloaded and installed the net/intel-ixl-kmod driver, and modified my /boot/loader.conf to use it during startup.

After bootup, I see this in my logs.

Code: 
Jul 10 13:56:56 mercury02 kernel: ixl2: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.40> mem 0xc5000000-0xc57fffff,0xc5808000-0xc580ffff irq 48 at device 0.0 numa-domain 0 on pci12
Jul 10 13:56:56 mercury02 kernel: ixl2: using 1024 tx descriptors and 1024 rx descriptors
Jul 10 13:56:56 mercury02 kernel: ixl2: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d923 oem 1.268.0
Jul 10 13:56:56 mercury02 kernel: ixl2: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
Jul 10 13:56:56 mercury02 kernel: ixl2: Using MSI-X interrupts with 9 vectors
Jul 10 13:56:56 mercury02 kernel: ixl2: Allocating 8 queues for PF LAN VSI; 8 queues active
Jul 10 13:56:56 mercury02 kernel: ixl2: Ethernet address: xx:xx:xx:xx:xx:xx
Jul 10 13:56:56 mercury02 kernel: ixl2: PCI Express Bus: Speed 8.0GT/s Width x8
Jul 10 13:56:56 mercury02 kernel: ixl2: SR-IOV ready
Jul 10 13:56:56 mercury02 kernel: ixl2: The device is not iWARP enabled
Jul 10 13:56:56 mercury02 kernel: ixl2: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Jul 10 13:56:56 mercury02 kernel: ixl2: link state changed to UP

So obviously, the ixl driver was in effect, I tried again, setting a static ip address but there wasn't any success. I got something like this in the logs when I tried dhcp too.

Code: 
Jul 10 13:48:59 mercury02 kernel: ixl2: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Jul 10 13:48:59 mercury02 kernel: ixl2: link state changed to UP
Jul 10 13:49:19 mercury02 kernel: ixl2: WARNING: queue 2 appears to be hung!
Jul 10 13:49:19 mercury02 kernel: ixl2: WARNING: queue 4 appears to be hung!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 0 still enabled!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 1 still enabled!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 2 still enabled!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 3 still enabled!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 4 still enabled!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 5 still enabled!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 6 still enabled!
Jul 10 13:49:33 mercury02 kernel: ixl2: TX queue 7 still enabled!

So I'm unsure of what to make about this.

2. sysctl changes.
I also tried the following:
  • setting hw.ixl.enable_head_writeback to 0 (yeah, /boot/loader.conf and rebooting).
  • setting dev.ixl.2.fc=2 and dev.ixl.2.advertise_speed to 0x00000020
but no success so far.

side note: I rebooted the whole machine on a live linux cd and dhcp on the QSFP+ interface was successfully obtained. So that led me to conclude that something could be wrong on the FreeBSD side.
I've also tried this out with FreeBSD 12.4 and had the same issues.

I'm wondering if anyone has encountered anything similar to what I went through and any help on this would be much appreciated :).
 
Sorry I see you are using that driver. The SFP module is the second place I would look.
Is it Intel branded or generic?
Take a look at its status with ifconfig -vv ixl2
See if you have temps reported for the module.
 
If all that checks out I would focus on that initial message.
These messages are bad in both SAS cards and 10G networking.
You should have the driver and firmware at same level.
Consider downgrading firmware to match one from ports.

Do you still see the driver mismatch error with the ports ixl driver?
 
calebk said:
Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Click to expand...
Do you have this stuff turned off with ifconfig? I would expect Autoneg True.

It is why you are having to set speed manually (and probably wont work)

Somethings going on here with the driver..
 
Please show ifconfig -vv ixl2 and ifconfig from other ports with SFP attached..

Can you tell me if there are other ixl interfaces in use? How about ifconfig -l

I am trying to figure out why you are showing ixl2. I would expect ixl0.
 
Phishfry thank you for your prompt reply.

Yes, ixl0 is in use by another ethernet NIC (the Intel(R) Ethernet Connection X722 for 10GBASE-T), but I've tried disabling that on the BIOS, therefore just leaving just ixl0 and ixl1 for the XL710 intact.
That didn't help, as I still have the same symptoms as i have described. I'm using a DAC cable not any fiber over SFP transceivers.

Here's the output of ifconfig -vv ixl2

Code: 
root@mercury02:~ # ifconfig -vv ixl2
ixl2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=606407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,VXLAN_HWCSUM,VXLAN_HWTSO>
    ether xx:xx:xx:xx:xx:xx
    media: Ethernet autoselect (40Gbase-CR4 <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    plugged: QSFP+ 40GBASE-CR4 (No separable connector)
    vendor: FCI Electronics PN: 10131215-2010LF SN: CN1AKS60L1 DATE: 2021-11-04
    compliance level: SFF-8636 rev <=1.5
    nominal bitrate: 10300 Mbps
    module temperature: 0.00 C voltage: 0.00 Volts
    lane 1: RX power: 0.00 mW (-inf dBm) TX bias: 0.00 mA
    lane 2: RX power: 0.00 mW (-inf dBm) TX bias: 0.00 mA
    lane 3: RX power: 0.00 mW (-inf dBm) TX bias: 0.00 mA
    lane 4: RX power: 0.00 mW (-inf dBm) TX bias: 0.00 mA

I have no idea why Autoneg is false, I tried to see in the ixl man page where I can forcefully set that to true, but I couldn't find it.
If there is a way to force it to true, then I'd like to know how. There isn't a 'ethtool' thing in freebsd unless it is possible to do so via intel-qcu (I've not tried that yet)

I also include the output of sysctl -a | grep ixl2 and that of dmesg.boot. Note that nothing about ixl2 is specified in the /etc/rc.conf at this time.

Code: 
ixl2: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.40> mem 0xc5000000-0xc57fffff,0xc5808000-0xc580ffff irq 48 at device 0.0 numa-domain 0 on pci12
ixl2: using 1024 tx descriptors and 1024 rx descriptors
ixl2: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d923 oem 1.268.0
ixl2: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
ixl2: Using MSI-X interrupts with 9 vectors
ixl2: Allocating 8 queues for PF LAN VSI; 8 queues active
<6>ixl2: Ethernet address: xx:xx:xx:xx:xx:xx
ixl2: PCI Express Bus: Speed 8.0GT/s Width x8
ixl2: SR-IOV ready
ixl2: The device is not iWARP enabled
<5>ixl2: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
<6>ixl2: link state changed to UP
<118>Starting Network: lo0 ixl0 ixl1 ixl2 ixl3.
<118>ixl2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>Starting Network: ixl2.
<118>ixl2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
irq134: ixl2:aq:267 @cpu0(domain0): 2
irq135: ixl2:q0:269 @cpu0(domain0): 0
irq136: ixl2:q1:271 @cpu1(domain0): 0
irq137: ixl2:q2:273 @cpu2(domain0): 0
irq138: ixl2:q3:275 @cpu3(domain0): 0
irq139: ixl2:q4:277 @cpu4(domain0): 0
irq140: ixl2:q5:279 @cpu5(domain0): 0
irq141: ixl2:q6:281 @cpu6(domain0): 0
irq142: ixl2:q7:283 @cpu7(domain0): 0

Code: 
root@mercury02:~ # grep ixl2 /var/run/dmesg.boot 
ixl2: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.40> mem 0xc5000000-0xc57fffff,0xc5808000-0xc580ffff irq 48 at device 0.0 numa-domain 0 on pci12
ixl2: using 1024 tx descriptors and 1024 rx descriptors
ixl2: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d923 oem 1.268.0
ixl2: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
ixl2: Using MSI-X interrupts with 9 vectors
ixl2: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl2: Ethernet address: xx:xx:xx:xx:xx:xx
ixl2: PCI Express Bus: Speed 8.0GT/s Width x8
ixl2: SR-IOV ready
ixl2: The device is not iWARP enabled
ixl2: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl2: link state changed to UP
 
Ok, I unplugged ixl2, and connected to ixl3 (the other port in the XL710 card) and it worked fine, I'm able to get a dhcp address and setting it to a static value also works.
All this is done via the dhclient ixl3 command at the command line. But when I added a ifconfig_ixl3="DHCP" and rebooted, it does not work, I disabled the X722 and rebooted but all I get is a lot of this:

Code: 
Jul 11 14:29:33 mercury02 kernel: ixl1: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.40> mem 0xc4800000-0xc4ffffff,0xc5800000-0xc5807fff irq 48 at device 0.1 numa-domain 0 on pci9
Jul 11 14:29:33 mercury02 kernel: ixl1: using 1024 tx descriptors and 1024 rx descriptors
Jul 11 14:29:33 mercury02 kernel: ixl1: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d923 oem 1.268.0
Jul 11 14:29:33 mercury02 kernel: ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
Jul 11 14:29:33 mercury02 kernel: ixl1: Using MSI-X interrupts with 9 vectors
Jul 11 14:29:33 mercury02 kernel: ixl1: Allocating 8 queues for PF LAN VSI; 8 queues active
Jul 11 14:29:33 mercury02 kernel: ixl1: Ethernet address: xx:xx:xx:xx:xx:xx
Jul 11 14:29:33 mercury02 kernel: ixl1: PCI Express Bus: Speed 8.0GT/s Width x8
Jul 11 14:29:33 mercury02 kernel: ixl1: SR-IOV ready
Jul 11 14:29:33 mercury02 kernel: ixl1: The device is not iWARP enabled
Jul 11 14:29:33 mercury02 kernel: ixl1: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
Jul 11 14:29:33 mercury02 kernel: ixl1: link state changed to UP
Jul 11 14:29:33 mercury02 kernel: ixl1: link state changed to DOWN
Jul 11 14:29:33 mercury02 kernel: ixl1: Link is up, 40 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
Jul 11 14:29:33 mercury02 kernel: ixl1: link state changed to UP
Jul 11 14:29:33 mercury02 kernel: ixl1: WARNING: queue 3 appears to be hung!
Jul 11 14:29:33 mercury02 kernel: ixl1: WARNING: queue 2 appears to be hung!
Jul 11 14:29:33 mercury02 kernel: ixl1: WARNING: queue 6 appears to be hung!
Jul 11 14:29:33 mercury02 kernel: ixl1: WARNING: queue 6 appears to be hung!
Jul 11 14:30:25 mercury02 kernel: ixl1: WARNING: queue 7 appears to be hung!
Jul 11 14:30:25 mercury02 kernel: ixl1: WARNING: queue 0 appears to be hung!
Jul 11 14:32:12 mercury02 kernel: ixl1: WARNING: queue 6 appears to be hung!
Jul 11 14:32:12 mercury02 kernel: ixl1: WARNING: queue 1 appears to be hung!

However, I swapped the connection back to the old port (now known as ixl0), and did a dhclient on it manually on the command line. I'm able to get a dhcp address.
It does seem that there's some race condition that /etc/rc.conf cannot be used when booting up, I don't know if there is a way to delay that and manually switch on the interface, or something that I don't know.

EDIT: I could still add ifconfig_ixl0="DHCP" to /etc/rc.conf and do service dhclient start ixl0 and it works fine... as long as I don't reboot. As soon as I do, ixl0 does not come back up upon reboot and the messages about queues hung appear.
 
Just for giggles I would consider adding a wait period somewhere in startup.
Maybe the firmware is not loaded fully before interface is trying to bring it up.

For testing try to configure the interface from a file you create called /etc/rc.local

Code: 
sleep 10
ifconfig ixl000 blah blah blah
 
calebk said:
vendor: FCI Electronics PN: 10131215-2010LF SN: CN1AKS60L1 DATE: 2021-11-04
Click to expand...
Is that Intel branded? If not there is a sysctl setting for non-compliant modules.
Intels driver does a check.

append
/etc/sysctl.conf
hw.ixl.unsupported_sfp=1
 
calebk said:
as I don't reboot. As soon as I do, ixl0 does not come back up upon reboot and the messages about queues hung appear.
Click to expand...
I see alot of strange results with reboots and firmwares. Some (Intels) wifi chips firmware act plain stupidly.
Like somehow the firmware survives a reboot from Linux to FreeBSD but won't load on cold boot from FreeBSD.
I thought with reboot memory is wiped. but something in card firmware survives a reboot. Blobs? What can you do.

Try and add some delay for interface as suggested. It is an easy check. Don't be afraid to increase sleep time for testing.

Also do you need all this:
calebk said:
options=606407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,VXLAN_HWCSUM,VXLAN_HWTSO>
Click to expand...

Mellanox switch may be finicky. MTUs agree? ixl manpage says max 9706

ixl(4)

man.freebsd.org man.freebsd.org

An example from my 10G gear to Cisco. No VLANs in use.
ifconfig_cxl0="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"

Last ditch suggestions.
Switch card in PCIe slots.
It sounds like your Intel 10G is built into motherboard so you can't yank as I would like.
So try switching slots.
If dual cpu box weird stuff happens across cpu boundries. Stick with a cpu0 slot if possible.
Slots are assigned per CPU in SMP boxes. SM Motherboard manpage will have details. BIOS shows it too.
Try your 40G card in a cpu1 slot if that goes nowhere.
Is it an Intel or AMD SM server?
 
This one is very recent PR on ixl but different problem.

It does have useful stuff to check like checksum errors.

But I found the firmware comment especially interesting.
I solved it by updating my x710's firmware to 12.
Perhaps your x710 firmware is outdated
Click to expand...

262093 – ixl(4): RX packet errors on Intel X710 after 12.2 to 12.3-RELEASE upgrade

bugs.freebsd.org bugs.freebsd.org

Which brings me to the versions you are using:
calebk said:
<Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.40>
[snip]
fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d923 oem 1.268.0
Click to expand...

The firmware numbers don't look anything like the driver version.
With 'OEM' in name am I to guess this is SuperMicro version of Intel 40G card?
 
Thank you Phishfry I'll try the timeout delay and sysctl setting for non-Intel branded cards.
Yes, my setup is a Supermicro, so should these still not work , I'll contact them for a firmware update.

Will report later if it works or not. :).
 
Ok, here are some results after some experiments.
  • delay does not work
    setting the delay in rc.local does not work.
  • hw.ixl.unsupported_sfp=1 in sysctl is unrecognised so I put it into /boot/loader.conf.
    that does not work either. I still receive a lot of queue hung errors.
    I did a lookup on the part number of FCI Electronics PN: 10131215-2010LF and this is actually the DAC cable part, so we might be barking up the wrong tree here.
  • I tried the ifconfig flags that you suggested, it does not work either.
  • I've not asked Supermicro for a firmware update, but from the forums, it appears that version 9 (which I have) is the latest.
The ONLY way it will work with FreeBSD is if I unplugged the DAC cable from both ports, restart FreeBSD and let FreeBSD boot totally to multi user mode and then plug the DAC cable in, then it works.
In other words, I cannot have my XL710 connected upon boot, this will not work, it can only work when it is connected after bootup.
 
I have nothing more. I never used sleep in rc.local but I would be real surprised if it didn't work. Maybe parallel processes are still being run from rc.d while rc.local is launched. I dunno. It should fire last.
 
Phishfry said:
I have nothing more. I never used sleep in rc.local but I would be real surprised if it didn't work.
Click to expand...

It works, but since rc.local is sourced (i.e. run in sequence by its 'caller' script (/etc/rc)) then the sleep also holds up any later local scripts in rcorder.

Yes you could use rcorder to make it run last, but easiest have rc.local launch another script in background then exit, so starting other local daemons can proceed.

bg script could:
logger started sleep
sleep $sleep
logger finished sleep
ifconfig $whatever
# test up and working ...
logger $results
exit

Verbose boot shows timings in /var/log/messages.

Phishfry said:
Maybe parallel processes are still being run from rc.d while rc.local is launched. I dunno. It should fire last.
Click to expand...

Dunno about 13.x, but that might be more a 'systemd' type of thing - fast but likely 'quirkily indeterminate'.

With a separate process you can play outside rc's timings.
 
That seems to be consistent, I don't think it's a rc.local issue as opposed to the adapter cannot be connected upon boot up, but rather after in FreeBSD.

calebk said:
The ONLY way it will work with FreeBSD is if I unplugged the DAC cable from both ports, restart FreeBSD and let FreeBSD boot totally to multi user mode and then plug the DAC cable in, then it works.
In other words, I cannot have my XL710 connected upon boot, this will not work, it can only work when it is connected after bootup.
Click to expand...

I've tried to prevent the loading of the ixl from boot (by setting if_ixl_updated="NO") but it does not work either.
http://web.mit.edu/freebsd/head/sys/dev/ixl/ also mentioned something about memory buffers, which I'm sure mine is adequate.

I also tried

dev.ixl.0.link_active_on_if_down=0

in /boot/loader.conf to prevent it from being active during startup, but the moment I do ifconfig, it shows this:

ixl0: TX queue 0 still enabled! ixl0: TX queue 1 still enabled! ixl0: TX queue 2 still enabled! ixl0: TX queue 3 still enabled! ixl0: TX queue 4 still enabled! ixl0: TX queue 5 still enabled! ixl0: TX queue 6 still enabled! ixl0: TX queue 7 still enabled! ...

I don't think changing the pci slot would help because I don't feel the problem lies within the hardware (because it works out-of-the-box with a linux live cd)

Thanks for all your replies, appreciate it. I think I'll have to go with a linux setup even though I would have preferred a freebsd one.
 
Actually, this issue

FreeBSD 11.3 ixl problems

FreeBSD 11.3 using driver 1.12.3 compiled from Intel's website. NVM version is 7.10. /etc/sysctl.conf contains: net.inet.tcp.tso=0 After booting the system the NICs are enabled and I'll see the following message: "ixl: TX queue still enabled!". This message will be repeated multiple times even...
community.intel.com

described my problem perfectly.
 
You must log in or register to reply here.
Back
Top