Intel X553 driver Support for Freebsd 12.1

[nRgia]

I have the following board: https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F

I use Netmap on FreeBSD 12.1:

https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4



The driver for FreeBSD as far as I know is this:

https://downloadcenter.intel.com/do...10-Gigabit-Network-Connections-Under-FreeBSD-

pciconf -lvc output:

ix3@pci0:8:0:1: class=0x020000 card=0x00008086 chip=0x15e58086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 1GbE'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) FLR RO
                 link x1(x1) speed 2.5(2.5) ASPM L0s/L1(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 0100c9ffff000000
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0080, VF RID Stride 0x0002
                     VF Device ID 0x15c5
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 000d[1b0] = ACS 1

1) After compilation the following if_if.ko is created as a driver, although in the driver "readme files" Intel mentions it should output "ixgbe.ko". Is this a bug?

2) Linux, FreeBSD and Netmap supports "ixgbe" driver, but not IX, what is IX driver? Shouldn't x553 support be included in "ixgbe" driver as in Linux?

3) Why in Linux after compilation we have "ixgbe.ko" and on FreeBSD "if_ix.ko" ? I don't find any info on the IX driver in FreeBSD?

Observation:

On Linux using ixgbe.ko driver, Netmap works and I achieve 960 Mbs/s

On Freebsd using if_ix.ko driver, Netmap works only in emulation mode, and I achieve only ~150 Mbs/s


4) If Netmap works on Linux with the same card and not on FreeBSD, I tend to believe the driver is the issue here.
5) I compiled from source also, using ix-kmod, but if I force Netmap to enabled in "make config", kldload will say that a symbol "D" is not recognized, and it will not load
6) Is this lack of support from Intel, I am asking this, because I wrote on their forums, and they redirected me to the FreeBSD forums, although I fail to see how a driver can be a FreeBSD issue, just need confirmation


dmesg output:

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.10> mem 0x7d7fc00000-0x7d7fdfffff,0x7d7fe04000-0x7d7fe07fff at device 0.0 on pci6
ix0: Using MSI-X interrupts with 5 vectors
ix0: Ethernet address: ac:1f:6b:45:fa:88
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.10> mem 0x7d7fa00000-0x7d7fbfffff,0x7d7fe00000-0x7d7fe03fff at device 0.1 on pci6
ix1: Using MSI-X interrupts with 5 vectors
ix1: Ethernet address: ac:1f:6b:45:fa:89
ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.10> mem 0x7d7f600000-0x7d7f7fffff,0x7d7f804000-0x7d7f807fff at device 0.0 on pci7
ix2: Using MSI-X interrupts with 5 vectors
ix2: Ethernet address: ac:1f:6b:45:fa:8a
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.3.10> mem 0x7d7f400000-0x7d7f5fffff,0x7d7f800000-0x7d7f803fff at device 0.1 on pci7
ix3: Using MSI-X interrupts with 5 vectors

Netmap output:

ix3: link state changed to UP
252.220553 [1130] generic_netmap_attach     Emulated adapter for ix3 created (prev was NULL)
252.220605 [1035] generic_netmap_dtor       Emulated netmap adapter for ix3 destroyed
ix3: permanently promiscuous mode enabled
252.245959 [1130] generic_netmap_attach     Emulated adapter for ix3 created (prev was NULL)
252.498624 [ 320] generic_netmap_register   Emulated adapter for ix3 activated

All information about NETMAP and accepted drivers are here:

https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4



Mainly it is used by Snort and Suricata for IDS/IPS hence I need a proper driver for it


Can you please shade light on the above,

Thank you

 

[SirDice]

As far as I've been able to find out the Intel X553 should be supported by ixgbe(4). No need to install anything.

 

[T-Daemon]

1) After compilation the following if_if.ko is created as a driver, although in the driver "readme files" Intel mentions it should output "ixgbe.ko". Is this a bug?

I assume you mean by if_if.ko if_ix.ko, no bug, old readme ( February 04, 2015 ). From Revision 280182 ( Mar 17 2015 ):

Log Message:
Update to the Intel ixgbe driver:
           - The driver loadables will now match the device names, something that
              has been requested for some time.
           - Rather than a modules/ixgbe there is now modules/ix and modules/ixv

2) Linux, FreeBSD and Netmap supports "ixgbe" driver, but not IX, what is IX driver? Shouldn't x553 support be included in "ixgbe" driver as in Linux?

3) Why in Linux after compilation we have "ixgbe.ko" and on FreeBSD "if_ix.ko" ?

See above response.

I don't find any info on the IX driver in FreeBSD?

man ix(4), if_ix(4), if_ixgbe(4), ixgbe(4) ( /usr/ports/net/intel-ix-kmod/Makefile MANPAGELINKS= ix if_ix if_ixgbe )
/usr/src/sys/dev/ixgbe

Intel Networking

reviews.freebsd.org

Observation:

On Linux using ixgbe.ko driver, Netmap works and I achieve 960 Mbs/s

On Freebsd using if_ix.ko driver, Netmap works only in emulation mode, and I achieve only ~150 Mbs/s

Have you tried the sysctl variable dev.netmap.admode ( netmap(4) )?

dev.netmap.admode:    0
         Controls the use of native    or emulated adapter mode.

         0 uses the    best available option;

         1 forces native mode and fails if not available;

         2 forces emulated hence never fails.

Also, if I'm not mistaken, ix has link speed advertise control flags:

For current setting:
sysctl hw.ix.0.advertise_speed

For available:
sysctl -d hw.ix.0.advertise_speed

5) I compiled from source also, using ix-kmod, but if I force Netmap to enabled in "make config", kldload will say that a symbol "D" is not recognized, and it will not load

This is from net/intel-ix-kmod Makefile:

MAX_NETMAP_OSVERSION=   1199999  # Doesn't build w/NETMAP on 12

 

[nRgia]

I assume you mean by if_if.ko if_ix.ko, no bug, old readme ( February 04, 2015 ). From Revision 280182 ( Mar 17 2015 ):

Log Message:
Update to the Intel ixgbe driver:
           - The driver loadables will now match the device names, something that
              has been requested for some time.
           - Rather than a modules/ixgbe there is now modules/ix and modules/ixv

So what to understand from here that ixgbe for FreeBSD driver will have a different name after compilation?

I compiled this one:

Intel® Network Adapters Driver for PCIe* 10 Gigabit Network Connections Under FreeBSD*

This release includes the 10 gigabit FreeBSD* Base Driver for Intel® Network Connections.

downloadcenter.intel.com

Is this the correct driver?

I don't find an ixgbe driver on Intel site for FreeBSD, only for Linux:

Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections under Linux*

Includes Linux*-based drivers version 5.12.5 for Intel® 10 Gigabit Ethernet Network Connections with PCI Express*.

downloadcenter.intel.com

Also what to understand from Netmap support page:

netmap(4)

www.freebsd.org

 netmap natively supports the following devices:

     On    FreeBSD: cxgbe(4), em(4), iflib(4) (providing igb, em and lem),
     ixgbe(4), ixl(4), re(4), vtnet(4).

That this mean no native Netmap support for my module? Or it should support it, because it's just a different module name?


See above response.


man ix(4), if_ix(4), if_ixgbe(4), ixgbe(4) ( /usr/ports/net/intel-ix-kmod/Makefile MANPAGELINKS= ix if_ix if_ixgbe )
/usr/src/sys/dev/ixgbe

Intel Networking

reviews.freebsd.org

Have you tried the sysctl variable dev.netmap.admode ( netmap(4) )?

dev.netmap.admode:    0
         Controls the use of native    or emulated adapter mode.

         0 uses the    best available option;

         1 forces native mode and fails if not available;

         2 forces emulated hence never fails.

If I force Netmap mode to 1 (Native mode) it will fail to start.
Only option 0 and 2 works

From my personal hands on experience, on FreeBSD 11.3 Netmap worked in native mode, after I installed FreeBSD 12.1 no luck

Also, if I'm not mistaken, ix has link speed advertise control flags:

For current setting:
sysctl hw.ix.0.advertise_speed

Output:

sysctl: unknown oid 'hw.ix.0.advertise_speed'
sysctl: unknown oid 'hw.ix.1.advertise_speed'
sysctl: unknown oid 'hw.ix.2.advertise_speed'
sysctl: unknown oid 'hw.ix.3.advertise_speed'

For available:
sysctl -d hw.ix.0.advertise_speed

Output:

sysctl hw.ix.0.advertise_speed
sysctl hw.ix.1.advertise_speed
sysctl hw.ix.2.advertise_speed
sysctl hw.ix.3.advertise_speed

This is from net/intel-ix-kmod Makefile:

MAX_NETMAP_OSVERSION=   1199999  # Doesn't build w/NETMAP on 12

I'm not questioning the English language here, but that this translate to, no NETMAP support for FreeBSD 12.1 for this driver/card? Or I cannot compile with Netmap enabled?

Do I need to ask Intel for a newer driver to enable Netmap support?

I appreciate that you have looked on forums and such for me.

If you still can, what can I do from here. I'm trying to understand to whom to turn for Netmap support, FreeBSD, Intel, SuperMicro?

Thank you

 

[nRgia]

As far as I've been able to find out the Intel X553 should be supported by ixgbe(4). No need to install anything.

You are correct, it works by default with the driver that comes with FreeBSD, it worked also with FreeBSD 11.3 but the transfer speed was slow, when Netmap was activated ~150 Mbs/s.

On FreeBSD 11.3 I downloaded and compiled this driver:

Intel® Network Adapters Driver for PCIe* 10 Gigabit Network Connections Under FreeBSD*

This release includes the 10 gigabit FreeBSD* Base Driver for Intel® Network Connections.

downloadcenter.intel.com

After i loaded the if_ix.ko Netmap worked in Native mode and the transfer speed was 950 Mbs/s

The issue here, is that on FreeBSD 12.1 if i load the same driver, Netmap will not start in Native mode anymore, only in emulated mode. From what I understand Netmap will start this way if the card or the driver is not supported. If it worked on FreeBSD 11.3, I tend to incline that the driver is the culprit.

Maybe I'm not compiling the correct driver. I don't find ixgbe driver for FreeBSD anymore on Intel site, it's only for Linux.

Also I tried with Debian and ixgbe driver. Netmap also works. So it's not a hardware issue IMHO

 

[T-Daemon]

So what to understand from here that ixgbe for FreeBSD driver will have a different name after compilation?

Exactly.

Is this the correct driver?

It seems so. There is no other current driver available for FreeBSD.

I don't find an ixgbe driver on Intel site for FreeBSD, only for Linux:

You won’t find a ixgbe named driver for current supported FreeBSD versions. Since 2015 the driver name has changed to ix.

That this mean no native Netmap support for my module?

If your testing of the driver results high speed in 11.3 but not in 12.1 and the entry in the Makefile of the port suggests it is not supported in 12.1, then no.

For current setting:
sysctl hw.ix.0.advertise_speed

Try sysctl hw.ix.advertise_speed (-d) , but I doubt changing the setting there will have an influence on the speed, if dev.netmap.admode has no effect.

If you still can, what can I do from here.

For the port I would contact the maintainer of net/intel-ix-kmod ( in this case someone / some people from INTEL ):

make -C /usr/ports/net/intel-ix-kmod maintainer
freebsd@intel.com

For base and port driver you could ask at freebsd-net@freebsd.org mailing list. You might find INTEL employees subscribed there ( I remember seeing subscriber @intel.com mail addresses there ).

At last file a bug report.

If you get any new information in this matter, consider letting the forum know, so others finding this thread can see what to expect.

 

[nRgia]

Sure I let the forum know, I already filled a ticket with SuperMicro and started a forum post with Intel here: https://forums.intel.com/s/profile/0050P000008v6nFQAQ
but I think the guy responding is Level 1 support or something, because I got nothing helpful. If I will get a proper answer I will re-post here.
Thank you for your time

 

[nRgia]

For the moment I have SuperMicro's response that confirmed what we have already found , that it's a driver issue as you can see:

As promised, I will add more information if relevant.

Thank you

 

[nRgia]

@T-Daemon

I got lucky and got a response from Intel:

He also told me that FreeBSD 12.1 generic kernel includes the driver by "device ix" in the kernel config file

Then I unloaded the if_ix.ko that I've compiled and ran
sysctl dev.ix.3:

dev.ix.3.iflib.driver_version: 4.0.1-k

dev.ix.3.%parent: pci7

dev.ix.3.%pnpinfo: vendor=0x8086 device=0x15e5 subvendor=0x8086 subdevice=0x0000 class=0x020000

dev.ix.3.%location: slot=0 function=1 dbsf=pci0:8:0:1 handle=\_SB_.PCI0.VRP1.LAN3

dev.ix.3.%driver: ix

dev.ix.3.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver

I found this about iflib https://www.freebsd.org/cgi/man.cgi?query=iflib&sektion=9&manpath=freebsd-release-ports


The card is seen by the system with the default drivers, but the transfer speed is very low, when Netmap is activated.

But I still cannot understand how I can update a driver to a newer version using iflib, in order to get the same performance as in FreeBSD 11.3

What changed:
1) Without the compiled module Netmap starts in Native mode, but the transfer speed is the same as in emulated mode which is ~150 Mbs/s
2) When running
sysctl hw.ix.advertise_speed

I got

hw.ix.advertise_speed: 0

and If I run
sysctl -d hw.ix.advertise_speed

I got

hw.ix.advertise_speed: Default advertised speed for all adapters

How to proceed forward:

1) Do I need to recompile the kernel to bring the driver to a newer version, or is just a tuning issue?
2) How can do I proceed further( and what is the best practice) ?
3) The driver version on Intel site is 3.3.10, and sysctl reports 4.0.1-k. Is it newer, or what that this mean ?

I like to point out I'm not an expert to FreeBSD, but with documentation, and a little guiding I can test further.

Thank you

 

[nRgia]

Having a look over FreeBSD github, and after coreleting my device ID, I see the FreeBSD driver indetifies my NIC as the following device:

freebsd-src/ixgbe_type.h at master · freebsd/freebsd-src

FreeBSD src tree (read-only mirror). Contribute to freebsd/freebsd-src development by creating an account on GitHub.

github.com

If someone can be so kind to respond to a few questions:
1) I would like to know if X553 chipset is supported,because I cannot find this information anywhere.
2) Can I find some release notes to correlate with Intel release notes, for example for the Linux release notes I have a clear despcription of which NICS the driver supports as seen here:
https://downloadcenter.intel.com/do...bit-Ethernet-Network-Connections-Under-Linux-
3) If this is the correct driver why I achieve only 150 Mbs/s when NETMAP is running, in comparisson to FreeBSD 11.2, where I could get at least 600 Mbs/s?

Thank you

 

[nRgia]

I got a very interesting response from Intel, after I asked if their latest release 3.3.10 supports X553 chipsets.

My big question is, because this is a real life case scenario. How can I track if Intel's code from 3.3.14 release is included already, or when it will be included with the FreeBSD upstream driver?

Thank you