Difficulties with SR-IOV on Chelsio T4 cards (driver can’t get interrupts)

arader

Member

Reaction score: 3
Messages: 23

Hi all,

I’m finally upgrading to 10Gbe with a pair of Chelsio T420-CR cards and one of my goals is to get off of `bridge` and `epair` devices for my jails and instead use VF devices from SR-IOV.

However, when trying to enable these virtual devices the drivers don’t attach. What I have so far:

12.0-RELEASE p3
Intel Xeon 1245v3

cat /boot/loader.conf
Code:
zfs_load="YES"
t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
if_cxgbe_load="YES"
if_cxgbev_load="YES"
t4_tom_load="YES"
Then I have an iovctl config file for my device:
cat /etc/iovctl.cxgbe0.conf
Code:
PF {
    device : "cxgbe0";
    num_vfs : 4;
}
I can confirm that this enables the VF portions of the card:
doas pciconf -lvc
Code:
t4iov0@pci0:1:0:0:      class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x8(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI disabled
                     4 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    iov bar  [1e8] = type Memory, range 64, base 0xef000000, size 4096, enabled
    iov bar  [1f8] = type Memory, range 64, base 0xef004000, size 8192, enabled
    ecap 000e[204] = ARI 1
However, when I look at ifconfig I don’t see any new interfaces. Sure enough, if I look at dmesg I see 4 entries of the following:

Code:
t4vf0: <Chelsio T420-CR VF> at device 1.0 on pci1
t4vf0: failed to find a usable interrupt type.  allowed=7, msi-x=0, msi=0, intx=1device_attach: t4vf0 attach returned 6
So it looks like the drivers can’t get any interrupts for the interfaces. Looking at other entries in pciconf I do see several using MSI and MSI-X, so it’s not totally broken, but it might be interesting that the Chelsio devices don’t have MSI/MSI-X enabled...

So, anyone have any thoughts? MSI-X appears to be an automatic thing on FreeBSD, should I be tweaking the BIOS or something?

thanks!
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

I don't know about your interrupt errors.

So I have 4 SR-IOV devices:
Code:
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4iov1@pci0:4:0:1:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4iov2@pci0:4:0:2:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4iov3@pci0:4:0:3:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4nex0@pci0:4:0:4:    class=0x020000 card=0x00001425 chip=0x44011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
I have only been able to get one interface working this way with VF's.
I ppt passthru one of the t4iov devices and t4nex0 in the loader.
With that I get a virtualized chelsio interface inside a bhyve VM.
It seems to take the combination of the two devices for some reason.

There is a totally separate second way. That may take another post...
/boot/loader.conf
#hw.cxgbe.num_vis=2
 
OP
OP
arader

arader

Member

Reaction score: 3
Messages: 23

Thanks Phishfry, the t4_tom module is needed for enabling TOE on these cards, since I'm not using it right now I removed that from my imports. Still the same issue.

I snooped around in my BIOS and couldn't find anything obvious, VT-d is listed as enabled. I couldn't find any interrupt related options, which makes sense (the days of IRQ numbers are long behind us) so I'm at a bit of a loss.

Phishfry can you do me a favor and post pciconf -lcv for your Chelsio devices? I'm curious if they're using MSI/MSI-X in your setup. I have the same t4iov/t4nex0 devices as you.

Frankly I'm a little disappointed in the documentation around SR-IOV and FreeBSD - this is the first time I'm having to really scrape the edges of the internet to get any details on how to set this up.
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

Here you go:
Code:
root@virt:~ # pciconf -lbcev t4iov0
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xc7380000, size 262144, enabled
    bar   [20] = type Memory, range 64, base 0xc744c000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x1(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    ecap 000e[204] = ARI 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Receiver Error
                 Advisory Non-Fatal Error
Intresting that you show this interface:
t4vf0: <Chelsio T420-CR VF> at device 1.0 on pci1
So this is in the VM?
I have never seen that interface. I did have vcxgbe0 with the second method.
Like I say if I pass thru both t4iov and t4nex0 I get one virtual interface but it uses cxgbe moniker.
So it looks like I am not doing it right. I need to add "if_cxgbev_load="YES"" to my setup.
Yes instructions would be nice. I am using two i-350 quad Intel NIC for now.
I did read where t4_tom is TOE so that is irrelevant. Offloading is interesting but not needed with my load.

Code:
root@virt:~ # sysctl -a | egrep -i 'hw.machine|hw.ncpu|hw.model'
hw.machine: amd64
hw.model: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
hw.ncpu: 48
hw.machine_arch: amd64
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

Arg I forgot to uncomment out this line in /etc/rc.conf
iovctl_files="/etc/cxgbe0.conf"
I have been down this road already...Hopefully I get further this time.
That is why the pciconf above shows no sr-iov.
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

Here is relevant dmesg for my card showing interrupts.
Code:
cxgbe0: <port 0> numa-domain 0 on t4nex0
cxgbe0: Ethernet address: 0
cxgbe0: 16 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
cxgbe1: <port 1> numa-domain 0 on t4nex0
cxgbe1: Ethernet address: 0
cxgbe1: 16 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
t4nex0: PCIe gen2 x1, 2 ports, 22 MSI-X interrupts, 71 eq, 21 iq
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

I have finally got somewhere. I have 4 VF's
Code:
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xc7380000, size 262144, enabled
    bar   [20] = type Memory, range 64, base 0xc744c000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x1(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled
                     4 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    iov bar  [1e8] = type Memory, range 64, base 0xc7000000, size 4096, enabled
    iov bar  [1f8] = type Memory, range 64, base 0xc7004000, size 8192, enabled
    ecap 000e[204] = ARI 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Receiver Error
                 Advisory Non-Fatal Error
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

And jacked up to full blast we get 16 VF's. That is only one interface. cxgbe0. So 32 VF's possible with a dual port card.
Code:
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xc7380000, size 262144, enabled
    bar   [20] = type Memory, range 64, base 0xc744c000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x1(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled
                     16 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    iov bar  [1e8] = type Memory, range 64, base 0xc7000000, size 4096, enabled
    iov bar  [1f8] = type Memory, range 64, base 0xc7010000, size 8192, enabled
    ecap 000e[204] = ARI 1
 
OP
OP
arader

arader

Member

Reaction score: 3
Messages: 23

From my reading, if you have “passthrough” set to true in the iovctl conf file you get a ‘ppt’ pci device that can be passed to a VM. If it’s false then you can directly attach in the host and create an interface for jails.

The t4vf0 comes from the cxgbev driver, after loading that i would expect to see virtual interfaces show in the host.

Looking through the cxgbev driver sys/dev/cxgbe/t4_vf.c I can definitely see that my interrupt error in dmesg is why i’m not seeing any interfaces. That gets logged in the failure path and nothing gets attached. If the driver were able to get some interrupts it would have moved on to attaching the devices.

I just noticed that pciconf shows my t4nex0 device as using MSI-X so it is working. I’m going to try flashing an updated BIOS tomorrow and see if that changes anything. Something is causing the kernel to report 0 available interrupts...
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

If it’s false then you can directly attach in the host and create an interface for jails.
This setting of 'false' was crashing my machine. It is odd because it was in my config from my earlier attempt.
I am sure I have something set wrong. For now it is all passthru. None for the host.
 
OP
OP
arader

arader

Member

Reaction score: 3
Messages: 23

So I just tried setting passthrough to true and i can get a ‘ppt’ device just fine. I’m beginning to suspect that “non-passthrough” just isn’t supported by these drivers (yet?) and I’ll have to just keep using bridge+epair in my jails.

At least I can pass through the VF for my VMs!
 
OP
OP
arader

arader

Member

Reaction score: 3
Messages: 23

Updated the BIOS on my Supermicro X10SAT and still no luck, I can't seem to get any interfaces attached for non-passthrough VFs on the T420-CR.

Additionally, if I create multiple VFs and set them all to "passthrough: true" I still only end up with 1 ppt device. I would assume I should end up with a ppt for each VF that I can pass through?

Man, I was told Chelsio was well supported on FreeBSD but I'm really not getting that vibe here...
 
Top