SR-IOV (Chelsio) error in guest

pos

Member

Reaction score: 10
Messages: 43

The Chelsio T520-BT card work if inserted into a linux host. Now it is placed in FreeBSD 12.0p3 RELEASE host and given out to a CentOS 7.6 guest with pass through. Then I can see this in the guest....

3.466926] cxgb4vf: Chelsio T4/T5/T6 Virtual Function (VF) Network Driver - version 2.0.0-ko
[ 3.466995] cxgb4vf 0000:00:06.0: enabling device (0004 -> 0006)
[ 3.488970] cxgb4vf 0000:00:06.0: bad SGE FL buffer sizes [2048, 4096]
[ 3.489005] cxgb4vf 0000:00:06.0: unable to use adapter parameters: err=-22
[ 3.489112] cxgb4vf: probe of 0000:00:06.0 failed with error -22


So I cannot make the passed though card to work in the guest. Maybe a linux bug. But this card worked with SR-IOV pci pass through from Linux host to a linux guest. But not if the host is FreeBSD. So therefor I wonder....


Any clues very much appreciated

/Peo
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

Well i finally got an interface up in a VM.
Code:
t4vf0@pci0:0:4:0:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
Code:
root@freebsd2:~ # ifconfig -v cxgbev0
cxgbev0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
    ether 06:44:14:f9:00:01
    inet 192.168.1.134 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet 10Gbase-SR <full-duplex>
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
So on FreeBSD 11.2 PF and VF work on Chelsio.
dmesg
Code:
t4vf0: <Chelsio T420-CR VF> mem 0xc0060000-0xc0060fff,0xc0062000-0xc0063fff at device 4.0 on pci0
cxgbev0: <port 0> on t4vf0
cxgbev0: Ethernet address: 06:44:14:f9:00:01
cxgbev0: 2 txq, 1 rxq (NIC)
t4vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

Host settings:
/etc/rc.conf
Code:
ifconfig_cxgbe0="DHCP"
ifconfig_cxgbe1="DHCP"
iovctl_files=/etc/iovctl.conf
/boot/loader.conf
Code:
t4fw_cfg_load="YES"
if_cxgbe_load="YES"
pptdevs="4/0/1";###<<<The special sauce<< Not the first PCI address(t4iov0) but second(was t4iov1).
/etc/iovctl.conf
Code:
PF {device:"cxgbe0";num_vfs:2;}
VF-0 {passthrough:false;}
VF-1 {passthrough:true;}
Here is pciconf for the new devices.
Code:
none173@pci0:4:0:8:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt5@pci0:4:0:12:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
So for my bhyve script I passthru 4/0/12
The other device is on the host. If I enable if_cxgbev="YES" in loader.conf it crashes the host.
But the device is present in pciconf.
So maybe I will figure that out one day. Maybe an issue having both if_cxgbe and if_cxgbev enabled on the host.
if_cxgbe must be enabled on the host.
.
VM settings:
/etc/rc.conf
Code:
ifconfig_cxgbev0="DHCP"
/boot/loader.conf
Code:
if_cxgbev_load="YES"
On the host:
Once you add the ppt device for the second PCI address(for me 4/0/1) new devices will appear at botton of pciconf -lv
For me the new devices started at 4/0/8 for the Chelsio VF interfaces.
I stumbled in this by accident. I meant to add the first address for t4iov0 which is 4/0/0 but added 4/0/1 by accident.
It was actually a good thing.
Now what was t4iov1 on 4/0/1 is now t4iov1 on 4/0/2.
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

I also noticed a special relationship with a file named /etc/iovctl.conf. It will mess with a iovctl setup.
If I have a /etc/cxgbe0.conf and have a /etc/iovctl.conf iov will fail to work.
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

I increased my VF's to 15 and passthru to bhyve 3 interfaces. 4/0/12, 4/0/16 and 4/0/20
Code:
cxgbev0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
        ether 06:44:14:f9:00:01
        inet 192.168.1.134 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cxgbev1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
        ether 06:44:14:f9:00:02
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cxgbev2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
        ether 06:44:14:f9:00:03
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Need to setup my /etc/rc.conf to get my additional DHCP leases.
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

I next want to setup a /etc/cxgbe1.conf to see if I can get IOV working on both interfaces for a total of 32 VF's.
The /etc/rc.conf setting of [iovctl_files=""] can have multiple conf files.
 
OP
OP
pos

pos

Member

Reaction score: 10
Messages: 43

> The other device is on the host. If I enable if_cxgbev="YES" in loader.conf it crashes the host.
> But the device is present in pciconf.
> So maybe I will figure that out one day. Maybe an issue having both if_cxgbe and if_cxgbev enabled on the host.
> if_cxgbe must be enabled on the host.

I have both loaded. Otherwise you wont se VFs in the host. No crashes


> I next want to setup a /etc/cxgbe1.conf to see if I can get IOV working on both interfaces for a total of 32 VF's.
> The /etc/rc.conf setting of [iovctl_files=""] can have multiple conf files.

I have put my files in /etc/iov/
 

arader

Member

Reaction score: 3
Messages: 23

Phishfry, congrats, it sounds like your setup is mostly working! I wish this wasn't such a mysterious thing, SR-IOV is hardly new functionality...

Am I corrrect in my understanding that for every VF you're creating, you're seeing a new device appear in pciconf?

Also, I'm not sure how, but according to Chelsio's documentation it should be possible to get 128 VFs off of these cards: https://www.chelsio.com/nic/t4-unified-wire-adapters/t420-cr/
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

for every VF you're creating, you're seeing a new device appear in pciconf?
Yes exactly. They will not show up unless I have ppt entry for the 'second' Chelsio device by number.
For example if t4iov0 shows up on your computer as 12/0/0 then you would add a ppt entry for 12/0/1
Then reboot and see the VF's.

So my Chesio t4iov0 interface was 4:0:0, So I need to add 4:0:1 for passthru.
Then pciconf shows the PF's at the very end.
Note I am also passing through a quad gigabit adapter I use for each VM currently.

Code:
root@virt:~ # cat /boot/loader.conf
coretemp_load="YES"
pptdevs="4/0/1 132/0/0 132/0/1 133/0/0 133/0/1"
vmm_load="YES"
nmdm_load="YES"
t4fw_cfg_load="YES"
if_cxgbe_load="YES"
#if_cxgbev_load="YES"
#t4_tom_load="YES"
#hw.cxgbe.num_vis=2
#hw.vmm.force_iommu=1
These devices end up in pciconf after I reboot with ppt settings and iovctl properly configured.
So none@173 here is my passthrough:false VF. I may reroute this to a guest as it does me no good.
cxgbev consistently causes a kernel crash on my host when enabled. Machine does looping reboot. cxgbev works great in a VM.
Code:
none173@pci0:4:0:8:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt5@pci0:4:0:12:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt6@pci0:4:0:16:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt7@pci0:4:0:20:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt8@pci0:4:0:24:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt9@pci0:4:0:28:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt10@pci0:4:0:32:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt11@pci0:4:0:36:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt12@pci0:4:0:40:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt13@pci0:4:0:44:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt14@pci0:4:0:48:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt15@pci0:4:0:52:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt16@pci0:4:0:56:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt17@pci0:4:0:60:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt18@pci0:4:0:64:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt19@pci0:4:0:68:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

This is as far as I get when trying to bring up the cxgbev interface manually on the host:
Code:
root@virt:~ # ifconfig cxgbev0
cxgbev0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 06:44:14:f9:00:00
    hwaddr 06:44:14:f9:00:00
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Fssh_packet_write_wait: Connection to 192.168.1.50 port 22: Broken pipe
Reboots after maybe 60 seconds.
Same if cxgbev added to /boot/loader.conf but requires USB stick to bring it back to life. fsck required for rescue mounting.
 

Phishfry

Son of Beastie

Reaction score: 1,247
Messages: 3,617

I wanted to add that I am using 11.2-STABLE for my host (legacy BIOS) and my VM's are all EFI 12-RELEASE
according to Chelsio's documentation it should be possible to get 128 VFs off of these cards
I wonder if this is marketing or for real. Notice the term they use. 128 Virtual Functions.
Maybe they are counting each filter you use as a function. ie passthrough and mac-addr.

I tried to use my second interface, cxgbe1 as a iov device as well and have failed.
Even if I try and only use cxgbe1 I get no VF's.
root@virt:~ # cat /etc/cxgbe1.conf
PF {device:"cxgbe1";num_vfs:4;}
DEFAULT {passthrough:true;}
Only cxgbe0 is able to attach a node at /dev/iov/
Tried this:
iovctl_files="/etc/cxgbe0.conf etc/cxgbe1.conf"As well as manually:
root@virt:~ # iovctl -f /etc/cxgbe1.conf -C
iovctl: Could not open device '/dev/iov/cxgbe1': No such file or directory

root@virt:~ # iovctl -d cxgbe1 -S
iovctl: Could not open device '/dev/iov/cxgbe1': No such file or directory
What is strange is there are 4 IOV devices in pciconf and I can only configure one for 16 VF, t4iov0.
t4iov0 is seemingly linked to cxgbe0. When I use iovctl it requires an interface node. Not a IOV node.
t4nex0 is the parent interface, but strangely it uses a PCI address of 4:0:4.
Usually a parent device uses the top level PCI address. In this case 4:0:0
If you look through the source there is interesting tidbits in the /usr/src/sys/dev/cxgbe/firmware directory.
t4fw_cfg.txt
Line 100 here alludes to t4nex0 PCI address of x:0:4
# PF4 is the resource-rich PF that the bus/nexus driver attaches to.
# It gets 32 MSI/128 MSI-X vectors.
 
Top