sriov passthru network card driver

Ofloo

Ofloo

I've installed both freebsd 11.2 and 12 both don't seem to support it work out the box

On host system configuration
Code: 
PF {
        device:         ix3;
        num_vfs:        20;
}
DEFAULT {
  passthrough:          true;
  allow-set-mac:        true;
  allow-promisc:        true;
}

Code: 
loader="bhyveload"
cpu=1
memory=256M
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="118457fb-2788-11e9-b711-001b21a2777c"

passthru0="8/0/131"

The passthrough worked however the card doesn't get attached to a driver on the guest system
Code: 
none0@pci0:0:5:0:       class=0x020000 card=0x00008086 chip=0x15c58086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'X553 Virtual Function'
    class      = network
    subclass   = ethernet
 
I am passing through onboard Intel controllers and 2 network cards with 4 ports each both igb and em.
Unsure how vm-bhyve works but I use this:
/boot/loader.conf
Code: 
pptdevs="6/0/0 6/0/1 7/0/0 7/0/1 132/0/0 132/0/1 134/0/0 134/0/1"
vmm_load="YES"
nmdm_load="YES"
bhyveload -m 8G -S -d /vm/freebsd/freebsd1.img freebsd1;
bhyve -S -m 8G -c 16 -AHP -s 0,hostbridge -s 1,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 5:0,passthru,6/0/0 -s30,xhci,tablet -l com1,/dev/nmdm1A freebsd1;
 
It's not a matter of being able to pass them through, there passed through, .. only there not getting any drivers, ..

Host system:
Code: 
ppt0@pci0:8:0:129:      class=0x020000 card=0x00008086 chip=0x15c58086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'X553 Virtual Function'
    class      = network
    subclass   = ethernet

Guest system:

Code: 
none0@pci0:0:5:0:       class=0x020000 card=0x00008086 chip=0x15c58086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'X553 Virtual Function'
    class      = network
    subclass   = ethernet
 
I have only seen SRV-IO on my Chelsio cards. But in my case I could not get them to passthrough right either.
Have you tried passing through the whole device 8/0/0 just to see what it does?
I got my Chelsio working by using a different PCI address other than SRV-IO ones.
 
Intel X553 Gigabit Ethernet Controller Quad device

FreeBSD test 11.2-RELEASE-p4:
Code: 
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:45:bb:3e
        hwaddr ac:1f:6b:45:bb:3e
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier

FreeBSD test3 12.0-RELEASE
Code: 
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:45:bb:3f
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 
I have a feeling that the SRV-IO interfaces are not meant to be assigned and maybe the parent device handles them.
 
I think its a problem with the ix driver, ..

Code: 
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver> at device 0.128 on pci6
ixv0: ...reset_hw() failure: Reset Failed!
ixv0: IFDI_ATTACH_PRE failed 5
device_attach: ixv0 attach returned 5
ppt2 at device 0.130 on pci6
ppt3 at device 0.132 on pci6
ppt4 at device 0.134 on pci6

Going to reboot and see what happens.
 
I can't imagine how they can get 64 Virtual Interfaces from 4 physical ones. There certainly is not enough bandwidth to provide 10G to 64 VM's. So I don't know what this magic does. I assume it is like VALE and it has a virtual switch onboard.

Will retry tomorrow with my Chelsio's now that I see what I need to do. It had the VF's created, but only 4 per interface.
 
It all depends on how bandwidth intensive your vms are, .. and how continuously they use it. you can split a gbic nic over a lot of interfaces, .. think of it as a time share, not everyone needs to be using it all the time, or at the same time.

What about 24port switches with 1gbic uplink.. i mean my internet is 500mbits .. doesn't mean i max it out all the time certainly got more then 64 devices !?

I use vms usually to separate things from each other, .. and to make them more portable, suppose my hardware fails, just setup bhyve host system import the zfs pool done.

And if you backup your configuration with tarsnap like rc.conf probably setup a script with basic install packages, then it should almost be painless to reinstall.
 
Downloaded intel driver had to edit the make file to enable sr-iov copied to /boot/modules named it if_ix_updated.ko, added if_ix_updated_load="YES" to /boot/loader.conf.

So locally it works already.

Code: 
ixv0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 1e:56:13:ac:40:b2
    media: Ethernet autoselect
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

FreeBSD 11.2 guest no support not even with updated driver

FreeBSD 12 guest only updated driver on host not on guest:

Code: 
ixv0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 02:17:08:89:d4:59
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

So what did i do to make it work:

  • Downloaded the latest intel driver https://downloadmirror.intel.com/22283/eng/23_5_1.zip
  • Compiled the intel driver with SRIOV_ENABLE = 1
  • Copied the driver if_ix.ko to /boot/modules
  • renamed it to if_ix_updated.ko
  • sysrc -f /boot/loader.conf if_ix_updated_load=yes
  • rebooted and done
Only works for FreeBSD12 guest though, need a find a way to make it work for FreeBSD11.2

At least the driver work, .. network connectivity is untested.

edit: haven't been able to route traffic over them yet.
 
I was able to solve the errors and make it function but make it functional yet. Got distracted but I'll soon look into that again. From what i can tell it's a driver issue. It still might be a configuration issue, .. since now the interface works without errors but it's not really passing through.
 
I can see the code in Intel:s ix-3.3.6 package seems to be broken. I both have the problem you have plus I want transparent vlan support. The transparent vlan feature seems to be partly added to the code. I.e not finished....
 
It was defiantly broke just like you guys said. Everything was OK except inside VM. Trying the newest driver now.
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver> mem 0xc0004000-0xc0007fff,0xc0008000-0xc000bfff at device 3.0 on pci0
ixv0: ...reset_hw() failure: Reset Failed!
ixv0: IFDI_ATTACH_PRE failed 5
device_attach: ixv0 attach returned 5
Click to expand...
 
Hi, I was running around this Problem since long time, but the solution is very simple. Just add

hw.pci.honor_msi_blacklist=0

to /boot/loader.conf -> reboot-> issue fixed ! A new ixv0 interface is up and running from now ...

Be aware, some other issues will raise after SR-IOV is running on ESXi with FreeBSD 12.

#1 > Make sure the Management Network is running on a independet Network Port without SR-IOV enabled!
#2 > Sometimes the ixv0 interface starts at boot, the entire SR-IOV Network Stack is hanging, the ESXi Host need a reboot to fix this! I try to find a solution for this at the moment and it looks like, that LRO and TSO needs to be disabled at earliest stage. If any find best solution for stable enviroment, just let the community know. I will do as well ...

Just to be clear: This Solution works with standard installation of FreeBSD 12.0 without any other 3rd party driver.
 
You must log in or register to reply here.
Back
Top