Solved SR-IOV Issues/Recommendations

Deviant0ne · 2025-01-16T20:58:16+0000

I've been wrestling with a Mellanox ConnectX-3 trying to make SR-IOV work on FreeBSD. I've completed the (unnecessarily difficult) process to create a custom firmware configuration with SR-IOV enabled, burning the configuration/firmware to the card, etc. However, when the card is reinstalled in a FreeBSD host, the /dev/iov directory remains unpopulated.

I've also hit a roadblock with an Intel X520-DA2 where upon enabling SR-IOV via iovctl, the card stops passing all network traffic until the host is rebooted.

I've resigned myself to the fact that purchasing a different 10Gbe adapter will be necessary, but I am having difficulty deciding between two. I've basically ruled out the Intel X710 due to higher cost, negative comments on the forums, etc. The cards I am interested in are the Mellanox ConnectX-4 Lx and the Chelsio T520-CR.

I would appreciate any insight as to which adapter would offer a better overall experience, or any tips on getting SR-IOV working on my existing adapters.

Phishfry · 2025-01-16T21:12:45+0000

Chelsio all the way.
In fact I will buy the card back from you if you can't get it working.
I am that confident. (don't go paying $1K and expect me to honor that)

Dell Chelsio T520 0JF0FV 0VV004 10Gb Dual Port SFP+ Network Adapter Low Profile | eBay

Up for Sale is a genuine Dell Chelsio T520 0JF0FV 0VV004 10Gb Dual Port SFP+ Network Adapter Low Profile unit. This Dell Chelsio T520 0JF0FV 0VV004 10Gb Dual Port SFP+ Network Adapter Low Profile is in good condition and ready to be redeployed.

www.ebay.com

Low Profile or Full Height bracket?

I just ugraded to used 25GB cards and recommend it if you can swing it. T580.
Cisco Nexus 25GB Switch for $125

I can't help you with other cards because they are broke. HPS worked at Mellanox and passed away.
So that is not good for thier driver. Intel IOV never worked for me on X710 or X550.

Deviant0ne · 2025-01-16T21:28:56+0000

Thank you for the confirmation, those are the exact cards I was looking at (the Dell variant).

I will be purchasing a full height adapter. As for 25Gbe, that's a good thought. I don't have a 25Gbe-capable switch but "future-proofing" the FreeBSD host to the best of my ability isn't a bad idea.

I've seen rumblings that the Chelsio drivers can problematic, but I am assuming this doesn't apply to FreeBSD given the amount of times I've seen those adapters referenced. I plan to use the information outlined in cxgbe(4) to get the card working. I've also come across some tuning information that I'll likely implement once the card is installed.

Phishfry · 2025-01-16T21:56:43+0000

There is no need for tuning off the bat.

You just need to remember that the interface name is different inside a VM.
Instead of cxl of host it is vcxl. You will need to add that to your VM's rc.conf.

You need to make a control file on host for each interface you want to use IOV on.
Manual says /etc/iov/ directory but it can be in /etc/ I have found. You must point to it in rc.conf anyway.

Code:

       iovctl_files
           (str) A space-separated list    of configuration files used by
           iovctl(8).  The default value is an empty string.

Phishfry · 2025-01-16T22:00:40+0000

There is a nice glossy writeup here for the Intel X710.

| FreeBSD Foundation

SR-IOV is a First Class FreeBSD Feature SR-IOV is a First Class FreeBSD Feature A detailed walkthrough of how to setup hardware-driven virtualization using SR-IOV capable devices in FreeBSD. By Mark McBride One of my favorite hardware features is called Single-Root Input/Output Virtualization...

freebsdfoundation.org

Personally I have not been sucessful.

But I will say the article mentions enabling SR-IOV in BIOS and I think that is a good point. You need to use a server board.
Also the chelsio needs airflow so you need to think about a fan unless in server chassis.

Deviant0ne · 2025-01-16T22:06:57+0000

I've read that article, which is why I started down the X710 route in the first place. I think I will just stick with the Chelsio recommendation to be safe, I don't want another headache.

The adapter will be installed in a 3U rack-mounted chassis with 2x 120MM Noctua fans pushing air across the add-in cards. I am planning to add 2x additional 60MM fans at the back of the case for added exhaust performance, but I'm thinking the existing fans should be sufficient for now.

Thank you for the resources and your input.

sko · 2025-01-17T08:54:04+0000

I've also tested with SR-IOV when I added the feature to pass vnet interfaces to jails to sysutils/iocell (not present in the port, this and other pull requests are pending since half a year - I'm preparing a new port/fork).

With Intel X5xx I could never get SR-IOV to work - as soon as a VF is passed to a jail, the whole NIC goes dark and needs a complete reset (i.e. reboot) to come back to life. This was regardless of any BIOS-settings an tested on at least 2 different Supermicro boards and one Atom C3758 appliance. If not enabled in BIOS, you can't even create VFs, so that part seems to work, but I suspect there's something broken in the firmware and/or how the FreeBSD implementation interacts with it when passing the VF to a jail (or VM).

With Mellanox CX3 the VFs were sucessfully passed to the jail, I could attach an address and send/receive traffic. I never fiddled around with firmware on those cards - AFAIK that's only a linux thing where you need to change the mode (ethernet/infiniband) in the firmware because their driver still isn't capable of doing that. On FreeBSD you just load the appropriate driver (mlx4en(4) or mlx4ib(4)) and it 'just works'™
Apart from some testing during the implementation in iocell and setting up an experimental gateway with VFs instead of epairs, I never ran SR-IOV/vnet jails for longer than a few days/weeks. For the sake of easy migration/recovery I keep jail configuration consistent across all hosts and hardware-agnostic - i.e. everything is connected to bridges which are identically named across hosts (e.g. br-dmz, br-wan, br-mgmt, etc...).

Although I do have 6 hosts at hand with CX3 (40G) NICs, all of them are production hosts so I can't really test with stuff that might interrupt anything. However, I also have a Cx3 in my server at home with which I could perform some more testing and another single-port CX3 for my desktop machine, but I have yet to find the full-size pci bracket which I have "put somewhere where it won't get lost" to be able to install it...

Regarding the 25G Chelsios: Phishfry could you point me in the right direction as to where they (T6225-CR?) can be found for less than ~200-300EUR (+ shipping and import into the EU, because they don't seem to be common at all here...)? There don't seem to be any 'white-box' variants of chelsios beyond the T520...
Given that one can easily find CX4s (which can be easily cross-flashed between the 10G and 25G variant) for 25-30EUR and sometimes even cheaper, the chelsio doesn't look very "bargain-y" at 10x that price point.
Especially if one counts in the fact that 25G capable ("proper") switches are either still relatively expensive or very power hungry. I've had an eye on the Nexus N9K-C92160YC-X for a while, as this seems to be the cheapest option to go beyond 10G that doesn't consume several hundred watts - but they are still ~800-1000eur (with fans/PSUs) here in germany - so what 25G capable nexus can be found for only 125$??[/port][/port]

EDIT:
I just wanted to re-check on my home server and found that /dev/iov now only contains the ix entries of the X540 interfaces on the riser. The mlxen interfaces are gone... same goes for the hosts with the 40G CX3 I just checked. So SR-IOV *does* seem to be broken now for those cards?
The commit for the vnet.interface feature in iocell is from july last year - so all my testing was then on 13.3-RELEASE, that regression must (might) have been introduced with 13.4-RELEASE or some patch or driver update since ~7/2024...

Deviant0ne · 2025-01-17T17:23:49+0000

sko said:
...

EDIT:
I just wanted to re-check on my home server and found that /dev/iov now only contains the ix entries of the X540 interfaces on the riser. The mlxen interfaces are gone... same goes for the hosts with the 40G CX3 I just checked. So SR-IOV *does* seem to be broken now for those cards?
The commit for the vnet.interface feature in iocell is from july last year - so all my testing was then on 13.3-RELEASE, that regression must (might) have been introduced with 13.4-RELEASE or some patch or driver update since ~7/2024...

Thank you for confirming that I haven't missed something in my struggles with the CX3. I'm still curious about the CX4's, as the overall cost is lower (and I've read they consume less power) but I think I will still move forward with purchasing a Chelsio, per Phishfry's suggestion.