general/other SR-IOV

PMc

PMc

Today I found this great article:

| FreeBSD Foundation

SR-IOV is a First Class FreeBSD Feature SR-IOV is a First Class FreeBSD Feature A detailed walkthrough of how to setup hardware-driven virtualization using SR-IOV capable devices in FreeBSD. By Mark McBride One of my favorite hardware features is called Single-Root Input/Output Virtualization...
freebsdfoundation.org freebsdfoundation.org

I had not yet bothered about what this SR-IOV exactly is, I only had recognized: my mainboard can do it. So I had a closer look,
and found it interesting.
Then I noticed with a bit of surprise: my 2-dollar-on-ebay network card can also do it. So I decided to give it a try.

I switched in on in the BIOS. It didn't work.

I checked my custom kernel config. It was disabled. Okay, recompile the kernel. But it didn't work.

Then I figured: ups, the network card is actually seated in the PCH slot (which may not work). So I disassembled the machine (that is difficult: 21 disks, 11 fans in a 486-high-tower-case).
And, still, it didn't work.

Then I started searching online, and found the problem: FreeBSD cannot do it. Linux would.


Thats why I hate people like Facebook, Google, Supermarkets, the Foundation, and similar: they spill the web with (literally)
misleading advertisment stuff (that specifically doesn’t tell you what you need to know), give a damn about the people they influence, and don't care if things do (partially) NOT work.

A manpage in contrast would normally tell precisely what should work and also, what not.
 
forums.freebsd.org

Enabling SR-IOV on intel driver

This is a new server I just installed. It has around 10 bhyve vm:s on it right now. I use latest Intel ix driver from intel-ix-kmod-3.3.6_1 package... As the network performance went down on the VMs when leaving KVM om Linux due to not optimal bridge code in FreeBSD I want to use SR-IOV on the...
forums.freebsd.org forums.freebsd.org
 
PMc said:
This is an 82576
Click to expand...
I think that's the problem. I have only heard of SR-IOV support for i350. I think it is the only 1GBe chipset that offers SR-IOV.
Generally SR-IOV only went into 10GBe drivers. Initially i350 did not work but support was added later. Mailing list had the details.

If you do pciconf -lvbce em0 you will see if it has SR-IOV support at driver level.

I would also say SR-IOV does not work well on Intel cards at all. I am a Chelsio fanboy so biased.
Many datacenters use them so when they upgrade the market floods.
www.ebay.com

Chelsio Communications MSIP-REM-CC2-T580-CR Dual Port Ethernet Adapter | eBay

Find many great new & used options and get the best deals for Chelsio Communications MSIP-REM-CC2-T580-CR Dual Port Ethernet Adapter at the best online prices at eBay! Free shipping for many products!
www.ebay.com
 
Phishfry said:
I would also say SR-IOV does not work well on Intel cards at all.
Click to expand...
This.
X710 works, but I haven't got any X5x0 to work. You can create VFs, but then the whole port of the NIC goes silent.
Mellanox OTOH (tried with CX3 and 4) are working as expected and are also available dirt-cheap as 10G or 40G (and 25/50 are also getting cheaper and cheaper...)
 
VladiBG said:
forums.freebsd.org

Enabling SR-IOV on intel driver

This is a new server I just installed. It has around 10 bhyve vm:s on it right now. I use latest Intel ix driver from intel-ix-kmod-3.3.6_1 package... As the network performance went down on the VMs when leaving KVM om Linux due to not optimal bridge code in FreeBSD I want to use SR-IOV on the...
forums.freebsd.org forums.freebsd.org
Click to expand...
I read it.
Phishfry said:
[82576]
I think that's the problem. I have only heard of SR-IOV support for i350. I think it is the only 1GBe chipset that offers SR-IOV.
Click to expand...
That is not the problem. See this discussion:
http://www.intel.com/content/dam/doc/design-guide/82576-sr-iov-driver-companion-guide.pdf
And also:

82576 vs i350

Hi all. What difference of this chips? I350 is a new product, but it have only 8 queues, why? I need ethernet card for virtualization. What is better to buy?
community.intel.com

The problem is annoying tabloid headlines like

SR-IOV is a First Class FreeBSD Feature​

The problem is people who care only for money, and consequentially (implicit or explicit) cater for the rich and despise the poor.

SR-IOV is apparently a delicate feature: it does not "just work", it needs a lot of implementation in the device driver. etc. Somebody needs to do that work. For the igb driver it seems nobody has done it. And that is alright.

But it has been done for some of the 10G drivers. And this is not as easy and timeless as Julian Elischer's netgraph which continues to "just work". It is something that will continuously evolve and expand with new hardware arriving. So you get into a treadmill, where you have to continuously invest into further development work. And you have to obtain the money to pay these developers.
And so you have to place advertisements, like this article. The article itself is fine, it's excellent. But the subscript, the bias is, hey we can do it, we can do it as good as linux, and we do it more structured. The subscript is to get a piece of the cake.

This feature is mainly of interest for people running really big servers. People running a lot more compute than they could use by themselves. Rich people ("data is tie oil of the 21st century").
And so we get into a cycle where more and more development is done specifically for those who live above the clouds (and the new intesified release-cycle goes along that line), and we become the foot-soldiers of a machinery, we are kindly allowed to use a product that is no longer ours.

Phishfry said:
Generally SR-IOV only went into 10GBe drivers. Initially i350 did not work but support was added later.
Click to expand...
Doesn't look like so:

i350 sr-iov freebsd

Dear Customer Support, I'd like to enable SR-IOV on a NIC with i350-T4, under FreeBSD 13.0. lspci -lvc shows: [...] ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 0 VFs configured out of 8 supported First VF RID Offset 0x0180, VF RID Stride 0x0004 VF Device ID 0x1520...
community.intel.com
forums.freebsd.org

Intel I350-T4 SR-IOV support

I am presently running 13.2 release with an Intel I350-T4 1GB NIC. It is working fine as intended. I would now like to utilize the SR-IOV function so that it can be passed through to jails. Here is the contents from /etc/iov/igb3.conf: PF { device: igb3; num_vfs: 4; } DEFAULT {...
forums.freebsd.org forums.freebsd.org
I can find only discussions concerning FreeBSD guests under Linux hosts running 82576 or i350.

Phishfry said:
If you do pciconf -lvbce em0 you will see if it has SR-IOV support at driver level.
Click to expand...
Voila:
Code: 
# pciconf -lvbce igb0
igb0@pci0:1:0:0:        class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10c9 subvendor=0x8086 subdevice=0xa04c
    vendor     = 'Intel Corporation'
    device     = '82576 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xc5420000, size 131072, enabled
    bar   [14] = type Memory, range 32, base 0xc5000000, size 4194304, enabled
    bar   [18] = type I/O Port, range 32, base 0xa020, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xc54c4000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks 
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR NS
                 max read 512
                 link x4(x4) speed 2.5(2.5) ASPM L1(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 90e2baffff7e80ac
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 8 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0002
                     VF Device ID 0x10ca
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error

The device can do it. But there is no mention in verbose boot that a driver would even try to attach. Because the driver does simply not exist. If it were just broken or flawed, I could try and start fixing it. But writing the whole thing from scratch, that is a different beast.
 
I can't find the mailing list post where somebody made the case to Jeffery Piper @intel (driver maintainer) why anybody would want to split up a 1GBe connection. But they did make the case and he acquiesced and the i350 was added.

The post was from before our mailing list change and I can't find the post. Probably in freebsd-net.
.
 
Phishfry said:
somebody made the case to Jeffery Piper @intel (driver maintainer) why anybody would want to split up a 1GBe connection. But they did make the case and he acquiesced and the i350 was added.
Click to expand...
It is a bit difficult to believe. If something was added, there should be a trace in the codebase.
And I dont find it. One might assume some of the functions similar as in ixgbe_sriov.h should exist somewere. They are obviousely there for ixl, cxgbe and qlnxe, but not much else.
Also, there is no specific I350. There is only code for 82575, on which both 82576 and I350 are based.

Are you sure that discussion was in FreeBSD?

Phishfry said:
The post was from before our mailing list change and I can't find the post. Probably in freebsd-net..
Click to expand...
You mean this one?
That guy actually got an offer from Kevin Bowling to get it exclusively implemented(!) - and he dismissed it because he couldnt test. :(

So this is the other side if the game - there is not only those above the clouds, but also those in the basement unwilling/unable to hack or even test. And both sides reinforce each other, as cause and effect.

sko said:
And here we go again... Yes, it's all a conspiracy :rolleyes:
Click to expand...

Great! You'r the rich guy. Obviousely, in a holographic universe, cause and effect must be a conspiracy.
 
tingo said:
That thread is from 2017.
Click to expand...
What exactly is not up to date in the linked message? It states the situation as it is. BTW, I was very happy to find that message after spending all night with screwdriver operations. So, what do you want to say?

tingo said:
Try to find as better and more up to date strawman, please?
Click to expand...
I don't know what you want. Can't you do that yourself?
 
Erichans said:
Which Intel® Ethernet Adapters and Intel® Ethernet Controllers support SR-IOV?
No mention of the 82576; that in and of itself could be for a multitude of reasons.
Click to expand...
Main reason would be that at the time of the discussions that model was no longer on sale.
I already linked the relevant things above:
PMc said:

82576 vs i350

Hi all. What difference of this chips? I350 is a new product, but it have only 8 queues, why? I need ethernet card for virtualization. What is better to buy?
community.intel.com
Click to expand...

Also this personal offer wouldn't make any sense if it would not potentially work (this was specifically 82576):

So, conclusion: it can be done, the implementation might be somewhat rudimentary, the usefulness is questionable, and when a developer was willing to code it, nobody did line up to test.

Erichans said:
One of them could be that since that 2009 http://www.intel.com/content/dam/doc/design-guide/82576-sr-iov-driver-companion-guide.pdf document SR-IOV things have progressed.
Click to expand...
So what? Lots of things do progress, all the time. And I do not say that this would still make sense to implement - now certainly not anymore. Nevertheless, one should be clear to state that (in contrast to Linux) we support SR-IOV only for 10+ Gb/s devices.
 
PMc said:
The problem is annoying tabloid headlines like

SR-IOV is a First Class FreeBSD Feature​

The problem is people who care only for money, and consequentially (implicit or explicit) cater for the rich and despise the poor.
Click to expand...

Hey. I'm the author of the linked article. Sorry you didn't like my title, it simply reflects my experience as compared to Linux. I can assure you no money or influence from the FreeBSD Foundation was involved. The article was already posted on my personal site and I was asked if I'd be up for including it the Feb '24 Journal as the focus would be on networking. Obviously, I accepted.

It's part of a series of writings on why I moved from Linux to FreeBSD:
markmcb.com

FreeBSD vs. Linux: Summary - Mark McBride

MarkMcB.com - Personal Site of Mark McBride
markmcb.com markmcb.com
 
You must log in or register to reply here.
Back
Top