Intel 82599 and 1 GbE RJ45 SFP modules

Tsuroerusu

Tsuroerusu

I am trying to use an Intel 82599-based NIC (Supermicro's AOC-STGN-i2S) with StarTech's SFP1000TXST RJ45 SFP module.
If I do "ifconfig -m ix0" when the module is plugged in, it reports:

Code: 
        supported media:
                media autoselect
                media 1000baseT

Which seems logical given that, that is the speed of the module. My switch also indicates that the connection is at 1 GbE in speed.

However, no matter what I do, I keep getting "no carrier". Whether I configure it using rc.conf or use ifconfig manually, it keeps giving me "no carrier" and no pings get through to the gateway.

The strange thing is that, when I look in my dmesg, I do not get the "unsupported sfp" message. On their website, Intel states at least one MSA-compliant RJ45 SFP module from Avago, and this one should be similar to that, which squares with the ix driver not saying "unsupported sfp".

Also, in the specs for the controller on Intel's website, they state that the port speed can be either 10 GbE or 1 GbE.

The ix driver has been driving me up the fall in other contexts, but this is just bizarre to me. Does anybody have any ideas if this can be made to work?
 
I had trouble getting some legacy 1GB gear interfaced with 10G gear. I think adding 'SFP to RJ45' adds trouble.
What you found with Intel board is common for them. I advise you use Intel branded SFP modules.
Even used they work better than clones. Some boards drivers check for SFP blob some don't.
The real problem you are facing is using 1GbE on a 10GbE network. I had to buy Cisco 1 GbE SFP modules for my SG300 and I found Finisar 1GbE SFP modules for my NetApp fiber switch. Note: Not SFP+ modules.
MGBSX1 << Cisco 1000-baseSX SFP module for my SG300 switch. They check for a blob.
FTLF8519P2BNL << Finisar modules seem to be very compatible on switches.

10GbE may be backwards compatible but I could not get 1GbE working over 10GbE. When I used 1 GbE modules on both ends it works. I use LC-LC OM3 multimode duplex cabling just like 10G cable.

One thing I did notice is some 10G drivers (chelsio) give more SFP details than others WRT ifconfig -vv and SFP details.

Does your downstrream switch not offer any SFP ports?
 
Phishfry said:
What you found with Intel board is common for them. I advise you use Intel branded SFP modules.
Even used they work better than clones. Some boards drivers check for SFP blob some don't.
Click to expand...
Judging by what Intel wrote on their website, I do not think they actually make RJ45 SFP modules themselves:

They mention two RJ45 modules that are, supposedly, tested to work with the 82599. Those being Avago ABCU-5710RZ and Finisar FCLF8522P2BTL. From what I could gather from looking into those, they are not Intel-branded, and at least the Avago one is an MSA-compliant module. But yeah, it would be interesting to test using one of those two, however at this point, having had to deal with the ix driver, my gut tells me that since these StarTech modules are MSA-compliant, and the driver does not say "unsupported sfp", but it still goes "no carrier", I think the result for those would probably be the same.

This is the page I was looking at:
www.intel.com

SFP+ Modules and Cables Compatible with Intel® Ethernet Converged...

SFP+ optical module, SFP module, and direct attach cable requirements for the Intel® Ethernet Converged Network Adapter X520 Series.
www.intel.com www.intel.com


Phishfry said:
Does your downstrream switch not offer any SFP ports?
Click to expand...
It does have 4 x 10 GbE SFP+ ports, but those are already in use for other purposes, and that was why I was hoping to be able to use these RJ45 SFP modules, because I do not actually need the 10 GbE speed for the purpose of these Intel NICs (Whereas I do need it for what it already does).

Basically, I originally had two of these Intel 82599 cards in my OPNsense firewalls, but since August 2018 they do not work with VLANs because Intel screwed up the driver, so that it goes "no carrier" if you create VLANs. So I installed Mellanox ConnectX-3 NICs in my firewalls, and then I was hoping to be able to simply re-use these Intel NICs for other stuff. One feature they have going for them is their ability to supply 126 virtual functions for SR-IOV, and that is useful to me.

At this point, I am tempted to just get a smaller SKU of the switch I use, and just hook-up these Intel NICs via DAC cables. That will probably be more economical as well given that if the Avago and Finisar modules turned out to yield the same result, I would have more door-stops than I already do, lol.
 
Tsuroerusu said:
So I installed Mellanox ConnectX-3 NICs in my firewalls,
Click to expand...
I use them too. The single port cards are 4X PCIe which worked out good for some client machines with limited IO.
Finisar on those work good.
Have not tried the Mellanox SR-IOV features yet. Using Chelsio T420/422 cards with SR-IOV and bhyve.
Tsuroerusu said:
and just hook-up these Intel NICs via DAC cables
Click to expand...
I had more trouble with DAC cables than anything. They are really brand specific.
Transciever is built-in so really best for connecting 2 pieces of equipment of the same brand.
I have all different brand gear and they were a waste of my money.

Here are the Intel branded SFP+ modules that I use with my X520 boards.
https://www.ebay.com/itm/113699659049
 
Phishfry said:
I use them too. The single port cards are 4X PCIe which worked out good for some client machines with limited IO.
Finisar on those work good.
Have not tried the Mellanox SR-IOV features yet. Using Chelsio T420/422 cards with SR-IOV and bhyve.
Click to expand...
Thanks for the tip, I was not aware that they had a single-port card. The Mellanox ConnectX-3 seem to, unfortuantely, only be able to do 8 VFs per port, and I need a good deal more than that for my purpose, where I need a VF per customer (And I need VF capacity to handle at least 100). I am hoping that once I find an extra switch that I can get the 82599 cards to work, because they can do 63 VFs per port.


Phishfry said:
I had more trouble with DAC cables than anything. They are really brand specific.
Transciever is built-in so really best for connecting 2 pieces of equipment of the same brand.
I have all different brand gear and they were a waste of my money.

Here are the Intel branded SFP+ modules that I use with my X520 boards.
https://www.ebay.com/itm/113699659049
Click to expand...
I am generally a Supermicro guy, and so far their DAC cables have worked great for me with the switch that I use. They work perfectly too going from the Mellanox CX3 cards in my firewalls to the switch. Amusingly, my switch detected that the cables were really Ampherol and Foxconn cables, so it seems quite generic in Supermicro's case. I appreciate your advice on matching brands, which is probably what I will generally be doing in the future to avoid wasting more money.
 
You must log in or register to reply here.
Back
Top