Mellanox MCX311A running at 10GbE.

Phishfry

Phishfry

Well I picked up a MCX311A card because I need a 4x PCIe 10-Gigabit Ethernet card for my firewall.
The card is so small it is amazing. Half the height of a Low Profile bracket. One SFP+ socket.
They are selling for $27 bucks used on ebay so I bought one.
I was happy to find out that it works fine at 10GbE with no kernel modifications necessary.

/etc/rc.conf
ifconfig_mlxen0="DHCP"

/boot/loader.conf
mlx4en_load="YES"

Code: 
# ifconfig -v mlxen0
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:02:c9:a1:d0:80
    hwaddr 00:02:c9:a1:d0:80
    inet 192.168.1.122 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active

This is using a Finnstar SFP+ module to a 2m fiber LC-LC om3 dualmode cable connected to my Aruba switch.
I am using the Aruba 10GbE ports as normal DHCP ports instead of uplinks for now. I have a Netapp 10G switch but need to learn it..
Will report back in the future with some speeds between FreeBSD 10GbE devices like Chelsio T420 and Intel X520-DA2.
 
Last edited:
Always fun to see mention of my home country in geeky hardware specs. AMD uses a device core codename Aruba for one of their graphics cards series as well. I ought to use one of their switches on the actual island, hehehe
Any particular reason you opted for the much more expensive (albeit much muuuch cheaper on eBay) ConnectX-3 cards?
Surely you meant to say Finisar SFP+
 
I wanted PCIe 3 bus. Max blast for a PCIe 3.0 4x slot. I have some machines with only 4X PCIe slot and one 10GBe is enough.
Yes those sound like the SFP modules. Amazing little lasers.
 
Can any fiber cable experts tell me why short cables are not more prevalent?
Most are =>1Meter. I am so fed up I am considering trying to make my own.
I want a tidy rack.
Maybe short runs most people use twinax direct attach instead?
I have gear stacked right net to one another and need shorty multimode LC OC3 10G patch cables.<=.3M
 
Not an expert, but I'd imagine because fiber was designed for long(er) haul connections. With < 1m length, you could (should?) use 10g copper. That being said, you may want to make sure you're not running too hot on your connections running a risk of burning out your SPFs.
 
Phishfry said:
Well I picked up a MCX311A card because I need a 4x PCIe 10-Gigabit Ethernet card for my firewall.
The card is so small it is amazing. Half the height of a Low Profile bracket. One SFP+ socket.
They are selling for $27 bucks used on ebay so I bought one.
I was happy to find out that it works fine at 10GbE with no kernel modifications necessary.

/etc/rc.conf
ifconfig_mlxen0="DHCP"

/boot/loader.conf
mlx4en_load="YES"

Code: 
# ifconfig -v mlxen0
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:02:c9:a1:d0:80
    hwaddr 00:02:c9:a1:d0:80
    inet 192.168.1.122 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active

This is using a Finnstar SFP+ module to a 2m fiber LC-LC om3 dualmode cable connected to my Aruba switch.
I am using the Aruba 10GbE ports as normal DHCP ports instead of uplinks for now. I have a Netapp 10G switch but need to learn it..
Will report back in the future with some speeds between FreeBSD 10GbE devices like Chelsio T420 and Intel X520-DA2.
Click to expand...
What transfer speed did you measure with it? I can do only 5-6 Gbps with iperf3, no matter how I try. I have a DAC cable between the computers. It is weird, because it seems to depend on the CPU of the sending computer a lot. My weaker computer can send only 4.7Gbps my stronger something around 5.8. The CPU is 3.4 GHz, I wonder if that is not enough for 10Gbps data transfer or something else is wrong with these systems. Another idea that I bought fake cards, but I would not expect fake cards to work with the Mellanox driver at 5-6 Gbps speeds. So it is weird, but I have no experience with high speed ethernet to know for sure...
 
I doubt you got fake cards. I do not remember the exact speeds I achieved. Somewhere around 8500 Mbps if I remember right.
I was using a single socket LGA2011 board with a V3 cpu for testing..
 
Phishfry said:
I doubt you got fake cards. I do not remember the exact speeds I achieved. Somewhere around 8500 Mbps if I remember right.
I was using a single socket LGA2011 board with a V3 cpu for testing..
Click to expand...
Thanks! It turned out that one of the motherboards has only x2 pcie2 lanes in the second x16 slot, which limits it speed to the theoretical 8Gbps. I guess in practice that means 6Gbps and that's why I don't have higher speeds. I'll buy a new motherboard and we'll see. In theory these cards can do 40Gbps too with a modified firmware.
 
Phishfry said:
I doubt you got fake cards. I do not remember the exact speeds I achieved. Somewhere around 8500 Mbps if I remember right.
I was using a single socket LGA2011 board with a V3 cpu for testing..
Click to expand...
I bought a new motherboard meanwhile with 4x pcie3 lanes. Now I have 9.55Gbps with iperf3. With FTP it is only about 2.5Gbps, but certainly not the Mellanox card is the bottleneck.
 
You must log in or register to reply here.
Back
Top