Mellanox Technologies MT26448 10GB interface driver problem

G

gnoma

Hello,

I recently buy the following interface:

Code: 
none2@pci0:3:0:0:       class=0x020000 card=0x001515b3 chip=0x675015b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
    class      = network
    subclass   = ethernet

I have installed this driver /usr/src/sys/ofed/drivers/net/mlx4
And after loading it, the pciconf -l command output changed as follows:
Code: 
mlx4_core0@pci0:3:0:0:  class=0x020000 card=0x001515b3 chip=0x675015b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
    class      = network
    subclass   = ethernet
Now the device is shown as mlx4_core0, also lots of kernel parameters appeared in the sysctl -a output.

However I couldn't find it with ifconfig as an interface. The sysctl parameter is set to eth mode instead of ib.
Code: 
sys.device.mlx4_core0.mlx4_port1: eth
I compiled the ibcore module, installed it and loaded it, tried to switch to ib mode, but I got error:
Code: 
root@sentinel:/usr/src/sys/ofed/drivers/net/mlx4 # sysctl sys.device.mlx4_core0.mlx4_port1=ib
sys.device.mlx4_core0.mlx4_port1: eth
sysctl: sys.device.mlx4_core0.mlx4_port1=ib: Invalid argument
Also the strangest thing is that I don't see link LED blinking on the interface, I have changed twice the cables but got no change. However, even with unplugged cable, the interface should be visible as disconnected.

Anybody have an idea how to troubleshoot this?
Thank you.

EDIT:
Code: 
unroot@sentinel:~ # uname -a
FreeBSD hostname.com 10.0-RELEASE FreeBSD 10.0-RELEASE #0: Fri Jun 27 19:46:08 EEST 2014     root@hostname.com:/usr/obj/usr/src/sys/PFKERNEL  amd64
PFKERNEL is GENERIC kernel with added some ALTQ options, nothing related to hardware and drivers was changed.
 
Re: Mellanox Technologies MT26448 10GB interface driver prob

Try adding one of these to your kernel configuration and rebuild the kernel:
Code: 
device          mlx4ib          # Mellanox ConnectX HCA InfiniBand
device          mlxen           # Mellanox ConnectX HCA Ethernet
device          mthca           # Mellanox HCA InfiniBand
I have a feeling you only built half of the driver.
 
Re: Mellanox Technologies MT26448 10GB interface driver prob

Hello,

Thank you for the help.

I did what you suggest few days ago and the driver works, I got my interface shown now. But there is another hoe you can step on. After building the driver and rebooting to the new kernel, the interface was shown in status "No carrier", I was wondering 2 days why my link is down and all the documentation says that in infiniband mode you need to start a subnet manager to bring your links up. However my cards doesn't even support IB and they are both in ethernet mode. The other side of the cable is Windows 2008 server which also shows "Link down". ibstat on my FreeBSD system also shows "link down". After 2 days thinking and reading documentation I tried just for the test to type
Code: 
 ifconfig mlxen0 up
and suddenly the link came up. Probably this is some strange way the interface and the driver works.

Connection is working fine in Ethernet mode, but now I am facing another issue with iSCSI. Tried it with ctld() and with istgt and I hit the same issue:
Code: 
Aug 24 04:51:13 sentinel kernel: WARNING: 10.0.80.2 (iqn.1991-05.com.unixhomenet:hola-pc): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 24 04:51:44 sentinel kernel: cfiscsi_ioctl_handoff: new connection from iqn.1991-05.com.unixhomenet:hola-pc (10.0.80.2) to iqn.sentinel.deltanews.lan:zfs-mirror
Aug 24 04:51:53 sentinel kernel: WARNING: 10.0.80.2 (iqn.1991-05.com.unixhomenet:hola-pc): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 24 04:51:56 sentinel kernel: cfiscsi_ioctl_handoff: new connection from iqn.1991-05.com.unixhomenet:hola-pc (10.0.80.2) to iqn.sentinel.deltanews.lan:zfs-mirror
Aug 24 04:52:07 sentinel kernel: WARNING: 10.0.80.2 (iqn.1991-05.com.unixhomenet:hola-pc): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 24 04:52:34 sentinel kernel: cfiscsi_ioctl_handoff: new connection from iqn.1991-05.com.unixhomenet:hola-pc (10.0.80.2) to iqn.sentinel.deltanews.lan:zfs-mirror
Aug 24 04:52:39 sentinel kernel: WARNING: 10.0.80.2 (iqn.1991-05.com.unixhomenet:hola-pc): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 24 04:53:34 sentinel kernel: cfiscsi_ioctl_handoff: new connection from iqn.1991-05.com.unixhomenet:hola-pc (10.0.80.2) to iqn.sentinel.deltanews.lan:zfs-mirror
Aug 24 04:53:39 sentinel kernel: WARNING: 10.0.80.2 (iqn.1991-05.com.unixhomenet:hola-pc): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 24 04:54:33 sentinel kernel: cfiscsi_ioctl_handoff: new connection from iqn.1991-05.com.unixhomenet:hola-pc (10.0.80.2) to iqn.sentinel.deltanews.lan:zfs-mirror
Aug 24 04:54:39 sentinel kernel: WARNING: 10.0.80.2 (iqn.1991-05.com.unixhomenet:hola-pc): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 24 04:55:33 sentinel kernel: cfiscsi_ioctl_handoff: new connection from iqn.1991-05.com.unixhomenet:hola-pc (10.0.80.2) to iqn.sentinel.deltanews.lan:zfs-mirror

Switching the iSCSI to a normal 1GB interface makes it running without errors, making SCP, SSH, FTP, or any other kind connection via the 10GB network works with no issues.

I am going to try FreeBSD to FreeBSD systems with these 2 10GB interfaces and see if the problem may comes from the Windows driver or iSCSI initiator.
If I got no success, I will open a topic in the proper forum section.

Thanks for the help.
 
Re: Mellanox Technologies MT26448 10GB interface driver prob

gnoma said:
I am going to try FreeBSD to FreeBSD systems with these 2 10GB interfaces and see if the problem may comes from the Windows driver or iSCSI initiator.
If I got no success, I will open a topic in the proper forum section.
Click to expand...
Yes, please test all the different configurations you can. It's possible the driver may contain some bugs, that could be a reason why it's not included in GENERIC. If you do run into issues don't hesitate to open a bug report.

https://bugs.freebsd.org/bugzilla/enter_bug.cgi
 
Re: Mellanox Technologies MT26448 10GB interface driver prob

Hello,

What I found out is that for some reason using the 10GB interface, I got package lost. I am not sure if this is the reason for iSCSI failure, because it's just 0.1% package lost, but from iSCSI point of view - alnost no traffic manage to get through.
Code: 
root@sentinel:~ # ping -f 10.0.80.2
PING 10.0.80.2 (10.0.80.2): 56 data bytes
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C.
--- 10.0.80.2 ping statistics ---
1537035 packets transmitted, 1536146 packets received, 0.1% packet loss
round-trip min/avg/max/stddev = 0.019/0.031/0.188/0.004 ms
root@sentinel:~ #
Today I'll try to change the drivers on both interfaces from the FreeBSD to the Mellanox provided drivers.

UPDATE:

Using the FreeBSD driver for mellanox interfaces got from here http://www.mellanox.com/downloads/Drivers/MLNX_EN_FreeBSD_v2.1.tgz, during flooging ping I still got some package lost but it seems not that much.
Code: 
root@sentinel:~ # ping -f 10.0.80.2
PING 10.0.80.2 (10.0.80.2): 56 data bytes
.....................................................^C
--- 10.0.80.2 ping statistics ---
1315990 packets transmitted, 1315937 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.021/0.033/0.613/0.005 ms
root@sentinel:~ #
Using iSCSI got no errors end everything worked perfectly fine. Package lost may be caused by not so tuff cable, or something wrong in my tuning. However I got this package lost only during high intensive ping flood and iSCSI is running over TCP, so I hope there will be no frames lost during iSCSI connection. Manage to get speed as fast as the limit of my disk controller.
As SirDice said, I am going to open a bug ticket.

UPDATE:
Bug submitted id is 193152, link is https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193152
Thank you.
 
Hi could you provide some instructions on how you installed the drivers. Or a provide a link that has one. I'm trying to install the drivers on FreeBSD 9.3-STABLE.
 
You must log in or register to reply here.
Back
Top