Problem with Infiniband

D

D06PbIU

I rebuilt world and kernel on two servers with ofed support. These servers work good together. They can ping linux-infiniband-servers and FreeBSD-inifiniband-servers, but linux-infiniband-servers can not ping FreeBSD-infiniband-servers.

I listen (tcpdump) on infiniband interfaces on FreeBSD-inifiniband-servers and see that: FreeBSD-infiniband-servers get linux-infiniband-servers request but don't reply for them.

But:
I listen (tcpdump) on infiniband interfaces on linux-infiniband-servers and see that: linux-infiniband-servers get FreeBSD-inifiniband-servers request and reply for them. It works!

FreeBSD infiniband doesn't want work with Linux infiniband?

Please help me to solve this problem.

P.S.
Sorry for my bad English.
 
@D06PbIU

Could yo be so kind a describe how you went about rebuilding both world and kernel?
Regarding FreeBSD kernel rebuilds and edits I am a complete n00b, but the prospect of getting a Mellanox Infiniband card working in my FreeBSD box (ZFS server) looks attractive.
A how-to style procedure would be greatly appreciated, thank you very much.

Pantagruel
 
Build world ---> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
Build kernel --> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig.html

Rebuilding world and kernel:
  1. Code: 
    echo "WITH_OFED=1" >> /etc/make.conf
    #InfiniBand support
  2. Code: 
    cd /usr/src/sys/<arch>/conf
  3. Code: 
    cp GENERIC MYKERNEL
    #crate your kernel configuration file
  4. Change this options in you kernel config file (MYKERNEL):

    Code: 
    include GENERIC
ident MYKERNEL

    For InfiniBand support, you kernel config file must contain this strings:

    Code: 
    options OFED # Infiniband protocol
options OFED_DEBUG_INIT
options SDP # Sockets Direct Protocol for infiniband
options SDP_DEBUG
options IPOIB_DEBUG
options IPOIB_CM # Use connect mode ipoib

device mlx4ib # ConnectX Infiniband support
device mlxen # ConnectX Ethernet support
device mthca # Infinihost cards
device ipoib # IP over IB devices
  5. Code: 
    cd /usr/src
  6. Code: 
    make buildworld
  7. Code: 
    make buildkernel KERNCONF=MYKERNEL
  8. Code: 
    make installkernel KERNCONF=MYKERNEL
  9. Reboot into single user mode.
  10. You need to read /usr/src/UPDATING
    Code: 
    fsck -p
    Code: 
    mount -u /
    Code: 
    mount -a
    Code: 
    swapon -a

  11. Code: 
    mergemaster -p
    Code: 
    make installworld
    Code: 
    mergemaster -i
    Code: 
    reboot
Good luck!

P.S.
Sorry for my very bad English.
 
D06PbIU said:
And you need using FreeBSD 9.0-RELEASE of course!
Click to expand...

FreeBSD 9 in combination with ZFS and OFED are the only two reasons why I'd go through the hassle of a buildworld/kernel and installworld/kernel procedure. One could try the OFED driver for Linux and ZFS on linux, but the latter is a bit too beta for my taste.

As soon as the mellanox cards arrive (customs is chewing on them acc. USPS) I'll find out if the test rig is able to absolve the procedure, get Infiniband running and hopefully able to shift more data to and from the ZFS test array than the current Gbit interface.
 
D06PbIU said:
Build world ---> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
Build kernel --> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig.html
..snipz.
Click to expand...

Thanks for the chunk sized how-to.
All went well up to the

Code: 
 mergemaster -p

 make installworld

at which point make installworld returned
Code: 
don't know how to make installworld

This was easily overcome with some help from the FreeBSD handbook which pointed out I should have cd'd into /usr/src.

So after a quick

Code: 
 cd /usr/src
 make installworld
 mergemaster -i
 reboot

I had my custom kernel running. Sadly customs is still chewing on my Mellanox cards so I can't check if they are truly supported (or not).
 
I noticed a very strange thing:
FreeBSD-infiniband-server sometimes reply for Linux-infiniband-server pings, but very rarely.
 
D06PbIU said:
I noticed a very strange thing:
FreeBSD-infiniband-server sometimes reply for Linux-infiniband-server pings, but very rarely.
Click to expand...

I received my mellanox cards yesterday and the custom OFED FreeBSD kernel detects the cards.
Unfortunately I am still waiting for my SFF8470 cables, so no transfer/ping tests yet.

Take a look at this blog, they had problems with pinging which boiled down to unanswered ARP requests.
So run a sniffer (Wireshark) and see what is happening on your IB interfaces.
 
It has been a while. In the meantime the cables have arrived and IPoIB is up and running.
Iperf test between a Win7 x64 box and FreeBSD9 OFED custom kernels tops at roughly 7 Gbps which is nice.

I have yet to install linux on the test rig and see if there are any ping problems when trying to reach a FreeBSD box.
 
Pantagruel said:
Iperf test between a Win7 x64 box and FreeBSD9 OFED custom kernels tops at roughly 7 Gbps which is nice.
Click to expand...
Curious to know which Mellanox cards you're using?

Thanks for reporting!
 
Pantagruel said:
Mellanox MHEA28-XTC HCA's, got them from Ebay
Click to expand...

That's good to hear! I got four MHET2x-1TC cards which I picked off ebay as well as a lot for $150+ shipping. I'm waiting for my other two servers to arrive (DL580 G4) which will get the InfiniBand treatment as soon as they arrive along with the RAM upgrade from 8 to 32 GB. Both units have Quad 3.4GHZ Xeon Duo Core Processors which should be plenty for a Media Server and Apache/IIS.

My switch Fortinet/ Woven Systems TRX100 has four 10 GBe backbone ports along with 48 GBe ports.
 
Adrculda said:
That's good to hear! I got four MHET2x-1TC cards which I picked off ebay as well as a lot for $150+ shipping. I'm waiting for my other two servers to arrive (DL580 G4) which will get the InfiniBand treatment as soon as they arrive along with the RAM upgrade from 8 to 32 GB. Both units have Quad 3.4GHZ Xeon Duo Core Processors which should be plenty for a Media Server and Apache/IIS.

My switch Fortinet/ Woven Systems TRX100 has four 10 GBe backbone ports along with 48 GBe ports.
Click to expand...

MMM, nice hardware.

So far I have received little 'love' from IB in conjunction with FreeBSD.
Sure the fabric works, but getting a perfect iperf run isn't the goal.
I am kind of stuck trying to get iSCSI to work over IPoIB, the Win 7 x64 workstation is able to see the device presented by the iSCSI service from the FreeBSD machine. The problem however is that disk management hangs completely when told to look for a new device.
I did have wireshark do a capture of the network chatter, but analyzing this data will take sometime. Any suggestion is welcome
 
I have no experience with that as my SAN is on fiber-channel.
The Controller shelf is an HP MSA1500cs running dual controllers in Active/Active configuration, which is attached to 2x HP MSA20 Drive shelf ( each supports a maximum of 12 x 3TB SAS/SATA drives ). From the controller there's 2 fiber feeds to 2 HP Storageworks AA979A 8 Port SAN Switch from which it splits to the servers.

I'm still waiting for the ERM modules for my R5500 UPS and the 36U rack to arrive.
Currently everything is kinda piled on top of each other with foam in between :/
I posted a bounty in the pfSense forums for IPoIB support, hopefully someone bites.
 
Sorry for the delay in updates regarding Linux and FreeBSD Infiniband interoperability.

Over the past few weeks I have tried to get FreeBSD use IB for more than just iperf or hperf runs and basically failed.
I eventually installed linux (Ubuntu 11.10 server) on the test rig and was able to get srp target setup and running using the IB fabric (Thanks to David Hunt's blog and some minor and quite obvious changes had to be made.)Using a RAM disk as target, the Infiniband fabric and cables used where able to deliver as promised: transfer size of 64K and up at a Queue depth of 8 will deliver read/write speeds in excess of 900MB/sec. I have even been able to use a disk as SRP target, even though the throughput was appalling (approx. 20 MB/sec Read and Write from a beat up 20 GB PATA WD 5400 rpm drive) it provided proof of principle.
At least it's comforting to know the hardware is in good shape, any future doubts regarding obtainable transfer speeds using this particular hardware are thus unwarranted.

Back on topic.
D06PbIU mentioned problems pinging to and fro linux-FreeBSD when using IPoIB. I was able to have Windows 7 x64 ping the Ubuntu IB install and vice versa, the FreeBSD OFED install however stayed mute and would not answer any ping be it from the Win7 machine nor the Ubtunu machine and vice versa. Since IB support in FreeBSD is virtually undocumented I have no clue where to even start looking, I have scoured the net but all the searched provided little or no info.
 
Pantagruel said:
Since IB support in FreeBSD is virtually undocumented I have no clue where to even start looking, I have scoured the net but all the searched provided little or no info.
Click to expand...

You think that's bad?

I'm trying to get the drivers ported into pfSense so my firewall PC is connected to my inside network by a 10Gb link. Clients running Gigabit connections while servers on 10G and on the "other side" of the servers is the SAN.

I had one someone that was willing to port the drivers that lives here in town, but when he came and saw my setup he just walked right back out :/

Adrculda said:
I had one someone that was willing to port the drivers that lives here in town, but when he came and saw my setup he just walked right back out :/
Click to expand...

Sad part is that he was charging $120/hr!
 
You must log in or register to reply here.
Back
Top