Problem with Infiniband

D06PbIU · #1 January 9th, 2012, 18:02

I rebuilt world and kernel on two servers with ofed support. These servers work good together. They can ping linux-infiniband-servers and FreeBSD-inifiniband-servers, but linux-infiniband-servers can not ping FreeBSD-infiniband-servers.

I listen (tcpdump) on infiniband interfaces on FreeBSD-inifiniband-servers and see that: FreeBSD-infiniband-servers get linux-infiniband-servers request but don't reply for them.

But:
I listen (tcpdump) on infiniband interfaces on linux-infiniband-servers and see that: linux-infiniband-servers get FreeBSD-inifiniband-servers request and reply for them. It works!

FreeBSD infiniband doesn't want work with Linux infiniband?

Please help me to solve this problem.

P.S.
Sorry for my bad English.

Pantagruel · #2 January 11th, 2012, 11:36

@D06PbIU

Could yo be so kind a describe how you went about rebuilding both world and kernel?
Regarding FreeBSD kernel rebuilds and edits I am a complete n00b, but the prospect of getting a Mellanox Infiniband card working in my FreeBSD box (ZFS server) looks attractive.
A how-to style procedure would be greatly appreciated, thank you very much.

Pantagruel

D06PbIU · #3 January 11th, 2012, 15:41

Build world ---> http://www.freebsd.org/doc/en_US.ISO...makeworld.html
Build kernel --> http://www.freebsd.org/doc/en_US.ISO...nelconfig.html

Rebuilding world and kernel:

Code:
```
echo "WITH_OFED=1" >> /etc/make.conf
```
#InfiniBand support
Code:
```
cd /usr/src/sys/<arch>/conf
```
Code:
```
cp GENERIC MYKERNEL
```
#crate your kernel configuration file

Change this options in you kernel config file (MYKERNEL):

Code:

include GENERIC
ident MYKERNEL

For InfiniBand support, you kernel config file must contain this strings:

Code:

options OFED # Infiniband protocol
options OFED_DEBUG_INIT
options SDP # Sockets Direct Protocol for infiniband
options SDP_DEBUG
options IPOIB_DEBUG
options IPOIB_CM # Use connect mode ipoib

device mlx4ib # ConnectX Infiniband support
device mlxen # ConnectX Ethernet support
device mthca # Infinihost cards
device ipoib # IP over IB devices

Code:
```
cd /usr/src
```
Code:
```
make buildworld
```
Code:
```
make buildkernel KERNCONF=MYKERNEL
```
Code:
```
make installkernel KERNCONF=MYKERNEL
```
Reboot into single user mode.
You need to read /usr/src/UPDATING
Code:
```
fsck -p
```
Code:
```
mount -u /
```
Code:
```
mount -a
```
Code:
```
swapon -a
```

Code:

mergemaster -p

Code:

make installworld

Code:

mergemaster -i

Code:

reboot

Good luck!

P.S.
Sorry for my very bad English.

D06PbIU · #4 January 11th, 2012, 15:45

And you need using FreeBSD 9.0-RELEASE of course!

Pantagruel · #5 January 11th, 2012, 20:23

Quote:

Originally Posted by D06PbIU

And you need using FreeBSD 9.0-RELEASE of course!

FreeBSD 9 in combination with ZFS and OFED are the only two reasons why I'd go through the hassle of a buildworld/kernel and installworld/kernel procedure. One could try the OFED driver for Linux and ZFS on linux, but the latter is a bit too beta for my taste.

As soon as the mellanox cards arrive (customs is chewing on them acc. USPS) I'll find out if the test rig is able to absolve the procedure, get Infiniband running and hopefully able to shift more data to and from the ZFS test array than the current Gbit interface.

Pantagruel · #6 January 12th, 2012, 20:21

Quote:

Originally Posted by D06PbIU

Build world ---> http://www.freebsd.org/doc/en_US.ISO...makeworld.html
Build kernel --> http://www.freebsd.org/doc/en_US.ISO...nelconfig.html
..snipz.

Thanks for the chunk sized how-to.
All went well up to the

Code:

 mergemaster -p

 make installworld

at which point make installworld returned

Code:

don't know how to make installworld

This was easily overcome with some help from the FreeBSD handbook which pointed out I should have cd'd into /usr/src.

So after a quick

Code:

 cd /usr/src
 make installworld
 mergemaster -i
 reboot

I had my custom kernel running. Sadly customs is still chewing on my Mellanox cards so I can't check if they are truly supported (or not).

D06PbIU · #7 January 17th, 2012, 09:19

I noticed a very strange thing:
FreeBSD-infiniband-server sometimes reply for Linux-infiniband-server pings, but very rarely.

Pantagruel · #8 January 20th, 2012, 15:13

Quote:

Originally Posted by D06PbIU

I noticed a very strange thing:
FreeBSD-infiniband-server sometimes reply for Linux-infiniband-server pings, but very rarely.

I received my mellanox cards yesterday and the custom OFED FreeBSD kernel detects the cards.
Unfortunately I am still waiting for my SFF8470 cables, so no transfer/ping tests yet.

Take a look at this blog, they had problems with pinging which boiled down to unanswered ARP requests.
So run a sniffer (Wireshark) and see what is happening on your IB interfaces.

Pantagruel · #9 January 29th, 2012, 12:13

It has been a while. In the meantime the cables have arrived and IPoIB is up and running.
Iperf test between a Win7 x64 box and FreeBSD9 OFED custom kernels tops at roughly 7 Gbps which is nice.

I have yet to install linux on the test rig and see if there are any ping problems when trying to reach a FreeBSD box.

aragon · #10 January 29th, 2012, 19:04

Quote:

Originally Posted by Pantagruel

Iperf test between a Win7 x64 box and FreeBSD9 OFED custom kernels tops at roughly 7 Gbps which is nice.

Curious to know which Mellanox cards you're using?

Thanks for reporting!

Pantagruel · #11 January 29th, 2012, 19:57

Quote:

Originally Posted by aragon

Curious to know which Mellanox cards you're using?

Thanks for reporting!

Mellanox MHEA28-XTC HCA's, got them from Ebay

Adrculda · #12 February 3rd, 2012, 05:53

Quote:

Originally Posted by Pantagruel

Mellanox MHEA28-XTC HCA's, got them from Ebay

That's good to hear! I got four MHET2x-1TC cards which I picked off ebay as well as a lot for $150+ shipping. I'm waiting for my other two servers to arrive (DL580 G4) which will get the InfiniBand treatment as soon as they arrive along with the RAM upgrade from 8 to 32 GB. Both units have Quad 3.4GHZ Xeon Duo Core Processors which should be plenty for a Media Server and Apache/IIS.

My switch Fortinet/ Woven Systems TRX100 has four 10 GBe backbone ports along with 48 GBe ports.

Pantagruel · #13 February 4th, 2012, 00:06

Quote:

Originally Posted by Adrculda

That's good to hear! I got four MHET2x-1TC cards which I picked off ebay as well as a lot for $150+ shipping. I'm waiting for my other two servers to arrive (DL580 G4) which will get the InfiniBand treatment as soon as they arrive along with the RAM upgrade from 8 to 32 GB. Both units have Quad 3.4GHZ Xeon Duo Core Processors which should be plenty for a Media Server and Apache/IIS.

My switch Fortinet/ Woven Systems TRX100 has four 10 GBe backbone ports along with 48 GBe ports.

MMM, nice hardware.

So far I have received little 'love' from IB in conjunction with FreeBSD.
Sure the fabric works, but getting a perfect iperf run isn't the goal.
I am kind of stuck trying to get iSCSI to work over IPoIB, the Win 7 x64 workstation is able to see the device presented by the iSCSI service from the FreeBSD machine. The problem however is that disk management hangs completely when told to look for a new device.
I did have wireshark do a capture of the network chatter, but analyzing this data will take sometime. Any suggestion is welcome

Adrculda · #14 February 4th, 2012, 15:15

I have no experience with that as my SAN is on fiber-channel.
The Controller shelf is an HP MSA1500cs running dual controllers in Active/Active configuration, which is attached to 2x HP MSA20 Drive shelf ( each supports a maximum of 12 x 3TB SAS/SATA drives ). From the controller there's 2 fiber feeds to 2 HP Storageworks AA979A 8 Port SAN Switch from which it splits to the servers.

I'm still waiting for the ERM modules for my R5500 UPS and the 36U rack to arrive.
Currently everything is kinda piled on top of each other with foam in between :/
I posted a bounty in the pfSense forums for IPoIB support, hopefully someone bites.

Adrculda · #15 February 4th, 2012, 23:07

Here's some teaser's pics of the switch...
TRX 100

Pantagruel · #16 March 7th, 2012, 20:40

Sorry for the delay in updates regarding Linux and FreeBSD Infiniband interoperability.

Over the past few weeks I have tried to get FreeBSD use IB for more than just iperf or hperf runs and basically failed.
I eventually installed linux (Ubuntu 11.10 server) on the test rig and was able to get srp target setup and running using the IB fabric (Thanks to David Hunt's blog and some minor and quite obvious changes had to be made.)Using a RAM disk as target, the Infiniband fabric and cables used where able to deliver as promised: transfer size of 64K and up at a Queue depth of 8 will deliver read/write speeds in excess of 900MB/sec. I have even been able to use a disk as SRP target, even though the throughput was appalling (approx. 20 MB/sec Read and Write from a beat up 20 GB PATA WD 5400 rpm drive) it provided proof of principle.
At least it's comforting to know the hardware is in good shape, any future doubts regarding obtainable transfer speeds using this particular hardware are thus unwarranted.

Back on topic.
D06PbIU mentioned problems pinging to and fro linux-FreeBSD when using IPoIB. I was able to have Windows 7 x64 ping the Ubuntu IB install and vice versa, the FreeBSD OFED install however stayed mute and would not answer any ping be it from the Win7 machine nor the Ubtunu machine and vice versa. Since IB support in FreeBSD is virtually undocumented I have no clue where to even start looking, I have scoured the net but all the searched provided little or no info.

Adrculda · #17 March 9th, 2012, 08:26

Quote:

Originally Posted by Pantagruel

Since IB support in FreeBSD is virtually undocumented I have no clue where to even start looking, I have scoured the net but all the searched provided little or no info.

You think that's bad?

I'm trying to get the drivers ported into pfSense so my firewall PC is connected to my inside network by a 10Gb link. Clients running Gigabit connections while servers on 10G and on the "other side" of the servers is the SAN.

I had one someone that was willing to port the drivers that lives here in town, but when he came and saw my setup he just walked right back out :/

Quote:

Originally Posted by Adrculda

I had one someone that was willing to port the drivers that lives here in town, but when he came and saw my setup he just walked right back out :/

Sad part is that he was charging $120/hr!

Adrculda · #18 June 30th, 2012, 17:17

Any updates? ?? [*]

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Is Infiniband supported under FreeBSD?	chodong	FreeBSD Development	6	June 26th, 2012 06:01
Samba-problem 1:doesn't start, 2:host-related problem	naguz	Web & Network Services	7	September 29th, 2009 15:40
geometry problem maby? boot problem	gulanito	Installing & Upgrading	6	August 13th, 2009 16:19
Problem with configuration of 2’nd DSL link on FreeBSD 7.2 (gateway problem?)	fanz	Networking	1	August 13th, 2009 04:04

The Following User Says Thank You to Pantagruel For This Useful Post:
D06PbIU (January 11th, 2012)

The Following User Says Thank You to D06PbIU For This Useful Post:
Pantagruel (January 11th, 2012)

The Following User Says Thank You to Pantagruel For This Useful Post:
D06PbIU (January 23rd, 2012)

The Following User Says Thank You to Pantagruel For This Useful Post:
D06PbIU (January 30th, 2012)

The Following User Says Thank You to Pantagruel For This Useful Post:
aragon (February 3rd, 2012)