LSI MegaRaid 9240-8i and FreeBSD 10.1 RELEASE LSI mrsas driver

[double]

Greetings to all!

# uname -a

FreeBSD freebsdfs 10.1-RELEASE FreeBSD 10.1-RELEASE #2: Fri May 15 14:52:31 MSK 2015     root@freebsdfs:/usr/obj/usr/src/sys/KERNEL1  amd64

# pciconf -lv mfi0

mfi0@pci0:5:0:0:        class=0x010400 card=0x92401000 chip=0x00731000 rev=0x03 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'MegaRAID SAS 2008 [Falcon]'
    class      = mass storage
    subclass   = RAID

Built in FreeBSD drivers:
mfi(4) - work fine, but it's older than latest driver on LSI site and I have some "mfi0 unexpected sense" messages in /var/log/messages, so I decide to try LSI native drivers.
mrsas(4) - didn't work (no surprise, 9240 not in supported hw list)
---

Drivers from lsi.com:
Latest (06.707.04.00) FreeBSD driver available on LSI site contains only mrsas(4) driver, but have LSI MR 9240-8i card in "Supported controllers" list.

I rebuild kernel without mfi(4) and mrsas(4) and replace /boot/kernel/mrsas.ko by driver compiled from LSI sources, but it didn't work. Driver loading at boot time, but didn't see RAID.

I tried to load only mrsas.ko, without mfi.ko - no success.
I tried to use hw.mfi.mrsas_enable="1" - same result.
---

Is it possible to make work LSI MegaRaid 9240-8i on FreeBSD 10.1 RELEASE using lastest (mrsas(4)) driver from lsi.com?

Or maybe better forget about this controller and use ZFS raid? This is small office fileserver for 25-30 users.

 

[Remington]

Are you using P19 firmware for your LSI card? P20 have issues with FreeBSD 10.x.

 

[Oko]

Or maybe better forget about this controller and use ZFS raid? This is small office fileserver for 25-30 users.

You would save $500-$600 by returning that RAID controller and getting a good Host Bus Adapter like LSI SAS9211-8i which can be found new on e-bay for $100 and going with ZFS raidz2.

Don't get me wrong LSI9240-8i is an excellent RAID controller and I have three of those in my Lab running Red Hat 6.6 and
using XFS. Unless you have legacy reasons to stick with UFS or hardware RAID or you don't want to increase hardware/software diversity in your work place ZFS is no brainier.

You get NFS/SMB aware Journaling with ZFS snapshots. You have practically built in backup utility with ZFS remote replication and it is easy to monitor (Hardware RAID is also very easy to monitor).

 

[Remington]

I agree SAS9211 is better and cheaper option. The only difference is that 9240 got cache built-in and its not needed if you have fast HDD or SSD. I have both cards but I like SAS9211 better with ZFS. To use SAS9211 or SAS9240 with ZFS, you'll have to flash the non-RAID firmware to the card for it to work. Never mix both hardware RAID and ZFS.

 

[gkontos]

During my recent experiences that were described in this thread, I talked with many people in IX Systems. We also had a debate regarding IR and IT mode on HBA controllers. My system uses multipath and the on board controller is in IR mode, while the HBA card is in IT mode. I was assured that this will not cause any issues and in fact, after 2 months running this in production I have not experienced any problem.

The reason I am mentioning this is because I see a lot of people trying to flash IT mode over IR. It is possible and there are procedures for erasing the BIOS completely first but it turns out that there is no reason to do so.

 

[phoenix]

I run my onboard controllers in IR mode, even when running ZFS on top. Mainly due to the royal pain in the keister it would be to flash the IT firmware over the onboard controller. It's a very long and convoluted process that requires some memory region finagling, and it just isn't worth the effort for the risk of botching things. No issues running ZFS with them in IR mode; just don't create any arrays (or create 1-disk RAID0 arrays) and use the individual drives.

All of my PCIe-based HBAs, though, are flashed with whichever firmware version matches the FreeBSD driver version.

 

[Remington]

I had to flash IT firmware otherwise my SSDs connected to the card won't boot up with ZFS. If not using the card as bootable device then IR mode will still work as long as the arrays are not created.

 

[double]

First of all - thank you all for ideas, there is a lot to think/try of.

Are you using P19 firmware for your LSI card? P20 have issues with FreeBSD 10.x.

Firmware version 2.130.404-3836
FW Package version 20.13.1-0208
And i'm not sure how to tell it's P19 or P20, but definitely this is lastest firmware for 9240-8i that i found on lsi.com

Unless you have legacy reasons to stick with UFS or hardware RAID

My main priority is stability. Speed is less important. So i choose LSI 9240-8i instead of Intel 5520 builtin ICH10(honestly speaking 9240 just gathering dust on shelf, but anyway i think this is better choise)
And i don't think much about file system... yet. I don't want to start holy war "which file system better for file server". Like i said before, i need stability plus ACL and if XFS is better, then i will use XFS.

To use SAS9211 or SAS9240 with ZFS, you'll have to flash the non-RAID firmware to the card for it to work. Never mix both hardware RAID and ZFS.

Wow, there is a lot to read about flashing, but i can't understand main concept and little bit confused.
1. Do i need special/custom firmware or i can flash 9211 firmware on my 9240 after some BIOS manipulations?
2. After flash 9240 became 9211 or something different?(i asking this because of possible driver issues, do i need special driver?)
3. Can flashing be reverted?

Of course i read posts gkontos and phoenix and will not flash my 9240 until i completly understand process and possible consequences.

No issues running ZFS with them in IR mode; just don't create any arrays (or create 1-disk RAID0 arrays) and use the individual drives.

after 2 months running this in production I have not experienced any problem

If i'm understand correctly i need to delete Virtual disk, leave disks as JBOD and make ZFS on them.
In case of 9240 failure, can i use ZFS on other controller? I'm talking about switching HDD to ICH10 on motherboard.
Is acceptable to install FreeBSD on USB flash drive and use whole ZFS volume as share?

 

[Remington]

Your card got P20 firmware so you will need to downgrade to 9211-8i P19 IT firmware. There is no such P19 firmware for 9240-8i card so you will have to download the P19 firmware and flash tools from LSI website. It's not that difficult to flash but do read their doc. Flashing firmware should be considered as last resort to try to correct the problems and many people flash it just because they need the latest firmware and often times it breaks something due to driver and firmware mismatched. So only flash when its absolutely necessary. P19 is known to be more stable than any other firmware for FreeBSD and it will recognize your card as 9211-8i. It will work.

If you don't use SAS hardware RAID feature then you should be able to migrate your HDD to another controller if your SAS card fails. It is rare for controller cards to fail. Its more likely HDDs or SSDs will fail before the controller card does.

If you plan to use hardware RAID with UFS then flash your card with IR firmware. if you plan to use non-RAID with ZFS then flash your card with IT firmware. So you will have to decide which one you will use... hardware RAID with UFS or non-RAID with ZFS.

I had to downgrade my SAS controllers from P20 to P19 for three reasons:
1) FreeBSD 10.x have issues with P20 firmware.
2) Needed IT firmware to boot from SAS controller card.
3) non-RAID mode for ZFS.

Using USB drive on production server is not recommended.

 

[double]

I found a lot of instructions about flashing 9240 to 9211 seems like they all have the same origin. And now iI have "itchy fingers" to flash : )
But first of all iI will make some tests.

It's a very long and convoluted process that requires some memory region finagling, and it just isn't worth the effort for the risk of botching things

If we talk about flashing 9240, it seems pretty easy for me(if there will be no unexpected troubles). Or you talking about flashing controllers at all?
I'm curious because iI understand that ROM memory is not made for many cycles of rewriting and if there is some warnings about flashing 9240 card, iI'd be very thankful for sharing them.

 

[Remington]

If we talk about flashing 9240, it seems pretty easy for me(if there will be no unexpected troubles). Or you talking about flashing controllers at all?
I'm curios because i understand that ROM memory is not made for many cycles of rewriting and if there is some warnings about flashing 9240 card, i'd be very thankfull for sharing them.

It's not convoluted if you know exactly what you're doing and follow the flashing instruction that came with the firmware. Flashing wrong firmware will botch things up and possibly turn the controller card into a paperweight.

He flashed ALL of his HBA controllers to match with FreeBSD driver version. I did the same thing too.