Installing FreeBSD 9 on the IBM System x3550 M3

A couple of days ago, we received a brand new IBM x3550 M3 model 7944AC1. My goal for this system was to do a simple mirrored ZFS root install. As there are some non-conclusive threads on whether this system works at all, I decided to document how I got it to work.

This system has an embedded IBM M1015 SAS Raid controller on a dedicated PCIe slot, as well as an UEFI bios (Version 1.11). The controller is an entry level RAID0/1 card based on the LSI SAS2008 chipset, very similar to the LSI 9240-8I controller, yet without RAID5. The controller's firmware is what LSI calls MegaRaid firmware. Unfortunately, there is no appropiate driver for this firmware in FreeBSD at the time of this writing.

The ordeal

As delivered, with the default BIOS settings, FreeBSD boots from CD, but is not able to see the disks, no mater what. If you want to boot it off a USB stick, you'll have to wait for the SAS controller's prompt to appear before plugging it in, or the system will hang.

Now, as discussed in several forums, the M1015 can be re-flashed to what LSI calls "IT" (initiator-target) mode, and then, used as a high performance 6Mbps SAS controller with the mps driver, which is what you want if you are going to use ZFS. You will find everything you'll need to know here.

http://lime-technology.com/forum/index.php?topic=12767.msg124393#msg124393

I used the latest P11 firmware for the 9210-8i from the LSI site, as it seems to be the very same hardware as the M1015. The re-flash was done booting from a DOS Usb stick on the x3550 itself. Do not forget to take notice of your controller's SAS Address before re-flashing it! After a successful re-flash, you should see the LSI controller at boot-up, and be able to enter the controller's configuration settings by pressing CTRL-C while the controller boots.

If you try to install FreeBSD right away, you will be disappointed. The boot process waits for ever with this message:
Code:
run_interrupt_driven_hooks - waiting for xpt_config
Desperate, thinking I had fried the SAS controller, I did a quick install of Ubuntu 10.04, which installed correctly, yet refused to boot as well.

Now, to the BIOS. The x3550 has an UEFI bios, however, the LSI 9210-8i firmware is not UEFI aware (the original M1015 is). I have reasons to believe that the BIOS is not able to properly enumerate/assign resources to the controller in this mode, see below.

I tried every switch I could find in the BIOS, with 8.2-STABLE as well as 9.0-BETA3 to no avail, until I arrived at the following configuration:

Code:
Main menu-> System Settings -> Devices and I/O Ports

  Enable/Disable onboard devices: 
   Everything disabled but the SAS controller
  Enable/Disable device option ROM: 
   Under Legacy, everything disabled but the SAS controllers, under UEFI, everything disabled.
  Set Option ROM execution order: 
   SAS Controller first

Main menu-> System Settings -> Legacy Support
  Rehook Int 19h -> Enable
  Legacy thunk support -> Disable

Main menu-> Boot Manager
  Primary boot options
  Add boot option: Legacy only
  Delete boot option: everything else
In this configuration, I was finally able to boot the installed Ubuntu 10.04, and, even better, I was able to boot FreeBSD 9 off both USB and DVD, with fully functional disks.

I then performed the documented ZFS install procedure, as shown in

http://www.aisecure.net/2011/05/01/root-on-zfs-freebsd-current/

Rebooted, enabled the Broadcom ethernet controllers in the BIOS under

Main menu-> System Settings -> Devices and I/O Ports

And got a fully functional FreeBSD-9 + ZFS root installation

To do:

1. Go through the BIOS again to find out the minimum set of changes that allows the installation to proceed
2. Try the integrated "lights-off" remote administration facility

Of note:

1. Booting FreeBSD of a USB stick is a hit and miss affair on this system, you'll be better off using a DVD
2. Raw disk access to my 500GB 7.2K SAS disks seems to be about 10% faster under FreeBSD than under Linux
3. The same procedure should apply to the very similar x3650 M3 when using an M1015 SAS adapter
 
I am having a new LSI MegaRAID SAS 9240-8i card on my PowerEdge server. R710 and R610 default H700 card works fine with FreeBSD 8.x and 9.0 RC1/RC2 but this new card has issues - no hard disk detected. Read through this post. Tried rebuilding the GENERIC kernel without mfi driver and inserting the new mfi driver from LSI website (http://pastebin.com/wVyWCVc2), it didn't help (http://pastebin.com/ykAf0Q1j). Created own patch, FreeBSD ISO through CVSUP but it lasted with mfi errors. Now when I read this post I got to know about reflashing in IT Mode. How shall I proceed with Dell PowerEdge server. Are those commands (which you mentioned) restricted to IBM servers. Please suggest.
 
The same procedure should apply to your LSI MegaRAID SAS 9240-8i, and, if you intend to use ZFS, you will be rewarded with an very fast disk subsystem. Please be absolutely sure to make a backup of your current firmware before you reflash the controller, should you need to reflash the card back. One big question is whether the bios of your system will recognize the reflashed card; some vendors implement bios-level restrictions in order to force you to use only their "approved hardware". I know for sure IBM does not; i have no idea however whether Dell does.
 
Thanks for the comments.
What commands and software do i need to reflash the controller.As I confirm its LSI MegaRAID SAS 9240.I am completely new to Firmware reflashing.Also, Why ZFS is being talked here?
Please it would be great if you can help me with reference or docs if any for PowerEdge server.
 
LSI 9240-4i working ok on FreeBSD 9.0-RC2/PRERELEASE

Hi!

I have a LSI 9240-4i working ok on FreeBSD 9.0-RC2/PRERELEASE.

FreeBSD 9.0 mfi driver does not support this board (the drivers attached to it are not detected), so I got the driver at LSI's page (for FreeBSD 7.x / 8.x - 64-bit version) and changed it to work with 9.0. It looks to be working well.

If you want to give it a try, here is what I have done (note: I am supposing all your disks are attached to LSI's board - so you see no disks after a normal boot from FreeBSD 9.0 installation CDROM):
- Get the modified driver at http://www.ime.unicamp.br/~rodolfo/files/mfi_freebsd90_x64_rbm.tgz
- Boot from CDROM and install FreeBSD 9.0 on a USB drive (flash or external drive). Don't forget to install the sources, and don't install the ports tree.
- Boot from the USB device
- Replace the /usr/src/sys/dev/mfi directory with the one on the file
- Follow the instructions on Readme.doc file to change the kernel configuration and /usr/src/sys/conf/files
- Compile the new kernel (cd /usr/src/sys/amd64/conf; configure MYKERNEL; etc)
- Boot. Your new installation (on USB drive) should be able to see your RAID disks now!

Now you have a working system on a pendrive. To install on the harddisk:
- Boot from Installation CDROM
- Escape to the loader prompt
- Use unload kernel command to unload the kernel from the CDROM
- Use load xxxxxx:/boot/kernel/kernel to load the kernel from the USB device (use the lsdev command to figure out what device to use in place of "xxxxxxx"). So use the boot command to boot the installation CDROM
- Install FreeBSD as usual.
- Boot from the RAID HD and escape to the loader prompt again. The kernel you have just installed on the system's HD is a standard "GENERIC" kernel - without support to your LSI controller
- Use the unload kernel / load kernel / boot trick again to boot your system using the kernel from the USB device (with root on HD)
- Replace the mfi directory and reconfigure your kernel (the one on HD) as you did before. Rebuild and install the new kernel.
- Reboot. Now you have a working system.

Note:
1. if you update your source tree (via cvsup), the mfi directory will be overwritten - and you will need to replace it again if you want to build a new kernel. Don't forget to change the /usr/src/sys/conf/files file!
2. This driver is experimental. Use at your own risk
3. This driver is based on FreeBSD 8.x architecture. It lacks the new FreeBSD 9.x features and the mfitool utility will fail to talk to it. Use LSI's MegaCli istead.

Cheers,
 

Attachments

  • mfi_freebsd90_x64_rbm.tgz
    49.2 KB · Views: 622
RodLophus,

Thanks a lot for your points.
Few important doubts:-

1. I am unable to read Readme.doc(as it displays lots of ^@ in FreeBSD). So I referred this from LSI.
If the above is incorrect, do let me know.
Code:
1)	Make sure that kernel source has been installed on the system. If the kernel sources have been installed then there will be /usr/src/sys directory.
2)	Move to the arch/conf directory and copy the GENERIC configuration file to the name you want to give your kernel. For example:
            # cd /usr/src/sys/i386/conf
            For 64 bit the path is
            # /usr/src/sys/amd64/conf			
	# cp GENERIC MYKERNEL
3)	Perform the following steps.
a) Comment the following line in the file MYKERNEL by placing # at the      
     beginning of the line.
     device     cbb    #cardbus (yenta) bridge
b)  Add the following line to the file /usr/src/sys/conf/files 
    dev/mfi/mfi_syspd.c optional mfi
  (Note add this line next to the line which has dev/mfi/mfi_cam.c optional mfi)
4)	Copy the zip (mfi.zip) file under the path /usr/src/sys/dev (Before proceeding to next step make sure that the directory with name “mfi” does not exist under path /usr/src/sys/dev/. If already a directory with name “mfi” exists, move it to some other folder. For example type, mv /usr/src/sys/dev/mfi /root/mfibackup).
5)	Go to the directory /usr/src/sys/dev and run the following command to unzip the source.
      tar –x –f mfi.zip
6)	To compile the kernel go to /usr/src directory and type
       make buildkernel KERNCONF=MYKERNEL
7)	To install the new kernel type
       make installkernel KERNCONF=MYKERNEL

2.You mentioned:
Use load xxxxxx:/boot/kernel/kernel to load the kernel from the USB device (use the lsdev command to figure out what device to use in place of "xxxxxxx"). So use the boot command to boot the installation CDROM
- Install FreeBSD as usual.

In my case, I have a Installation-CD on my CD-Drive, disk0(BIOS drive C: which refers to RAID HD) and disk1(BIOS drive D: with partitions disk1p1, disk1p2 and disk1p3, USB partitions).


I ran load disk1p2 (corresponding to FreeBSD UFS partition): /boot/kernel/kernel and it went well. Next I ran boot command. It proceeded but during the installation, it showed me just da0 (pertaining to USB drive) during the partioning.
Didn't it detect the RAID HD? :(

When I am trying to boot from CD-Drive ISO, performed the loading of /boot/kernel/kernel,load mps, load mfi, boot it stopped at:

Code:
mountroot> 

<fstype>:<device> option 
I provided with:
ufs:/dev/disk1
and it rebooted.

No idea if it really recognized my disk. All I can see it does detect during the Loader Prompt.

Please suggest.
 
I got this working now.
As you mentioned:
if you update your source tree (via cvsup), the mfi directory will be overwritten - and you will need to replace it again if you want to build a new kernel. Don't forget to change the /usr/src/sys/conf/files file!

I tried building CVS Repo, replaced with your mfi driver on my 8.2-RELEASE system, but it threw the error:
Code:
/usr/src/usr.sbin/mfiutil/mfi_evt.c: In function 'mfi_decode_evt':
/usr/src/usr.sbin/mfiutil/mfi_evt.c:364: error: 'struct mfi_evt_detail' has no member named 'class'
/usr/src/usr.sbin/mfiutil/mfi_evt.c:365: error: 'struct mfi_evt_detail' has no member named 'class'
/usr/src/usr.sbin/mfiutil/mfi_evt.c: In function 'show_events':
/usr/src/usr.sbin/mfiutil/mfi_evt.c:557: error: 'struct <anonymous>' has no member named 'class'
/usr/src/usr.sbin/mfiutil/mfi_evt.c:567: error: 'struct <anonymous>' has no member named 'class'
*** Error code 1

Stop in /usr/src/usr.sbin/mfiutil.
*** Error code 1

Stop in /usr/src/usr.sbin.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.
+ exit 1
+ umount /dev
*** Error code 1

Stop in /usr/src/release.

Any idea how shall I fix this?
 
RodLophus,

Just to add more points.
I faced the above issue when I tried this command:
Code:
make release RELEASETAG=RELENG_8_1 PORTSRELEASETAG=HEAD BUILDNAME=8.1-RELEASE CHROOTDIR=/usr/home/releng NODOC=yes CVSROOT=/usr/home/ncvs LOCAL_PATCHES=/root/patch.mfi

It would be great if you can suggest me on the above thread.
 
MegaRAID SAS 9240 (LSI2008) on FreeBSD 8.2

Hi, linuxunix!

For FreeBSD 8.2 (I didn' test 8.1), the procedure are slightly different:

1. The file I mentioned before will not work. Use the one at http://www.ime.unicamp.br/~rodolfo/files/mfi_fbsd82_x64_rbm.tgz instead.

2. The "unload / load" trick doesn't work with 8.x: it will unload the mfs too and the new kernel won't load it again (once booted, the kernel will drop to the "mountroot>" prompt). Istead, escape to the loader prompt, unload the kernel (command: "unload"), set the "kernel" variable so it points to the patched kernel on your USB stick and use the "boot" command to boot the system - like this:

Code:
OK unload
OK set kernel=disk0p1:/boot/kernel/kernel
OK boot

Note: the device (disk0p1) is not the UDF partitions on your hard drive: it is the partition on the USB memory stick where you have your patched kernel. Use the "lsdev" command to figure out wich device to use.

At this point, you should have a system booting from your hard disk, but using the kernel on the USB memory stick. Now, you can patch and re-compile the kernel on your hard disk the same way you did with the one on the memory stick - and you will end up with a fully functional system.

BTW: The "Readme.doc" file is not plain text: it is a Windows Word file. It is the same file supplied with the original driver from LSI. You should be able to read it with MSWord, OpenOffice, Lotus Symphony, etc. The instructions on it are exactly the one you wrote in your post.

I hope it can help you.

Have fun!
 

Attachments

  • mfi_fbsd82_x64_rbm.gz
    48.9 KB · Views: 571
Thanks.

I will try it out on 8.2. As of now, I am setting up CVS Repo for building FreeBSD ISO for 9.0 RC1. Hope it works.

It would be great if you can help me with /root/patch.mfi (since it seems like it always failed for me).
 
Also, you mentioned:
f you update your source tree (via cvsup), the mfi directory will be overwritten - and you will need to replace it again if you want to build a new kernel. Don't forget to change the /usr/src/sys/conf/files file!

I didn't get the above statement.Wont I build up a new ISO through CVS for mfi driver.

All I am doing is following:
1. make release RELEASETAG=RELENG_8_2 PORTSRELEASETAG=HEAD BUILDNAME=8.1-RELEASE CHROOTDIR=/usr/home/releng NODOC=yes CVSROOT=/usr/home/ncvs LOCAL_PATCHES=/root/patch.mfi
2. sh /usr/src/release/amd64/mkisoimages.sh -b custom-FreeBSD82-amd
/usr/home/8.2-RELEASE-dvd1.iso /usr/home/releng/R/cdrom/dvd


Please suggest if I am wrong or I need to do anything else.
 
One quick question, why the mfi driver provided by LSI din't work while the modified version worked?It would be great if you can brief me about the same. This is just for technical understanding for self-help purpose.
Thanks.
 
Why LSI's driver doesn't works on FreeBSD 8.2 and 9.0

Hi, linuxunix!

I think the driver itself would work on FreeBSD 8.2. The problem is the interface with tools like mfiutil. The mfiutil tool on 8.2 has support for more interface commands, status querying and things link this - all of them are defined on the mfireg.h file. Also, there are one or two functions called from mfiutil whose names have changed.

The changes was deeper on FreeBSD 9.0. There are function calls that changed names and parameter syntax - so the driver itself doesn't work. The interface of the driver with mfitool changed a lot too.

To be honest, I didn't try to fix the interface between LSI's driver and mfitool. I just updated the definitions on the driver so it now matches the one expected by mfitool. It ends up that mfitool doesn't talk with this board (it returns "no such file or directory" for every command you try). I'm not concerned about this because LSI's MegaCli do all mfitool does - and a bit more.

BTW, there was a syntax error on LSI's driver (search the original code for something like "PORGRESS_RECON"). It would not prevent the driver from compiling, but would, for example, stop a "make world" (when building mfitool). I guess even the original LSI driver might not actually work with mfitool.


Regards!
 
Rodpholus,

It would be great if you can help me with ISO rather than couple of steps for making mfi work.
Though I have FreeBSD running without any issue, I tried creating cvsup repo but the patch dint work.
Can you help me with this?
 
Any idea if the driver has been considered in the latest 9.0 GA?
If not, how shall I raise a request for including it in FreeBSD tree?
How shall I create a new FreeBSD ISO with this driver?
 
What about LSI MegaRAID SAS 9265 card? I am unable to get this work?
Do you have driver for that? This mfi driver which worked for 9240 is not working for 9265.
Please suggest.
 
It would be nice if this driver could be included in the next release (9.1, 8.4).
Several Dell, IBM, Supermicro servers need this driver.

Thank you.
 
@rdeiriar

I have the same hardware except he server is an x3560. I also try almost every BIOS setting but I can't get past the boot initialisation process of the server. If I set BIOS exactly as you did then the boot partition (GPT or MBR) is not recognized and the system starts booting from DVD. But if I enable Legacy thunk support or enable UEFI in 'Enable/Disable device option ROM' the system stops when trying to initialize boot devices. It helps to get past to unplug all the drives.

But if disks are plugged after boot from DVD I can install FreeBSD without a problem.

You're probably using GPT and I also tried with MBR. I've read somewhere that Legacy mode is not working with GPT.

One thing I thought is to change the boot ROM on the controler. I found on LSI some other EFI package SAS2_UEFI_BSD_P11 and there are two ROMs, ebcsas2.rom and x64sas2.rom. Do you use any of these two or mptsas2.rom from the original LSI IT firmware?

Regards

Uros
 
linuxunix, can you stop attaching mfi-related posts to every single topic on the forums, please?
 
I can confirm that with the ebcsas2.rom file server boots normally and everything else works as it should. The only glitch remains long zfs boot when enumerating the drives.
 
Back
Top