A couple of days ago, we received a brand new IBM x3550 M3 model 7944AC1. My goal for this system was to do a simple mirrored ZFS root install. As there are some non-conclusive threads on whether this system works at all, I decided to document how I got it to work.
This system has an embedded IBM M1015 SAS Raid controller on a dedicated PCIe slot, as well as an UEFI bios (Version 1.11). The controller is an entry level RAID0/1 card based on the LSI SAS2008 chipset, very similar to the LSI 9240-8I controller, yet without RAID5. The controller's firmware is what LSI calls MegaRaid firmware. Unfortunately, there is no appropiate driver for this firmware in FreeBSD at the time of this writing.
The ordeal
As delivered, with the default BIOS settings, FreeBSD boots from CD, but is not able to see the disks, no mater what. If you want to boot it off a USB stick, you'll have to wait for the SAS controller's prompt to appear before plugging it in, or the system will hang.
Now, as discussed in several forums, the M1015 can be re-flashed to what LSI calls "IT" (initiator-target) mode, and then, used as a high performance 6Mbps SAS controller with the mps driver, which is what you want if you are going to use ZFS. You will find everything you'll need to know here.
http://lime-technology.com/forum/index.php?topic=12767.msg124393#msg124393
I used the latest P11 firmware for the 9210-8i from the LSI site, as it seems to be the very same hardware as the M1015. The re-flash was done booting from a DOS Usb stick on the x3550 itself. Do not forget to take notice of your controller's SAS Address before re-flashing it! After a successful re-flash, you should see the LSI controller at boot-up, and be able to enter the controller's configuration settings by pressing CTRL-C while the controller boots.
If you try to install FreeBSD right away, you will be disappointed. The boot process waits for ever with this message:
Desperate, thinking I had fried the SAS controller, I did a quick install of Ubuntu 10.04, which installed correctly, yet refused to boot as well.
Now, to the BIOS. The x3550 has an UEFI bios, however, the LSI 9210-8i firmware is not UEFI aware (the original M1015 is). I have reasons to believe that the BIOS is not able to properly enumerate/assign resources to the controller in this mode, see below.
I tried every switch I could find in the BIOS, with 8.2-STABLE as well as 9.0-BETA3 to no avail, until I arrived at the following configuration:
In this configuration, I was finally able to boot the installed Ubuntu 10.04, and, even better, I was able to boot FreeBSD 9 off both USB and DVD, with fully functional disks.
I then performed the documented ZFS install procedure, as shown in
http://www.aisecure.net/2011/05/01/root-on-zfs-freebsd-current/
Rebooted, enabled the Broadcom ethernet controllers in the BIOS under
Main menu-> System Settings -> Devices and I/O Ports
And got a fully functional FreeBSD-9 + ZFS root installation
To do:
1. Go through the BIOS again to find out the minimum set of changes that allows the installation to proceed
2. Try the integrated "lights-off" remote administration facility
Of note:
1. Booting FreeBSD of a USB stick is a hit and miss affair on this system, you'll be better off using a DVD
2. Raw disk access to my 500GB 7.2K SAS disks seems to be about 10% faster under FreeBSD than under Linux
3. The same procedure should apply to the very similar x3650 M3 when using an M1015 SAS adapter
This system has an embedded IBM M1015 SAS Raid controller on a dedicated PCIe slot, as well as an UEFI bios (Version 1.11). The controller is an entry level RAID0/1 card based on the LSI SAS2008 chipset, very similar to the LSI 9240-8I controller, yet without RAID5. The controller's firmware is what LSI calls MegaRaid firmware. Unfortunately, there is no appropiate driver for this firmware in FreeBSD at the time of this writing.
The ordeal
As delivered, with the default BIOS settings, FreeBSD boots from CD, but is not able to see the disks, no mater what. If you want to boot it off a USB stick, you'll have to wait for the SAS controller's prompt to appear before plugging it in, or the system will hang.
Now, as discussed in several forums, the M1015 can be re-flashed to what LSI calls "IT" (initiator-target) mode, and then, used as a high performance 6Mbps SAS controller with the mps driver, which is what you want if you are going to use ZFS. You will find everything you'll need to know here.
http://lime-technology.com/forum/index.php?topic=12767.msg124393#msg124393
I used the latest P11 firmware for the 9210-8i from the LSI site, as it seems to be the very same hardware as the M1015. The re-flash was done booting from a DOS Usb stick on the x3550 itself. Do not forget to take notice of your controller's SAS Address before re-flashing it! After a successful re-flash, you should see the LSI controller at boot-up, and be able to enter the controller's configuration settings by pressing CTRL-C while the controller boots.
If you try to install FreeBSD right away, you will be disappointed. The boot process waits for ever with this message:
Code:
run_interrupt_driven_hooks - waiting for xpt_config
Now, to the BIOS. The x3550 has an UEFI bios, however, the LSI 9210-8i firmware is not UEFI aware (the original M1015 is). I have reasons to believe that the BIOS is not able to properly enumerate/assign resources to the controller in this mode, see below.
I tried every switch I could find in the BIOS, with 8.2-STABLE as well as 9.0-BETA3 to no avail, until I arrived at the following configuration:
Code:
Main menu-> System Settings -> Devices and I/O Ports
Enable/Disable onboard devices:
Everything disabled but the SAS controller
Enable/Disable device option ROM:
Under Legacy, everything disabled but the SAS controllers, under UEFI, everything disabled.
Set Option ROM execution order:
SAS Controller first
Main menu-> System Settings -> Legacy Support
Rehook Int 19h -> Enable
Legacy thunk support -> Disable
Main menu-> Boot Manager
Primary boot options
Add boot option: Legacy only
Delete boot option: everything else
I then performed the documented ZFS install procedure, as shown in
http://www.aisecure.net/2011/05/01/root-on-zfs-freebsd-current/
Rebooted, enabled the Broadcom ethernet controllers in the BIOS under
Main menu-> System Settings -> Devices and I/O Ports
And got a fully functional FreeBSD-9 + ZFS root installation
To do:
1. Go through the BIOS again to find out the minimum set of changes that allows the installation to proceed
2. Try the integrated "lights-off" remote administration facility
Of note:
1. Booting FreeBSD of a USB stick is a hit and miss affair on this system, you'll be better off using a DVD
2. Raw disk access to my 500GB 7.2K SAS disks seems to be about 10% faster under FreeBSD than under Linux
3. The same procedure should apply to the very similar x3650 M3 when using an M1015 SAS adapter