1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Installing FreeBSD 9 on the IBM System x3550 M3

Discussion in 'System Hardware' started by rdeiriar, Oct 26, 2011.

  1. rdeiriar

    rdeiriar New Member

    Messages:
    18
    Thanks Received:
    5
    A couple of days ago, we received a brand new IBM x3550 M3 model 7944AC1. My goal for this system was to do a simple mirrored ZFS root install. As there are some non-conclusive threads on whether this system works at all, I decided to document how I got it to work.

    This system has an embedded IBM M1015 SAS Raid controller on a dedicated PCIe slot, as well as an UEFI bios (Version 1.11). The controller is an entry level RAID0/1 card based on the LSI SAS2008 chipset, very similar to the LSI 9240-8I controller, yet without RAID5. The controller's firmware is what LSI calls MegaRaid firmware. Unfortunately, there is no appropiate driver for this firmware in FreeBSD at the time of this writing.

    The ordeal

    As delivered, with the default BIOS settings, FreeBSD boots from CD, but is not able to see the disks, no mater what. If you want to boot it off a USB stick, you'll have to wait for the SAS controller's prompt to appear before plugging it in, or the system will hang.

    Now, as discussed in several forums, the M1015 can be re-flashed to what LSI calls "IT" (initiator-target) mode, and then, used as a high performance 6Mbps SAS controller with the mps driver, which is what you want if you are going to use ZFS. You will find everything you'll need to know here.

    http://lime-technology.com/forum/index.php?topic=12767.msg124393#msg124393

    I used the latest P11 firmware for the 9210-8i from the LSI site, as it seems to be the very same hardware as the M1015. The re-flash was done booting from a DOS Usb stick on the x3550 itself. Do not forget to take notice of your controller's SAS Address before re-flashing it! After a successful re-flash, you should see the LSI controller at boot-up, and be able to enter the controller's configuration settings by pressing CTRL-C while the controller boots.

    If you try to install FreeBSD right away, you will be disappointed. The boot process waits for ever with this message:
    Code:
    run_interrupt_driven_hooks - waiting for xpt_config
    Desperate, thinking I had fried the SAS controller, I did a quick install of Ubuntu 10.04, which installed correctly, yet refused to boot as well.

    Now, to the BIOS. The x3550 has an UEFI bios, however, the LSI 9210-8i firmware is not UEFI aware (the original M1015 is). I have reasons to believe that the BIOS is not able to properly enumerate/assign resources to the controller in this mode, see below.

    I tried every switch I could find in the BIOS, with 8.2-STABLE as well as 9.0-BETA3 to no avail, until I arrived at the following configuration:

    Code:
    Main menu-> System Settings -> Devices and I/O Ports
    
      Enable/Disable onboard devices: 
       Everything disabled but the SAS controller
      Enable/Disable device option ROM: 
       Under Legacy, everything disabled but the SAS controllers, under UEFI, everything disabled.
      Set Option ROM execution order: 
       SAS Controller first
    
    Main menu-> System Settings -> Legacy Support
      Rehook Int 19h -> Enable
      Legacy thunk support -> Disable
    
    Main menu-> Boot Manager
      Primary boot options
      Add boot option: Legacy only
      Delete boot option: everything else
    

    In this configuration, I was finally able to boot the installed Ubuntu 10.04, and, even better, I was able to boot FreeBSD 9 off both USB and DVD, with fully functional disks.

    I then performed the documented ZFS install procedure, as shown in

    http://www.aisecure.net/2011/05/01/root-on-zfs-freebsd-current/

    Rebooted, enabled the Broadcom ethernet controllers in the BIOS under

    Main menu-> System Settings -> Devices and I/O Ports

    And got a fully functional FreeBSD-9 + ZFS root installation

    To do:

    1. Go through the BIOS again to find out the minimum set of changes that allows the installation to proceed
    2. Try the integrated "lights-off" remote administration facility

    Of note:

    1. Booting FreeBSD of a USB stick is a hit and miss affair on this system, you'll be better off using a DVD
    2. Raw disk access to my 500GB 7.2K SAS disks seems to be about 10% faster under FreeBSD than under Linux
    3. The same procedure should apply to the very similar x3650 M3 when using an M1015 SAS adapter
     
    Sylgeist thanks for this.
  2. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    I am having a new LSI MegaRAID SAS 9240-8i card on my PowerEdge server. R710 and R610 default H700 card works fine with FreeBSD 8.x and 9.0 RC1/RC2 but this new card has issues - no hard disk detected. Read through this post. Tried rebuilding the GENERIC kernel without mfi driver and inserting the new mfi driver from LSI website (http://pastebin.com/wVyWCVc2), it didn't help (http://pastebin.com/ykAf0Q1j). Created own patch, FreeBSD ISO through CVSUP but it lasted with mfi errors. Now when I read this post I got to know about reflashing in IT Mode. How shall I proceed with Dell PowerEdge server. Are those commands (which you mentioned) restricted to IBM servers. Please suggest.
     
  3. rdeiriar

    rdeiriar New Member

    Messages:
    18
    Thanks Received:
    5
    The same procedure should apply to your LSI MegaRAID SAS 9240-8i, and, if you intend to use ZFS, you will be rewarded with an very fast disk subsystem. Please be absolutely sure to make a backup of your current firmware before you reflash the controller, should you need to reflash the card back. One big question is whether the bios of your system will recognize the reflashed card; some vendors implement bios-level restrictions in order to force you to use only their "approved hardware". I know for sure IBM does not; i have no idea however whether Dell does.
     
  4. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    Thanks for the comments.
    What commands and software do i need to reflash the controller.As I confirm its LSI MegaRAID SAS 9240.I am completely new to Firmware reflashing.Also, Why ZFS is being talked here?
    Please it would be great if you can help me with reference or docs if any for PowerEdge server.
     
  5. RodLophus

    RodLophus New Member

    Messages:
    10
    Thanks Received:
    2
    LSI 9240-4i working ok on FreeBSD 9.0-RC2/PRERELEASE

    Hi!

    I have a LSI 9240-4i working ok on FreeBSD 9.0-RC2/PRERELEASE.

    FreeBSD 9.0 mfi driver does not support this board (the drivers attached to it are not detected), so I got the driver at LSI's page (for FreeBSD 7.x / 8.x - 64-bit version) and changed it to work with 9.0. It looks to be working well.

    If you want to give it a try, here is what I have done (note: I am supposing all your disks are attached to LSI's board - so you see no disks after a normal boot from FreeBSD 9.0 installation CDROM):
    - Get the modified driver at http://www.ime.unicamp.br/~rodolfo/files/mfi_freebsd90_x64_rbm.tgz
    - Boot from CDROM and install FreeBSD 9.0 on a USB drive (flash or external drive). Don't forget to install the sources, and don't install the ports tree.
    - Boot from the USB device
    - Replace the /usr/src/sys/dev/mfi directory with the one on the file
    - Follow the instructions on Readme.doc file to change the kernel configuration and /usr/src/sys/conf/files
    - Compile the new kernel (cd /usr/src/sys/amd64/conf; configure MYKERNEL; etc)
    - Boot. Your new installation (on USB drive) should be able to see your RAID disks now!

    Now you have a working system on a pendrive. To install on the harddisk:
    - Boot from Installation CDROM
    - Escape to the loader prompt
    - Use unload kernel command to unload the kernel from the CDROM
    - Use load xxxxxx:/boot/kernel/kernel to load the kernel from the USB device (use the lsdev command to figure out what device to use in place of "xxxxxxx"). So use the boot command to boot the installation CDROM
    - Install FreeBSD as usual.
    - Boot from the RAID HD and escape to the loader prompt again. The kernel you have just installed on the system's HD is a standard "GENERIC" kernel - without support to your LSI controller
    - Use the unload kernel / load kernel / boot trick again to boot your system using the kernel from the USB device (with root on HD)
    - Replace the mfi directory and reconfigure your kernel (the one on HD) as you did before. Rebuild and install the new kernel.
    - Reboot. Now you have a working system.

    Note:
    1. if you update your source tree (via cvsup), the mfi directory will be overwritten - and you will need to replace it again if you want to build a new kernel. Don't forget to change the /usr/src/sys/conf/files file!
    2. This driver is experimental. Use at your own risk
    3. This driver is based on FreeBSD 8.x architecture. It lacks the new FreeBSD 9.x features and the mfitool utility will fail to talk to it. Use LSI's MegaCli istead.

    Cheers,
     

    Attached Files:

    linuxunix and torzak thanked for this.
  6. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    RodLophus,

    Thanks a lot for your points.
    Few important doubts:-

    1. I am unable to read Readme.doc(as it displays lots of ^@ in FreeBSD). So I referred this from LSI.
    If the above is incorrect, do let me know.
    Code:
    1)	Make sure that kernel source has been installed on the system. If the kernel sources have been installed then there will be /usr/src/sys directory.
    2)	Move to the arch/conf directory and copy the GENERIC configuration file to the name you want to give your kernel. For example:
                # cd /usr/src/sys/i386/conf
                For 64 bit the path is
                # /usr/src/sys/amd64/conf			
    	# cp GENERIC MYKERNEL
    3)	Perform the following steps.
    a) Comment the following line in the file MYKERNEL by placing # at the      
         beginning of the line.
         device     cbb    #cardbus (yenta) bridge
    b)  Add the following line to the file /usr/src/sys/conf/files 
        dev/mfi/mfi_syspd.c optional mfi
      (Note add this line next to the line which has dev/mfi/mfi_cam.c optional mfi)
    4)	Copy the zip (mfi.zip) file under the path /usr/src/sys/dev (Before proceeding to next step make sure that the directory with name “mfi” does not exist under path /usr/src/sys/dev/. If already a directory with name “mfi” exists, move it to some other folder. For example type, mv /usr/src/sys/dev/mfi /root/mfibackup).
    5)	Go to the directory /usr/src/sys/dev and run the following command to unzip the source.
          tar –x –f mfi.zip
    6)	To compile the kernel go to /usr/src directory and type
           make buildkernel KERNCONF=MYKERNEL
    7)	To install the new kernel type
           make installkernel KERNCONF=MYKERNEL
    
    


    2.You mentioned:
    In my case, I have a Installation-CD on my CD-Drive, disk0(BIOS drive C: which refers to RAID HD) and disk1(BIOS drive D: with partitions disk1p1, disk1p2 and disk1p3, USB partitions).


    I ran load disk1p2 (corresponding to FreeBSD UFS partition): /boot/kernel/kernel and it went well. Next I ran boot command. It proceeded but during the installation, it showed me just da0 (pertaining to USB drive) during the partioning.
    Didn't it detect the RAID HD? :(

    When I am trying to boot from CD-Drive ISO, performed the loading of /boot/kernel/kernel,load mps, load mfi, boot it stopped at:

    Code:
    mountroot> 
    
    <fstype>:<device> option 
    I provided with:
    ufs:/dev/disk1 
    

    and it rebooted.

    No idea if it really recognized my disk. All I can see it does detect during the Loader Prompt.

    Please suggest.
     
  7. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    I got this working now.
    As you mentioned:
    I tried building CVS Repo, replaced with your mfi driver on my 8.2-RELEASE system, but it threw the error:
    Code:
    /usr/src/usr.sbin/mfiutil/mfi_evt.c: In function 'mfi_decode_evt':
    /usr/src/usr.sbin/mfiutil/mfi_evt.c:364: error: 'struct mfi_evt_detail' has no member named 'class'
    /usr/src/usr.sbin/mfiutil/mfi_evt.c:365: error: 'struct mfi_evt_detail' has no member named 'class'
    /usr/src/usr.sbin/mfiutil/mfi_evt.c: In function 'show_events':
    /usr/src/usr.sbin/mfiutil/mfi_evt.c:557: error: 'struct <anonymous>' has no member named 'class'
    /usr/src/usr.sbin/mfiutil/mfi_evt.c:567: error: 'struct <anonymous>' has no member named 'class'
    *** Error code 1
    
    Stop in /usr/src/usr.sbin/mfiutil.
    *** Error code 1
    
    Stop in /usr/src/usr.sbin.
    *** Error code 1
    
    Stop in /usr/src.
    *** Error code 1
    
    Stop in /usr/src.
    *** Error code 1
    
    Stop in /usr/src.
    + exit 1
    + umount /dev
    *** Error code 1
    
    Stop in /usr/src/release.
    


    Any idea how shall I fix this?
     
  8. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    RodLophus,

    Just to add more points.
    I faced the above issue when I tried this command:
    Code:
    make release RELEASETAG=RELENG_8_1 PORTSRELEASETAG=HEAD BUILDNAME=8.1-RELEASE CHROOTDIR=/usr/home/releng NODOC=yes CVSROOT=/usr/home/ncvs LOCAL_PATCHES=/root/patch.mfi
    


    It would be great if you can suggest me on the above thread.
     
  9. RodLophus

    RodLophus New Member

    Messages:
    10
    Thanks Received:
    2
    MegaRAID SAS 9240 (LSI2008) on FreeBSD 8.2

    Hi, linuxunix!

    For FreeBSD 8.2 (I didn' test 8.1), the procedure are slightly different:

    1. The file I mentioned before will not work. Use the one at http://www.ime.unicamp.br/~rodolfo/files/mfi_fbsd82_x64_rbm.tgz instead.

    2. The "unload / load" trick doesn't work with 8.x: it will unload the mfs too and the new kernel won't load it again (once booted, the kernel will drop to the "mountroot>" prompt). Istead, escape to the loader prompt, unload the kernel (command: "unload"), set the "kernel" variable so it points to the patched kernel on your USB stick and use the "boot" command to boot the system - like this:

    Code:
    OK unload
    OK set kernel=disk0p1:/boot/kernel/kernel
    OK boot


    Note: the device (disk0p1) is not the UDF partitions on your hard drive: it is the partition on the USB memory stick where you have your patched kernel. Use the "lsdev" command to figure out wich device to use.

    At this point, you should have a system booting from your hard disk, but using the kernel on the USB memory stick. Now, you can patch and re-compile the kernel on your hard disk the same way you did with the one on the memory stick - and you will end up with a fully functional system.

    BTW: The "Readme.doc" file is not plain text: it is a Windows Word file. It is the same file supplied with the original driver from LSI. You should be able to read it with MSWord, OpenOffice, Lotus Symphony, etc. The instructions on it are exactly the one you wrote in your post.

    I hope it can help you.

    Have fun!
     

    Attached Files:

  10. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    Thanks.

    I will try it out on 8.2. As of now, I am setting up CVS Repo for building FreeBSD ISO for 9.0 RC1. Hope it works.

    It would be great if you can help me with /root/patch.mfi (since it seems like it always failed for me).
     
  11. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    Also, you mentioned:
    I didn't get the above statement.Wont I build up a new ISO through CVS for mfi driver.

    All I am doing is following:
    1. make release RELEASETAG=RELENG_8_2 PORTSRELEASETAG=HEAD BUILDNAME=8.1-RELEASE CHROOTDIR=/usr/home/releng NODOC=yes CVSROOT=/usr/home/ncvs LOCAL_PATCHES=/root/patch.mfi
    2. sh /usr/src/release/amd64/mkisoimages.sh -b custom-FreeBSD82-amd
    /usr/home/8.2-RELEASE-dvd1.iso /usr/home/releng/R/cdrom/dvd


    Please suggest if I am wrong or I need to do anything else.
     
  12. DutchDaemon

    DutchDaemon Administrator Staff Member Administrator Moderator

    Messages:
    10,728
    Thanks Received:
    2,130
  13. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    One quick question, why the mfi driver provided by LSI din't work while the modified version worked?It would be great if you can brief me about the same. This is just for technical understanding for self-help purpose.
    Thanks.
     
  14. RodLophus

    RodLophus New Member

    Messages:
    10
    Thanks Received:
    2
    Why LSI's driver doesn't works on FreeBSD 8.2 and 9.0

    Hi, linuxunix!

    I think the driver itself would work on FreeBSD 8.2. The problem is the interface with tools like mfiutil. The mfiutil tool on 8.2 has support for more interface commands, status querying and things link this - all of them are defined on the mfireg.h file. Also, there are one or two functions called from mfiutil whose names have changed.

    The changes was deeper on FreeBSD 9.0. There are function calls that changed names and parameter syntax - so the driver itself doesn't work. The interface of the driver with mfitool changed a lot too.

    To be honest, I didn't try to fix the interface between LSI's driver and mfitool. I just updated the definitions on the driver so it now matches the one expected by mfitool. It ends up that mfitool doesn't talk with this board (it returns "no such file or directory" for every command you try). I'm not concerned about this because LSI's MegaCli do all mfitool does - and a bit more.

    BTW, there was a syntax error on LSI's driver (search the original code for something like "PORGRESS_RECON"). It would not prevent the driver from compiling, but would, for example, stop a "make world" (when building mfitool). I guess even the original LSI driver might not actually work with mfitool.


    Regards!
     
  15. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    That was pretty good explanation.Thanks for the quick response.
     
  16. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    Rodpholus,

    It would be great if you can help me with ISO rather than couple of steps for making mfi work.
    Though I have FreeBSD running without any issue, I tried creating cvsup repo but the patch dint work.
    Can you help me with this?
     
  17. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    Any idea if the driver has been considered in the latest 9.0 GA?
    If not, how shall I raise a request for including it in FreeBSD tree?
    How shall I create a new FreeBSD ISO with this driver?
     
  18. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    What about LSI MegaRAID SAS 9265 card? I am unable to get this work?
    Do you have driver for that? This mfi driver which worked for 9240 is not working for 9265.
    Please suggest.
     
  19. torzak

    torzak New Member

    Messages:
    3
    Thanks Received:
    0
    It would be nice if this driver could be included in the next release (9.1, 8.4).
    Several Dell, IBM, Supermicro servers need this driver.

    Thank you.
     
  20. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
    Very true! How shall I go ahead and raise this request to FreeBSD community?
     
  21. RodLophus

    RodLophus New Member

    Messages:
    10
    Thanks Received:
    2
    SAS 9265 is LSI2208-based. I think DragonFly has a mfi driver for this board:

    http://www.shiningsilence.com/dbsdlog/category/device-support

     
  22. urosgruber

    urosgruber New Member

    Messages:
    29
    Thanks Received:
    0
    @rdeiriar

    I have the same hardware except he server is an x3560. I also try almost every BIOS setting but I can't get past the boot initialisation process of the server. If I set BIOS exactly as you did then the boot partition (GPT or MBR) is not recognized and the system starts booting from DVD. But if I enable Legacy thunk support or enable UEFI in 'Enable/Disable device option ROM' the system stops when trying to initialize boot devices. It helps to get past to unplug all the drives.

    But if disks are plugged after boot from DVD I can install FreeBSD without a problem.

    You're probably using GPT and I also tried with MBR. I've read somewhere that Legacy mode is not working with GPT.

    One thing I thought is to change the boot ROM on the controler. I found on LSI some other EFI package SAS2_UEFI_BSD_P11 and there are two ROMs, ebcsas2.rom and x64sas2.rom. Do you use any of these two or mptsas2.rom from the original LSI IT firmware?

    Regards

    Uros
     
  23. linuxunix

    linuxunix New Member

    Messages:
    73
    Thanks Received:
    0
  24. DutchDaemon

    DutchDaemon Administrator Staff Member Administrator Moderator

    Messages:
    10,728
    Thanks Received:
    2,130
    linuxunix, can you stop attaching mfi-related posts to every single topic on the forums, please?
     
  25. urosgruber

    urosgruber New Member

    Messages:
    29
    Thanks Received:
    0
    I can confirm that with the ebcsas2.rom file server boots normally and everything else works as it should. The only glitch remains long zfs boot when enumerating the drives.