HP Gen8 Microserver: Booting from a separate bootloader?

[stoe]

Dear Forum

Not having found any helpful information on the internet (and beeing quite new to FreeBSD, so maybe I haven’t looked for the relevant terms), I would like to ask you for some advice. I’m installing FreeBSD 10.3-Release on an HP ProLiant Gen8 Microserver, a system with four drive bays plus one ODD SATA channel. I intend to run the OS from a ZFS pool on an SSD attached to this ODD SATA port.

Due to some HP specific limitations, I run into problems with this setup: the microserver is equipped with an onboard HP Dynamic Smart Array B120i RAID-Controller I can turn on or off in BIOS. When turned off, i.e. in AHCI mode, the system will not boot any OS from the ODD SATA Port, while when turned on, FreeBSD will not boot correctly from the SSD (unlike e.g. Debian) but produce a BTX loader error and halt as described in Thread 46457. A system with the B120i controller turned off will boot an attached USB flash drive, though; but for reliability reasons I don’t want to run the OS off such hardware.

The workaround for this situation is to turn off the B120i controller, to install a separate bootloader on a USB flash drive connected to the server’s motherboard and to have this bootloader point to the OS on the SSD. I’ve found articles and instructions on how to achieve this for several OS’s, but not for FreeBSD. The closest I’ve found may be this answer on stackexchange, covering the solution for Solaris. Still I don't know how to translate this answer to FreeBSD. So my questions are:

• Has anyone of you successfully set up an HP Gen8 Microserver to boot FreeBSD off the ODD SATA port and if so, may I ask how you did this?
• Alternatively, how would I install a correctly configured bootloader on a USB drive? Would I have to install a full base system on this drive (i.e. clone of the actual OS installation) or just the required boot code?

Any input is appreciated very much. Thank you in advance.

stoe

 

[ondra_knezour]

You may want to look at boot(8) and the BOOTSTRAPING section of gpart(8)

 

[stoe]

You may want to look at boot(8) and the BOOTSTRAPING section of gpart(8)

Thank you ondra_knezour, I have been for quite a while now and I think the solution is somehow linked to gpart bootcode. It's just that I haven't figured out yet how to utilize it correctly and hoped that someone had done the same thing before.

Best regards
stoe

 

[wblock@]

I think Plop boot manager can do it. With a normal install on the SSD, install Plop on the USB and configure it to boot the SSD.

There might be a way to do that with FreeBSD bootcode, but I can't think of it.

 

[stoe]

Thank you wblock@, I think that's the way to go until I've improved my knowledge about FreeBSD quite a bit...
Doing some research on Grub2 now as well.

Best regards
stoe

 

[tsarya]

Take a look at the FreeBSD Boot Process section of the handbook and the boot0cfg(8).
I also have a microserver Gen8 and I need to try it out because in AHCI mode, the system will not boot from the OOD port when you have a drive in any of the 4 drive bays.

Update:
I tried the following in a VM and it appears to be working:
1) I created a FreeBSD 11-RC2 VM with 4 drives (ada0 through ada3)
2) Installed the bare minimum (base + kernel) on ada0, MBR scheme, 1 partition, 1fs for the /, no swap
3) Installed FreeBSD 11-RC2 on ada3 (MBR, BSD, one partition for /, one for swap)
4) Booted from ada0 (this could be the SD card, da0 on the HP microserver) and I issued the following command to install boot0 and set the 4th drive as the default boot drive (ada3):

# boot0cfg -Bv -o setdrv -d 0x82 ada0

The result is this:

After reboot, I get this:

Now the fun part is try this on the real machine, having a good backup!!!

 

[stoe]

Hi Tsarya

Thank you for your post; good to know I'm not the only one concerned
Kudos for your research on the topic. For now I'm focussing on setting up a quick solution, i.e. with hardware involved rather than boot code: I've just bought a USB to SATA cable (like this one on amazon) and I plan to connect the SSD in the ODD bay directly to the motherboard's USB2 port, instead of keeping it connected to the ODD's SATA port and booting it using code on a flash drive on the motherboard's USB port. If this works, I hope this gives me time to learn and research a bit more, like about the downsides of running an OS over USB2... don't know yet what impact the system disk's speed has for a storage server.
BTW: I thought about the possibility to abuse FreeBSD's mini-memstick.img on a USB flash drive for booting FreeBSD on the SSD. Do you think this could be done by adjusting the image's content?
Good luck! Looking forward to reading from you.

Best regards
stoe

 

[tsarya]

Hi Stoe,

about the FreeBSD's mini-memstick.img on a USB flash, I can't tell, haven't played with it at all.

I tried what I wrote in my previous post on my microserver yesterday but it did not work out well with the boot0 mangling. It does boot once, after that the MBR record on the SD card becomes corrupted. Not sure what really happens. Maybe one day I will dig into Grub2
I also tried connecting an external drive the front usb ports of the microserver, setup the first boot device to "external usb flash drive" and it works well. The speed of usb2 is great, but for loading the OS is perfect.
Then you could use the 5th sata port (ODD) for a SSD for the L2ARC/ZIL.

Regarding your idea with the adapter, I think it's great. Please keep me posted on the outcome.

 

[wblock@]

don't know yet what impact the system disk's speed has for a storage server

None. Don't put swap on it, though.

 

[tsarya]

Hello again Stoe,

I managed to make the system boot from the OOD port using Grub2 installed on a USB thumb drive (plugged in the modo usb port).

This is how I did it, probably not the most elegant way, but at least it works!

• install FreeBSD on the usb thumb drive (MBR, UFS, only one BSD partition for /). install FreeBSD on the fifth drive (same layout, but with added swap). The result:

# camcontrol devlist
<WDC WD10EZRX-00A8LB0 01.01A01>  at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD10EZRX-00DC0B0 80.00A80>  at scbus1 target 0 lun 0 (ada1,pass1)
<WDC WD10EFRX-68FYTN0 82.00A82>  at scbus2 target 0 lun 0 (ada2,pass2)
<WDC WD10EFRX-68FYTN0 82.00A82>  at scbus3 target 0 lun 0 (ada3,pass3)
<ST9250410AS 0002SDM1>  at scbus4 target 0 lun 0 (ada4,pass4) <= this is my 2.5 hdd for this test
<AHCI SGPIO Enclosure 1.00 0001>  at scbus6 target 0 lun 0 (ses0,pass5)
<Generic Flash Disk 8.07>  at scbus7 target 0 lun 0 (da0,pass6)

# gpart show da0s1
=>  0  16099263  da0s1  BSD  (7.7G)
  0  16099262  1  freebsd-ufs  (7.7G)
  16099262  1  - free -  (512B)

# gpart show ada4s1
=>  0  488397103  ada4s1  BSD  (233G)
  0  480247808  1  freebsd-ufs  (229G)
  480247808  8149294  2  freebsd-swap  (3.9G) <= set your swap size as appropriate, I picked the default for the example
  488397102  1  - free -  (512B)

• install sysutils/grub2 using pkg install grub2
• once installed, run grub-install /dev/da0. You will see a folder 'grub' under boot created.
• run grub-mkconfig -o grub.cfg. This will create a file named grub.cfg.new under /boot/grub. You will see several warnings saying the syntax of the file is wrong. MAKE SURE you fix those before you reboot!
• make a copy of the file -> cp -p grub.cfg.new grub.cfg
• edit grub.cfg, first to fix the errors, second to add the entry for booting from the drive connected to the OOD. What you will notice is that some lines start with 't' or 'tt', which simply does not make sense.
• add a menu entry right under "### BEGIN /usr/local/etc/grub.d/10_kfreebsd ###". What you need to do is to replace the is the UFSID with the one from the ufs on your drive (the one connected to the OOD port). Replace in all lines '57c9d7617c48682b'. Mine looks like this:

### BEGIN /usr/local/etc/grub.d/10_kfreebsd ###
menuentry 'FreeBSD (Main from OOD port)' --class freebsd --class bsd --class os $menuentry_id_option 'kfreebsd-simple-57c9d7617c48682b' {
insmod part_msdos
insmod ufs2
set root='(hd5,msdos1,bsd1)'
if [ $feature_platform_search_hint = y ]; then
  search --no-floppy --fs-uuid --set=root  57c9d7617c48682b
else
  search --no-floppy --fs-uuid --set=root 57c9d7617c48682b
fi
  echo  'Loading FreeBSD kernel from the right place...'
  kfreebsd  /boot/kernel/kernel
  kfreebsd_loadenv  /boot/device.hints
insmod part_msdos
insmod ufs2
if [ $feature_platform_search_hint = y ]; then
  search --no-floppy --fs-uuid --set=root  57c9d7617c48682b
else
  search --no-floppy --fs-uuid --set=root 57c9d7617c48682b
fi
  kfreebsd_module_elf  /boot/kernel/ufs.ko
  set kFreeBSD.vfs.root.mountfrom=ufs:ufsid/57c9d7617c48682b
  set kFreeBSD.vfs.root.mountfrom.options=rw
}
....

When you first boot from the usb drive, at the Grub menu, hit 'c' to fall into command prompt and type ls -l to see the the disk layout and the UFSID. Since we have the same hw and I suppose you have all drive bays populated, you should see: (hd5,msdos1,bsd1). This is what you'll use for the 'set root=' directive.

• save the grub.cfg and if everything went well, then you should be able to boot from the OOD port. In the BIOS, set the USB boot first, then set "Internal USB key" as the first usb boot device.

Cheers!

 

[tsarya]

I continued exploring the possibilities and I managed to boot from a GPT/ZFS drive, this time with sysutils/grub2-pcbsd.

Here are the steps:
1) Install FreeBSD on a thumb drive as explained in my previous post (MBR, one slice, one BSD partition for /)
2) Install FreeBSD on the 5th drive (better unplug the other 4 four drives). I used the default ZFS installation option, which uses GPT and creates 3 partitions, one for boot, one for swap (you may need to adjust the size) and one for the zroot pool.
3) Boot from the USB drive and install sysutils/grub2-pcbsd:
pkg install grub2-pcbsd
The port will tell you that in case you will use boot environments, you must install sysutils/beadm. Let's do that, why not:
pkg install beadm
Once complete, run rehash in case you still use csh, then install grub on the thumb drive, sd card, whichever you use:
grub-install /dev/da0.
If you have more that one removable device connected, you better check the node name before executing the command. If there are no errors, you should see:

# grub-install /dev/da0
Installing for i386-pc platform.
Installation finished. No error reported.

4) Now you're ready to configure grub:

# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
done

5) BEFORE YOU REBOOT, adjust the grub.cfg file. There are a few things you need to edit before using it 'as' is (below is only a part of the file):

### BEGIN /usr/local/etc/grub.d/10_kfreebsd ###
menuentry "FreeBSD - Chainload Disk" {
  set root=(hd5)
  chainloader +1
}
menuentry 'FreeBSD (BOOT)' --class freebsd --class bsd --class os $menuentry_id_option 'kfreebsd-simple-57c9fba3b2a0ef3e' {
insmod part_msdos
insmod ufs2
   search --no-floppy --fs-uuid --set=root 57c9fba3b2a0ef3e
   echo       'Loading kernel of FreeBSD kernel ...'
   kfreebsd     /boot/kernel/kernel
   kfreebsd_loadenv   /boot/device.hints
insmod part_msdos
insmod ufs2
   search --no-floppy --fs-uuid --set=root 57c9fba3b2a0ef3e
   kfreebsd_module_elf   /boot/kernel/ufs.ko
   set kFreeBSD.vfs.root.mountfrom=ufs:ufsid/57c9fba3b2a0ef3e
   set kFreeBSD.vfs.root.mountfrom.options=rw
}
### END /usr/local/etc/grub.d/10_kfreebsd ###

• 57c9fba3b2a0ef3e - this is the UUID of the UFS partition, you need to replace with yours.
  
• set root=(hd5) - this will change if you have less than 4 drives in the front bays. Counting starts from 1 => hd1 - first disk, hd2 second and so on...

I am trying to upload the grub.cfg but for some reason the forum is refusing to do so. I can email it to you if you decide to go this way, just let me know.
...and I guess that's all. My HP Microserver Gen8 is configured this way, no issues at this point.

 

[stoe]

None. Don't put swap on it, though.

Thank you. None at all or just not on an SSD (or a ZFS pool)? I let the installer do its magic and thus I do have a swap partition on the zroot...

 

[stoe]

I continued exploring the possibilities and I managed to boot from a GPT/ZFS drive, this time with grub2-pcbsd.

Here are the steps:


I am trying to upload the grub.cfg but for some reason the forum is refusing to do so. I can email it to you if you decide to go this way, just let me know.
...and I guess that's all. My HP Microserver Gen8 is configured this way, no issues at this point.

Hi tsarya

Wow, seems as if you've been busy. Thank you for your documentation and for sharing it with us, I appreciate that very much.
Today finally my USB adapter arrived and I plan to do some testing now. In case I stumble upon problems I cannot resolve, I think I might just follow your excellent instructions.
For now I'd like to avoid setting up a bootloader with "relative addresses" like hd5; you are correct about my intentions to populate all drive bays in the microserver, but I just don't know yet how the system will react in case a drive fails and I have to pull it -> only 3 drives left -> unbootable system until grub.cfg is adjusted using a rescue | live system. After replacing the drive, grub.cfg will have to be altered back again, right? I would prefer to use "absolute addresses" instead like e.g. the UUID of a UFS partition, but with the ZFS drive as in your setup. So there's still some research to be done by me.

Anyway I'll keep you updated about how things go.

Thank you again and best regards
stoe

 

[wblock@]

Thank you. None at all or just not on an SSD (or a ZFS pool)? I let the installer do its magic and thus I do have a swap partition on the zroot...

A USB SSD or hard drive would be fine for speed. USB memory sticks write very slowly, though, so it would not be suited for swap.

 

[stoe]

A USB SSD or hard drive would be fine for speed. USB memory sticks write very slowly, though, so it would not be suited for swap.

Ah ok, thank you. No intention to use a USB stick, instead I will connect the SATA SSD to the motherboard's USB port.
So I can keep the swap space as set up by FreeBSD's installer.

 

[stoe]

Please keep me posted on the outcome.

Hi tsarya

Just a quick update: it's working great. Encrypted FreeBSD 10.3 RELEASE on a SATA SSD connected to the motherboard's USB port, and the system boots in AHCI mode without any problem.
Now to the work with the server ;-) I'll let you know about the results.

Thank you again for your care.

Best regards
stoe

 

[tsarya]

Glad to hear it stoe!
The adapter that you bought looks pretty good, I will also buy myself one to have a backup solution for connecting 2.5" drives.

For now I'd like to avoid setting up a bootloader with "relative addresses" like hd5; you are correct about my intentions to populate all drive bays in the microserver, but I just don't know yet how the system will react in case a drive fails and I have to pull it -> only 3 drives left -> unbootable system until grub.cfg is adjusted using a rescue | live system. After replacing the drive, grub.cfg will have to be altered back again, right?

As for the 'relative addresses', you are correct, if a drive from the main bays is unplugged and the system is rebooted, grub will start counting from 1, hence the drive in the OOD port will no longer be 'hd5'. I don't think you'll need to boot from a rescue CD to fix it, rather you will have to set the drive as root drive in the grub's command line in order to boot from it. I haven't tried it though.
My backup solution is the OS on the USB drive, you can always boot from it, fix grub in case your replacement drive does not arrive quickly ;-)

Regards,

 

[stoe]

rather you will have to set the drive as root drive in the grub's command line in order to boot from it.

I agree on that and that's exactly what I would need a working OS for.

My backup solution is the OS on the USB drive, you can always boot from it, fix grub in case your replacement drive does not arrive quickly ;-)

I think in case of need I'd prefer to boot off some .iso (over iLO) instead of disassembling the server and re-connecting the system disk twice. But never mind, it's just theory for now.

Looking forward to reading about your progress.

Regards
Stoe

 

[tsarya]

I think in case of need I'd prefer to boot off some .iso (over iLO) instead of disassembling the server and re-connecting the system disk twice.

iLO is definitely "the last line of defense" but what I meant was that I have FreeBSD installed on the USB stick, which under normal circumstances is never used. Its only purpose is to install sysutils/grub2-pcbsd and configure it in the first place, then I boot (chainload) the OS from the drive in the OOD port. So if I need to remove a disk due to a failure, I can always boot the OS on the USB stick (kinda like a rescue OS), modify grub.cfg until I get my new replacement drive, no additional disassembling is required.

 

[stoe]

Hi tsarya

I can see your point now;-). I've allready left the idea of a 2nd full OS installed in the system, but you're right: that sounds like a big advantage of the USB boot stick with its own FreeBSD installation.
But this discussion is leaving the FreeBSD subject. I will report again as soon as I'm progressing with the server's purpose: holding data reliably. I was inspired by this article, btw: https://www.skelleton.net/2015/06/14/how-to-built-a-fully-encrypted-file-server-with-zfs-and-linux/
Just the "wrong" OS ;-)

Best regards
stoe

 

[Phishfry]

I would make sure you don't install FreeBSD on the USB drive with a swap partition.

I would like to try dual USB sticks in a RAID1 mirrored arrangement for reliability for cases like yours with limited SATA ports.
They also sell USB-DOM made for internal usage.

Are you aware of the mfsBSD install? This would probably be a better way if you can live with its minimalism.

 

[rickvanderzwet]

I managed to load the SSD in the ODD slot using an microSD card installed in the slot inside the system and use the standard BSD bootloader to load the correct disk.

After the instalation, which I did using an USB memstick image, I switched to the live editor and used the following steps:

Create an partioning on the microSD card (mine was blank):
gpart create -s GPT da1

Install the boot0 (first & second stage) bootloader:
gpart bootcode -b /boot/boot0 da1

Configure bootloader to point to the 3rd disk (-o setdrv -d 0x82) in the system and make sure it boots this disk by default (-s 5):
boot0cfg -Bv -o setdrv -d 0x82 -s 5 da1

I have 2 drives installed in the bays, thus my SSD in the ODD SATA channel is the 3rd disk. This will be enumerated in the BIOS as 0x82 (0x80 is the first disk, 0x81 the second disk, 0x82 the third disk, 0x83 the fourth disk and 0x84 the fifth disk). So do mind if you have more disks installed in your system to select the proper -d flag.

 

[tsarya]

After stripping down the /boot/grub/grub.cfg, this is the final working result:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /usr/local/etc/grub.d and settings from /usr/local/etc/default/grub
#
### BEGIN /usr/local/etc/grub.d/00_header ###
insmod part_gpt
if [ -s $prefix/grubenv ]; then
  load_env
fi

if [ "${feature_menuentry_id}" = y ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

function load_video {
  insmod vbe
  insmod vga
  insmod video_bochs
  insmod video_cirrus
}

if loadfont $font ; then
  if [ "${grub_platform}" = pc ] ; then
    set gfxmode=auto
    load_video
    insmod gfxterm
  else
    # EFI
    insmod efi_gop
    insmod gfxterm
    insmod font
    insmod videotest
    insmod videoinfo
    set gfxmode=auto
  fi
  set locale_dir=$prefix/locale
  set lang=
  insmod gettext
fi
terminal_output gfxterm
set default="0"
set timeout=3
### END /usr/local/etc/grub.d/00_header ###
### BEGIN /usr/local/etc/grub.d/10_kfreebsd ###
menuentry "FreeBSD (MAIN) from HDD on ODD port" {
   set root=(hd5)
   chainloader +1
}
menuentry "FreeBSD (BOOT) from SD Card" {
   set root=(hd0,1)
   chainloader +1
}
### END /usr/local/etc/grub.d/10_kfreebsd ###

 

[funkz]

I just got a Gen8 to upgrade my older mid-tower server. I transferred the SSD with FreeBSD on it over and plugged it into the ODD port. Without any other hard drives, it booted fine. But when I transferred my 3 x HDDs (ZFS RAID-Z) over to the hotswap slots, it failed to boot (no OS). Mucked around with bios settings but no ability to set the boot order on the drives.

In my old server I had a cheap PCIe SATA3 card like this as the mobo only did SATA2. So I figured I transfer that over, first unscrewed the metal backplate so it fit in (as the PCI slot is a mini height type). Plugged the SSD into that, then in the bios I set the boot order for that SATA controller to be first. Done, boots good. I would prefer to utilise the the onboard SATA adaptor (Intel Cougar Point) but, without the above use of USB/SD cards and GRUB, this is the only alternative I saw to get it up and running.