Booting from an arbitrary disk in ZFS RAIDZ on 8.x

W

wiscodisco

Hello,

I'm running ZFS filesystem ver 3, storage pool ver 14, on 8-STABLE amd64. The kernel build is rather dated from around Feb 2010.

I have 6 disks in a RAIDZ configuration. All disks were sliced the same with gpart (da(n)p1,p2,p3) with bootcode written to index 1, swap on index 2 and freebsd-zfs on index 3.

Given this configuration, I should be able to boot from any of the 6 disks in the RAIDZ. If this is a true statement, how do I make that happen from the loader prompt?

At the loader prompt when I type show, I get the following relevant
variables:
Code: 
currdev="zfs0"
loaddev="disk1a:"
vfs.root.mountfrom="zfs:rpool"

A peek at loader(1) shows me two interesting variables:
Code: 
root_disk_unit
rootdev

If disk0 is the legacy floppy device, presumably disk1a maps to da0.
What variable do I set to signify boot from what I know as:
Code: 
da0p1 == gpt/boot0 
da0p2 == gpt/swap0 
da0p3 == gpt/disk0
...
da5p1 == gpt/boot5 
da5p2 == gpt/swap5 
da5p3 == gpt/disk5

Thanks very much in advance.
 
The way it works is that the there's a small boot loader on every disk (or at least should be) on the first GPT partition of type freebsd-boot that can load and execute /boot/loader from the ZFS pool. When you get the loader menu you are already past the point where you can choose the disk to boot from. You have to use the BIOS boot menu if available to choose the disk to boot from.
 
Apologies to phoenix for not formatting post...

Thanks kpa, the boxes in question are HP DL360/380G4 running HP Smart Array 6i. I don't recall a menu choice for which disk to boot from. Is there a work-around?
 
Untried and absolutely dangerous if something goes wrong, make backups and have a recovery system ready:

Use boot0cfg(8) to replace the protective MBR code with code that provides a simple boot menu before the boot loader on the freebsd-boot partition is launched.

# boot0cfg -B da0


Edit: You could boot from a small USB memory stick and install the boot0cfg(8) boot loader on its MBR. This way the disks that make up the ZFS pool do not have to be modified in any way.
 
If I recall correctly the boot manager doesn't support ZFS.
 
I'm thinking the boot0 loader should be able to chainload the MBR and the boot loaders from the selected disk so no support for GPT or ZFS needed.
 
Forgive my ignorance, but with 6 disks in one pool I don't see any reason to want to boot from another disk. The BIOS should try each disk in turn looking for a valid boot device. The reason for installing bootcode to all the disks is so that the system can still boot if the first (or even second) disk is missing (assuming the pool still has enough devices to function). If you only put bootcode on disk1, the system will only boot if disk1 is working even though the pool may be fine without it.

Some systems allow you to choose the disk order in the BIOS config for booting, but most just give the option for HDD and are hard-coded to try port 1,2,3...

The only reason I can see to want to boot from another disk is to test that it works, in which can it's easier to just switch the machine on with disk 1 unplugged.

If you want to be able to boot into different root file systems I suggest following the advice in the 'ZFS Madness' Howto making use of sysutils/beadm

Also, a lot of ZFS fixes and improvements have been made in the last 3 years. I highly recommend using a more recent build.
 
In my case I have a HP SmartArray 6i. If the behavior @usdmatt describes (trying to boot every drive in succession) is happening, then my RAIDZ should survive a failure of any device, even in slot 0.

My intent was to upgrade to 8.3-RELEASE but I started to get drive errors in slot 0 and do not want a degraded array and try to upgrade the OS and ZFS at the same time.
 
Last edited by a moderator:
The ciss0 driver finally had enough of my disk and took it offline. zpool status reports gpt/disk5 as removed. I've pulled the old disk and ciss0 seems to be doing it's job
Code: 
ciss0: *** Hot-plug drive removed: SCSI port 2 ID 5
ciss0: *** Hot-plug drive inserted: SCSI port 2 ID 5
ciss0: *** Media exchanged detected, logical drive 5
ciss0: logical drive 5 (da5) media exchanged, ready to go online
ciss0: *** State change, logical drive 5
ciss0: logical drive (b0t5): RAID 0, 34304MB online
ciss0: logical drive 5 (da5) changed status failed->OK, spare status 0x0
(probe5:ciss0:0:5:0): Down reving Protocol Version from 2 to 0?

However, I don't see a da5 device in /dev. I'm concerned about the statement:
Code: 
ciss0: logical drive 5 (da5) changed status failed->OK, spare status 0x0

camcontrol shows it's there but gpart is dumping core on me after da4
Code: 
=>      34  71122493  da4  GPT  (34G)
        34       128    1  freebsd-boot  (64K)
       162   2097152    2  freebsd-swap  (1.0G)
   2097314  69025213    3  freebsd-zfs  (33G)

Segmentation fault (core dumped)

I also tried a camcontrol rescan all and camcontrol inquiry, both ID the new drive but no device is created in /dev.


Any suggestions?
 
Out of curiosity I thought I would put the old drive back in, but it reports:
Code: 
(pass5:ciss0:0:5:0): lost device
(pass5:ciss0:0:5:0): removing device entry
ciss0: *** Hot-plug drive inserted: SCSI port 2 ID 5
ciss0: *** Media exchanged detected, logical drive 5
ciss0: logical drive 5 (da5) media exchanged, ready to go online
ciss0: *** State change, logical drive 5
ciss0: logical drive (b0t5): RAID 0, 34304MB online
ciss0: logical drive 5 (da5) changed status failed->OK, spare status 0x0
(probe5:ciss0:0:5:0): Down reving Protocol Version from 2 to 0?
ciss0: logical drive (b0t5): RAID 0, 34304MB online
pass5 at ciss0 bus 0 scbus0 target 5 lun 0
pass5: <COMPAQ RAID 0  VOLUME OK> Fixed Direct Access SCSI-0 device 
pass5: 135.168MB/s transfers
pass5: Command Queueing enabled
cam_periph_alloc: attempt to re-allocate valid device da5 rejected
daasync: Unable to attach to new device due to status 0x6

Running out of ideas of things to try.
 
More curiosity, while gpart show coredumps, gpart status shows the device
Code: 
 Name  Status  Components
da0p1     N/A  da0
da1p1     N/A  da1
da2p1     N/A  da2
da3p1     N/A  da3
da4p1     N/A  da4
da5p2     N/A  da5

Although I don't know why it says da5p2 instead of da5p1. Also the it's listed in camcontrol:
Code: 
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 0 lun 0 (pass0,da0)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 1 lun 0 (pass1,da1)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 2 lun 0 (pass2,da2)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 3 lun 0 (pass3,da3)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 4 lun 0 (pass4,da4)
<COMPAQ RAID 0  VOLUME OK>         at scbus0 target 5 lun 0 (pass5,da5)

and in /dev/pass:
Code: 
ll /dev/pa*
crw-------  1 root  operator    0,  78 Feb 23 19:06 /dev/pass0
crw-------  1 root  operator    0,  79 Feb 23 19:06 /dev/pass1
crw-------  1 root  operator    0,  80 Feb 23 19:06 /dev/pass2
crw-------  1 root  operator    0,  81 Feb 23 19:06 /dev/pass3
crw-------  1 root  operator    0,  82 Feb 23 19:06 /dev/pass4
crw-------  1 root  operator    0, 112 Mar  7 19:51 /dev/pass5
 
See, that´s what I don´t like about the HP controllers. In one of our HP servers we took out the original and replaced it with a LSI controller instead.

/Sebulon
 
gpart(8) dumping core is quite worrying, are the world and kernel in sync? I would try to upgrade to latest 8-STABLE because there are number of ZFS related bug fixes merged during last three years.
 
Kernel and world are in sync, albeit quite dated. I was thinking about upgrading too. I had a freshly built 8.3 kernel and world ready to install when I started having issues with the original drive. I didn't want to combine a major upgrade and degraded pool into one reboot operation. I may now have no choice.
 
An installkernel and installworld went flawlessly. After reboot the new disk was recognized and a zpool replace is resilvering now. Thanks for your help!
 
You must log in or register to reply here.
Back
Top