Solved ZFS Boot with very "special" special device

Hi. It is very unfortunate to return to the forum only with problems, but I have another one.
Before my recent experiments I have stable 2-way mirror root-on-zfs setup with following characteristics:

Pretty ordinary setup indeed.
Code:
=>        40  5860533088  da6  GPT  (2.7T)
          40        1024    1  freebsd-boot  (512K)
        1064      408536       - free -  (199M)
      409600    16384000    2  freebsd-swap  (7.8G)
    16793600  5843738624    3  freebsd-zfs  (2.7T)
  5860532224         904       - free -  (452K)

=>        40  5860533088  da7  GPT  (2.7T)
          40        1024    1  freebsd-boot  (512K)
        1064      408536       - free -  (199M)
      409600    16384000    2  freebsd-swap  (7.8G)
    16793600  5843738624    3  freebsd-zfs  (2.7T)
  5860532224         904       - free -  (452K)

But as preparing for 13.0-RELEASE with fresh new zfs (special allocation class vdevs) features I bought Sun Oracle F80 Flash Accelerator. This piece of hardware is simply four 200GB ssd's and LSI controller in IT mode (JBOD) residing on single board. It visible from system as:
Code:
kernel: mps0: <Avago Technologies (LSI) SSS6200> port 0xe000-0xe0ff mem 0xdebc0000-0xdebcffff,0xdea80000-0xdeabffff irq 16 at device 0.0 on pci1
kernel: mps0: Firmware: 113.05.03.01, Driver: 21.02.00.00-fbsd
kernel: mps0: IOCCapabilities: 984c<ScsiTaskFull,DiagTrace,EEDP,TransRetry,IR,MSIXIndex>

Disks are:
Code:
kernel: da2 at mps0 bus 0 scbus0 target 7 lun 0
kernel: da2: <ATA 2E256-TU2-510B00 UIO6> Fixed Direct Access SPC-4 SCSI device
kernel: da2: Serial Number 11000398537
kernel: da2: 600.000MB/s transfers
kernel: da2: Command Queueing enabled
kernel: da2: 190734MB (390625001 512 byte sectors)
kernel: da0: <ATA 2E256-TU2-510B00 UIO6> Fixed Direct Access SPC-4 SCSI device
kernel: da0: Serial Number 11000398559
kernel: da0: 600.000MB/s transfers
kernel: da0: Command Queueing enabled
kernel: da0: 190734MB (390625001 512 byte sectors)
kernel: da3 at mps0 bus 0 scbus0 target 8 lun 0
kernel: da3: <ATA 2E256-TU2-510B00 UIO6> Fixed Direct Access SPC-4 SCSI device
kernel: da3: Serial Number 11000398489
kernel: da3: 600.000MB/s transfers
kernel: da3: Command Queueing enabled
kernel: da3: 190734MB (390625001 512 byte sectors)
kernel: da1 at mps0 bus 0 scbus0 target 6 lun 0
kernel: da1: <ATA 2E256-TU2-510B00 UIO6> Fixed Direct Access SPC-4 SCSI device
kernel: da1: Serial Number 11000398371
kernel: da1: 600.000MB/s transfers
kernel: da1: Command Queueing enabled
kernel: da1: 190734MB (390625001 512 byte sectors)
So far so good.
I have added two of them as special device to my zroot pool:
Code:
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 03:50:05 with 0 errors on Thu Apr 15 08:20:05 2021
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da6p3   ONLINE       0     0     0
            da7p3   ONLINE       0     0     0
        special
          mirror-1  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0

And after reboot I have:
Code:
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
gptzfsboot: failed to mount default pool zroot

Special devices cannot be detached from pool. Everything is readble and in good health when booting from live CD. Can be this solved by some manipulations, or I need recreate my pool? Can this be result of invisibility this SSDs from BIOS (as stated from Sun documentation: "Note - The Sun Flash Accelerator F80 PCIe Card cannot be used as a boot device")? Thank you in advance.
 
I think the reason isn't ZFS so much as the GPT ZFS boot mechanism. It was probably configured when the two disks that you really boot from were da0 and da1, and it remembers that. Now you have added extra disks, and your old disks (that you likely booted from) have become da6 and da7.

I can't give specific advice, since I don't boot from ZFS disks, but look at the forum for how to set up gptzfsboot, and read the documentation. I suspect this can be easily fixed and prevented. There is probably no need to re-create your zroot pool.
 
Is UEFI boot enabled?
Sun Flash Accelerator F80 PCIe Card support in Sun Server X4-2L is restricted to Legacy BIOS boot mode. UEFI BIOS boot mode is not supported for servers with installed Sun Flash Accelerator F80 PCIe Cards.

Edit: some benchmarks, please?
 
I think the reason isn't ZFS so much as the GPT ZFS boot mechanism. It was probably configured when the two disks that you really boot from were da0 and da1, and it remembers that. Now you have added extra disks, and your old disks (that you likely booted from) have become da6 and da7.
ZFS is pretty smart related to what disks it using. I many times manipulate disk count on my server. Adding/removing them. Disk names often change, but always I have correct pools and boot without problems. This case is no exception. I have added Sun F80, disks renumbered again, and reboots were without problems. Only after special mirror device was created and attached to the root problem appears.
I can't give specific advice, since I don't boot from ZFS disks, but look at the forum for how to set up gptzfsboot, and read the documentation. I suspect this can be easily fixed and prevented. There is probably no need to re-create your zroot pool.
I have tried rewrite gptzfsboot using suggested methods (gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da[6,7] ), but this gives nothing.
Is UEFI boot enabled?


Edit: some benchmarks, please?
I it is GPT. As for benchmarking, nothing is wotking now :)
 
What if you (physically) disconnect off da6 and da7 ?
 
What if you (physically) disconnect off da6 and da7 ?
If I disconnect da7 AND da6 there are no boot partitions anymore in the system. da0 + da1 is whole disk mirror (no boot/any other partitions on this disks).

I think problem with metadata, which transferred to da0+da1. This disks is not visible from bios, so, I assume, is not visible from boot loader. And during boot process/zfs inialization zroot want to have ordinary mirror (partitions from da6 and da7) and special device mirror, which is not visible yet. So we have corrupted pool situation? Is this make sense?
 
If anyone interested, I've found the source of this issue. And I was right, this was because of invisible Sun F80 SSD's during boot process. By using LiveCD I have manage to import zroot pool without any problems. Then, using spare HDD (it is good to have many available HDD slots in JBOD enclosure), I run:
Code:
zpool replace zroot /dev/da1 /dev/da15
effectively substituting one of F80 SSDs with ordinary HDD in special device mirror. After successful replacement I have succesfull boot process. To make absoultely sure about my "fix", I take da15 off again by making it offline leaving special device in degrated state. After reboot I got same zpool errors as before.

So morale of this story is simple and obvious: zfs pools cannot work in faulted state, even if this state is only during boot process. Now I will replace Sun F80 SSD's with ordinary SATA SSD's, which will be visible from BIOS.

TLDR: Don't use Sun flash accelerators in zfs boot pools, because this can make your pools faulted during boot process.
 
Last edited:
Back
Top