ZFS UEFI loader cannot find ZFS bootable partition after mirroring and detaching

Z

zldrobit

I encountered a boot failure after switching the boot drive from a sata ada0 to a nvme drive nda0 with attaching/detaching, by executing the following commands:
zpool attach root ada0p5 nda0p4
# after resilvering done
zpool detach root ada0p5

Also, I installed a new ESP on nda0
mount_msdos /dev/nda0p1 /mnt/efi
mkdir -p /mnt/efi/efi/freebsd
cp /boot/loader.efi /mnt/efi/efi/freebsd

Finally, I invoked efibootmgr(8) to setup this loader.
The partition tables are as below (ada0p5 is the old bootable, nda0p4 is the new bootable, and ada0p4 is empty):
Code: 
=>        34  1953525101    ada0  GPT  (932G)
          34           6          - free -  (3.0K)
          40      532480  ada0p1  efi  (260M)
      532520        1024  ada0p2  freebsd-boot  (512K)
      533544         984          - free -  (492K)
      534528     4194304  ada0p3  freebsd-swap  (2.0G)
     4728832   209715200  ada0p4  freebsd-zfs  (100G)
   214444032  1738539008  ada0p5  freebsd-zfs  (829G)
  1952983040      542095          - free -  (265M)

=>        40  1953525088    nda0  GPT  (932G)
          40      532480  nda0p1  efi  (260M)
      532520        1024  nda0p2  freebsd-boot  (512K)
      533544         984          - free -  (492K)
      534528    50331648  nda0p3  freebsd-swap  (24G)
    50866176  1902657536  nda0p4  freebsd-zfs  (907G)
  1953523712        1416          - free -  (708K)
The output of zfs list is as follows:
Code: 
NAME                USED  AVAIL  REFER  MOUNTPOINT
root                164G   635G  15.9G  none
root/linuxdisk0     130G   757G  7.36G  -
root/linuxdisk1    1.02G   635G   858M  -
root/root          17.0G   635G    96K  none
root/root/DEFAULT  17.0G   635G  17.0G  /
The reboot was failed similar to ZFS: can't read MOS of pool rpool.
I notice that using a USB FreeBSD Installer stick could import the zpool from shell, and zpool status -x is normal.
Within the installer shell, rebooting without kernel reload works with zpool root correctly by using:
Code: 
kldload zfs
kenv vfs.root.mountfrom=zfs:root/root/DEFAULT
reboot -r
Other configuration I tried:
  1. SUCCESS. zfs offline ada0p5 without detaching and reboot.
  2. SUCCESS. Unplug the sata cable from ada0 hard drive and reboot.
  3. SUCCESS. gpart destroy -F ada0 to destroy the partition table of ada0 and reboot.
  4. SUCCESS. zpool labelclear ada0p5 and reboot.
  5. FAILED. Changing the partition type of ada0p1 from EFI to freebsd (let the loader ignore this ESP).
  6. FAILED. Erase all the content in the ada0p1 ESP.
I came to a conclusion that loader.efi(8) selects the bootable zpool by VDEV label no matter what TXG it has (nda0p4 has a bigger TXG than ada0p5), and one have to clear the label on the unused drive with zpool labelclear after zpool detach.

Related threads:
ZFS: can't read MOS of pool rpool
Mirrored zroot doesn't boot from second disk
How to read error messages at initial stage of the boot process
 
The link you posted about the error messages at boot refers to the use of BIOS/CMS booting.
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool rpool
gptzfsboot: failed to mount default pool rpool
Click to expand...

To clarify the point: in fact, you do not see the last line about gptzfsboot?
You said you used efibootmgr, so you must be in EFI booting. Right?
 
Emrion said:
The link you posted about the error messages at boot refers to the use of BIOS/CMS booting.


To clarify the point: in fact, you do not see the last line about gptzfsboot?
You said you used efibootmgr, so you must be in EFI booting. Right?
Click to expand...
Yes, I manually select UEFI booting method from the boot options by pressing F12.
 
Erichans said:
Is your system capable of booting from an NVMe? Check you manual and/or BIOS/Firmware settings if you don't know.
Click to expand...
After clearing the vdev label of ada0p5, I could boot from the nda0p4 NVME partition.

Erichans said:
zldrobit, if it is still easy to boot from your old sata ada0, please post the output of sysctl machdep.bootmethod
Click to expand...
Erichans I'll try resetting the boot partition in ada0p5 and reproducing this problem tomorrow. It'll take some time. I didn't backup the vdev label of ada0p5
 
I can confirm your problem to have tested it in a VM (14.1-RELEASE). For some reasons, detach a disk from a zpool mirror and leave it accessible perturbs the efi loader. It seems to be more or less the same with BIOS booting. Hard to say if it is a bug, but it's worrying.
 
zldrobit said:
I encountered a boot failure after switching the boot drive from a sata ada0 to a nvme drive nda0 with attaching/detaching, by executing the following commands:
Code: 
 zpool attach root ada0p5 nda0p4
# after resilvering done
zpool detach root ada0p5
Click to expand...
I don't know if ada0 is a sata SSD or made of spinning rust platters.

This is not the procedure I would have chosen. I would have preferred to create a ZFS pool on the NVMe seperately; followed by a zfs send-receive. This would have alllowed me to set the ashift value at creation time individually; alternatively, you could have specified an ashift value explicitly with zpool-attach(8). By this resilvering procedure, you have copied the ashift value of your sata ada0 onto your NVMe. Can you post the output of zdb -C <pool> | grep ashift of your pool on the NVMe?
 
Erichans said:
I don't know if ada0 is a sata SSD or made of spinning rust platters.

This is not the procedure I would have chosen. I would have preferred to create a ZFS pool on the NVMe seperately; followed by a zfs send-receive. This would have alllowed me to set the ashift value at creation time individually; alternatively, you could have specified an ashift value explicitly with zpool-attach(8). By this resilvering procedure, you have copied the ashift value of your sata ada0 onto your NVMe. Can you post the output of zdb -C <pool> | grep ashift of your pool on the NVMe?
Click to expand...
ada0 is a spinning rust hard drive.

You're right. I am just testing and benchmarking ZFS/FreeBSD in some extreme cases to determine whether they could be deployed into our production environment, since we have never used them before. The output of zdb -C root | grep asfhit is
Code: 
ashift: 12
ashift: 12
The first line is for the indirect vdev created automatically after I delete a vdev from the pool. The second line is for the NVMe hard drive.
 
Emrion said:
I can confirm your problem to have tested it in a VM (14.1-RELEASE). For some reason, detach a disk from a zpool mirror and leave it accessible perturbs the efi loader. It seems to be more or less the same with BIOS booting. Hard to say if it is a bug, but it's worrying.
Click to expand...
I am also worried that the vdev label in the older bootable partition is invisible from zpool status/list, zdb -C <pool>, and even zpool import [-D], while disturbing the boot process. It is only visible in the output of zdb -l ada0p5. This introduces some extent of inconsistency between zpool/zdb command and the bootloader. Maybe a deprecated tag could be added in the vdev label of the detach disk to avoid the booting disturbance.
 
This could be complicated in some edge cases.
If any of the disks are kinda upgrades from old installation, ESP on it could be too small (for exapmple, dd'ed ESP with now-nonexistent boot1.efifat), thus recent loader.efi couldn't be copied to it.
If I recall correctly, even on early UEFI days, ESP was suggested to be 200MB or larger. If all ESPs are created as such, things would have been simpler.

Unfortunately, I still see someone worries that /boot/boot1.efifat no longer exists and panics with boot failures because he/she dd'es /boot/boot1.efi to ESP, which shouldn't work.

And this thread would be related.
 
I found a thread from mail list 14 years ago discussing booting from multiple zpools (https://muc.lists.freebsd.fs.narkive.com/xVvLJ04g/multiple-zfs-pools-and-booting#post4),
There is no way to configure that at this point. The boot code just
boots off of first pool it finds - if there is no bootfs property then
it will just try to boot from the top level dataset (ie. tank).
Click to expand...
And, it seems the boot code logic hasn't changed since then. Having two zpools with bootfs set is problematic even one of them was detached!
 
zldrobit said:
And, it seems the boot code logic hasn't changed since then.
Click to expand...
It's not 100% correct.
Yes, stock loader.efi, boot1.efi nor (if I recall correctly) BIOS boot code doesn't support selecting boot pool on boot time. (BIOS boot code can choose which MBR partition to boot, though.)

On the other hand, boot1.efi has patches (currently 2 variants) to choose boot partition (including disk) and boot pool as PR 207940.
Latest 2 patches should work. The difference between those 2 variants is how candidates are shown. You can use whichever one, but cannot apply both at the same time.

Unfortunately, boot1.efi would be deleted sometime in the future. I hope the same kind of functionality is implemented into loader.efi before the time to come.
 
Without having had a look at the code, I guess that a detached disk must have something in its vdev label saying that it is detached.
I guess that the boot code (whether EFI or not) that probes / searches ZFS vdev-s should check for that property and ignore detached vdev-s.
 
Andriy said:
Without having had a look at the code, I guess that a detached disk must have something in its vdev label saying that it is detached.
I guess that the boot code (whether EFI or not) that probes / searches ZFS vdev-s should check for that property and ignore detached vdev-s.
Click to expand...
It should but it doesn't.
 
You must log in or register to reply here.
Back
Top