Is the zpool’s bootfs property set? What is the exact message when the system doesn’t boot?
After importing the pool, what do you get by runningI just tried the FreeBSD 13.0 RC3 loader and the issue remains - still unable to boot from the pool.
zpool get bootfs
?The message reads:What is the exact message when the system doesn’t boot?
Reboot and Select proper Boot device
Please press the <DEL> key into BIOS setup menu.
or Press the <CTRL-ALT-DEL> key to reboot the system.
I booted into FreeBSD 13.0 RC3 on the affected host, imported theIs the zpool’s bootfs property set?
zroot
pool and inspected the properties using zpool get
. The bootfs
is set to YES
. $ zpool get bootfs newsys
NAME PROPERTY VALUE SOURCE
newsys bootfs newsys/ROOT/13.0-RC2.2 local
bootfs=(unset)|pool/dataset
Identifies the default bootable dataset for the root pool. This
property is expected to be set mainly by the installation and
upgrade programs. Not all Linux distribution boot processes use
the bootfs property.
After importing the pool, what do you get by runningzpool get bootfs
?
root@:~ # zpool get bootfs
NAME PROPERTY VALUE SOURCE
zroot bootfs zroot/ROOT/default local
Looking at the output provided above - I take it that no further action is needed here?It needs to be set to the actual boot (root) filesystem:
Not for us right now, but for yourself in case you accidentally destroy your partition table.If it's of any help, here is the output ofgpart backup
prior to overwriting the MBR with zero:
View attachment 9419
This does not look like loader message. This is more like BIOS message. That may mean that the loader has not been started.The message reads:
Code:Reboot and Select proper Boot device Please press the <DEL> key into BIOS setup menu. or Press the <CTRL-ALT-DEL> key to reboot the system.
efibootmgr -v
. See efibootmgr(8)Yes, that is correct. Hence the topic titleThis does not look like loader message. This is more like BIOS message. That may mean that the loader has not been started.
Technically I can - just gonna take some time for me to get there.Can you try booting on another motherboard?
As far as I can tell the boot drives (the two NVMe drives that make up the mirroredAlso, when you have booted from another drive, you can tryefibootmgr -v
.
zroot
zpool) do not show up there. zfs upgrade
.Those are two m.2 form-factor NVMe drives attached to PCIe adapters. These are pure physical adapters and don't contain any sort of controller chips. Each of the two NVMe drives has its own PCIe adapter.How are the nvme devices attached? On the motherboard? PCIe card?
Yes there is. In this case it's F11. The boot list is empty unless the USB stick with the FreeBSD system is plugged in.Is there a “Select Boot Device” or similar boot option you can activate (typically a function key) during the boot process? This will usually still list other devices even if they aren’t in the “active” boot priority list.
This system has been running for more than two years without any issues. It is perfectly capable of booting from NVMe drives. At least it was until I ran theCheck your UEFI/BIOS and the docs of your MB. Some systems can not boot off a NVMe device. Then you can perform any vodoo, it just won't boot.
zfs upgrade
.I think that this is gonna be my next step.As a last resort, you could make a USB boot stick that has the EFI loader (in an efi boot partition, similar to what you have on your nvd devices) and nothing else; it (the loader) will scan other devices after it doesn’t find a bootable filesystem on the device.
I've just done this. I am able to boot from the created USB drive. From there the system complains that it cannot find a bootable partition with:As a last resort, you could make a USB boot stick that has the EFI loader (in an efi boot partition, similar to what you have on your nvd devices) and nothing else; it (the loader) will scan other devices after it doesn’t find a bootable filesystem on the device.
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
:Those are two m.2 form-factor NVMe drives attached to PCIe adapters. These are pure physical adapters and don't contain any sort of controller chips. Each of the two NVMe drives has its own PCIe adapter.
This has already been set to EFI.Hrmm. Perhaps the PCI devices need option rom scan turned on?
Perhaps this: https://www.supermicro.com/support/faqs/faq.cfm?faq=25543 ?
zfs upgrade
.If this is too risky, or too much trouble, you can clone the system using the method I have practiced many times.My next step will be moving the two NVMe drives to a different host and trying booting from there.
I am open for more suggestions.
zpool attach
connect this new partition to the existing pool. Now you have a 3 way mirror. Let it resilver and and shut down after that. Then just physically remove the new drive and bring it to another computer. It should boot right away. This is how I have cloned my FreeBSD laptop. zpool detach
the now non existing drives from mirror. If the new partition is bigger, you can automatically have extra space in pool. This is how I changed the drive in may laptop and also moved the clone of my laptop system to test system. zpool add
instead of zpool attach
or you are in trouble! Read the manual zpool(8).blk1
and blk2
. I have selected (?) one by typing blk1:
in the shell. Then I tried to ls
but the EFI shell reported: ls/dir: Cannot open current directory - Not found
. zfs upgrade
zroot
pool, mounting the filesystem(s) and browsing them.So, just to recap;In the meantime I removed the two NVMe drives and added them to a different host: Same symptoms: They show up as devices, I can boot a live OS and import the pool and browse files but they don't show up as boot devices. I have done the same on yet another system ending up with the same symptoms.
Both other systems I tried to boot from those two NVMe drives are similar in design: Supermicro mainboards, Intel CPU and both of them boot from NVMe drives themselves in regular operation (I just took them down for this test). The mainboards and CPU are slightly different between the systems but the overall architecture is the same and everything used to be able to boot from NVMe drives.
I am really not sure what's going on here. I have more than one hosts which uses this exact configuration: Two separate NVMe drives (Samsung 970 Pro) via a m.2 PCIe adapter in a ZFS mirror pool. And again: The system in question used to work flawlessly for over two years so... wtf?
@Argentum: Thank yo ufor outlining that procedure. I have plenty of spare drives of all varieties to give this a go. I'll just take a regular old 512GB SATA SSD (my NVMe drives are both 256GB devices).
Eric A. Borisch: I am currently trying to figure out how I can select a drive and boot from it in the EFI shell. The drives show up asblk1
andblk2
. I have selected (?) one by typingblk1:
in the shell. Then I tried tols
but the EFI shell reported:ls/dir: Cannot open current directory - Not found
.
At the moment I am unsure whether this is simply because it's a device from a ZFS pool and the EFI shell doesn't know how to handle that.
<fsroot>/EFI/BOOT/bootx64.efi
, such that if nvmep1 is mounted at /boot/efi (not a necessity, but this is what the installer is going to start doing by default), you can see this (excepting ada vs. nvme), and your sizes are likely different from what I have (I'm running 13-RC3 built from source...) $ mount | grep efi
/dev/ada0p1 on /boot/efi (msdosfs, local)
$ ls -l /boot/efi/EFI/BOOT/bootx64.efi /boot/loader.efi
-rwxr-xr-x 1 root wheel 896512 Mar 11 22:37 /boot/efi/EFI/BOOT/bootx64.efi
-r-xr-xr-x 2 root wheel 896512 Mar 10 08:53 /boot/loader.efi
$ gpart show -rp | grep p1
40 4096 ada0p1 c12a7328-f81f-11d2-ba4b-00a0c93ec93b (2.0M)
Yeah - I believe so, but can't promise for 100%.Is the first picture you shared really the state just after you did the zfs upgrades and nothing else ? Didn't you try to do something else before posting the image ?
Because in the layout you shared you're missing the freebsd-boot partition. EFI is just a fancy partition you can jump from to OS bootloader.
gpart backup nvd0 | gpart restore -F ada0
/boot/loader.efi
from a FreeBSD 13.0 RC3 live system to the EFI partitionzroot
and resilvering it dd
clone of the SATA SSD so I have a spare when things go wrong (I do have an off-site backup but re-installing and pulling the backups was the idea to avoid). After that I will further investigate as per Eric A. Borisch 's last post.This; but it's still unclear why it would have been working and then stop working after aEnter in your bios, navigate to PCIe/PCI/PnP Configuration and enable Option ROM support and set it to EFI. Save and exit. Then enter again in the bios.
In your bios setup under boot verify if "boot mode select" is set to UEFI if it's not set it then save and exit, then enter again in the bios.
Under boot menu verify if you see the names of your nvme disks. If you don't see them as boot device you can try manual to create a new boot record or use efibootmgr. Also check the content of your startup.nsh in ESP it must point to bootx64.efi.
zpool upgrade
. If the drives don't show up in the bios boot list, I'm not sure how anything else on the disk matters.