Solved Can't boot after zpool upgrade

R

rihad

I did a freebsd-update upgrade from 10.3 to 11.2, everything was upgraded and ran fine up to the point that I decided to run "zpool upgrade" of the boot pool and the suggested gpart bootcode update. Now the machine doesn't boot.

6487


It's an OVH machine with their usual root-on-zfs setup. It has a nvme disk (/dev/nvd0).

The command that was suggested by zpool upgrade and that I ran was this:
sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0

by simply replacing the boilerplate ada0 with nvd0, but ignoring the other parts.

Booting into rescue mode I can see that the zpool imports fine, but file -s /dev/nvd0p1 returns simply "data". Now I ran gpart bootcode -b /boot/pmbr -p /boot/zfsboot -i 1 nvd0 hoping for the better. Now I'm getting something more meaningful than data:
# file -s /dev/nvd0p1
/dev/nvd0p1: DOS/MBR boot sector; partition 4 : ID=0xa5, active, start-CHS (0x0,0,1), end-CHS (0x3ff,254,63), startsector 0, 50000 sectors
Click to expand...

I will try rebooting shortly. From our other boxes on OVH I can see that p1 should look something like this:

$ sudo file -s /dev/ada0p1
/dev/ada0p1: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "BSD4.4 ", root entries 512, sectors 1575 (volumes <=32 MB), sectors/FAT 5, sectors/track 63, heads 1, serial number 0x2cd61511, label: "EFI ", FAT (12 bit), followed by FAT
Click to expand...

(although there's no FAT anywhere, it's a normal FreeBSD box zfs on root):

zroot 122G 304G 96K none
zroot/ROOT 5.74G 304G 96K none
zroot/ROOT/default 5.74G 304G 5.74G /
Click to expand...

How can I get it to boot?
 
zfs upgrade and the bootcode update are evil for the general population. Only use them if you're a zfs/bootcode hacker.
 
The output from gpart show would be helpful.
Still, I just guess into the wild that you are using a GPT partitioning scheme and thus should have used -p/boot/gptzfsboot

Edit:
just for your info. What you posted from your other machines boot partition seems to be EFI bootcode, but what you have on the machine that refuses to boot looks like MBR.
To be sure, post output from gpart show.
 
As I wrote above gptzfsboot was the boot code suggested by zpool upgrade and that I installed. I just did it too and it kind of erased the boot sector of nvd0p1 with garbage:

root@rescue-bsd:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0
partcode written to nvd0p1
bootcode written to nvd0
root@rescue-bsd:~ # file -s /dev/nvd0
/dev/nvd0: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 879097967 sectors
root@rescue-bsd:~ # file -s /dev/nvd0p1
/dev/nvd0p1: data
root@rescue-bsd:~ #
Click to expand...
 
root@rescue-bsd:~ # gpart show
=> 34 879097901 nvd0 GPT (419G)
34 6 - free - (3.0K)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 8388608 3 freebsd-swap (4.0G)
8392704 870705152 4 freebsd-zfs (415G)
879097856 79 - free - (40K)

=> 34 879097901 nvd1 GPT (419G)
34 6 - free - (3.0K)
40 879097888 1 freebsd-zfs (419G)
879097928 7 - free - (3.5K)

=> 34 879097901 diskid/DISK-CVPF71620087450RGN GPT (419G)
34 6 - free - (3.0K)
40 879097888 1 freebsd-zfs (419G)
879097928 7 - free - (3.5K)

root@rescue-bsd:~ #
Click to expand...

It does look like efi. But how do I install it? boot.efifat?
root@rescue-bsd:~ # ll /boot/*efi*
-r-xr-xr-x 1 root wheel 96768 May 10 12:18 /boot/boot1.efi*
-r--r--r-- 1 root wheel 819200 May 10 12:18 /boot/boot1.efifat
-r-xr-xr-x 1 root wheel 409600 May 10 12:18 /boot/loader.efi*
Click to expand...

So the complete command will be:
gpart bootcode -b /boot/pmbr -p /boot/boot1.efifat -i 1 nvd0
?
 
You have both, EFI and legacy boot partitions and you wrote legacy bootcode to your EFI boot partition...
I don't know if you where booting in EFI or legacy mode, but I'd install updated bootcode to both boot partiitons.
gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0
 
Yeah you are right, couldn't make much of the picture, but looks like you where booting in EFI mode.
That explains, why you couldn't boot anymore, since you wrote the wrong bootcode to the EFI partition.

To be clear, the first command I posted fixes your broken EFI bootcode.
The second command updates the legacy boot partition. This is only for completeness, it's not required to boot in EFI mode.
 
Thanks a lot, the machine booted perfectly after those two steps! The command suggested by zpool upgrade is misleading to say the least...
 
Good to hear.
If you find it hard to remember, which bootcode to use, you could label the partitions with the filename of the bootcode, like
gpart modify -l boot1.efifat -i 1 nvd0
gpart modify -l gptzfsboot -i 2 nvd0
Next time you are unsure which bootcode to install run gpart show -l :)

That doesn't do any harm, it's GPT labels, you could give any partition any name you want.
 
k.jacker said:
You have both, EFI and legacy boot partitions and you wrote legacy bootcode to your EFI boot partition...
I don't know if you where booting in EFI or legacy mode, but I'd install updated bootcode to both boot partiitons.
gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0
Click to expand...

On FreeBSD 13.0 there's no /boot/boot1.efifat.
Would executing the second line only be enough?

Code: 
$ sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0
partcode written to nvd0p2
bootcode written to nvd0
$
 
Uhm, where's the EFI directory? It isn't under /boot

Here's the current setup:

Code: 
$ gpart show
=>        40  1875384928  nvd0  GPT  (894G)
          40        1600     1  efi  (800K)
        1640        1024     2  freebsd-boot  (512K)
        2664        1432        - free -  (716K)
        4096    16785408     3  freebsd-swap  (8.0G)
    16789504  1858594816     4  freebsd-zfs  (886G)
  1875384320         648        - free -  (324K)

=>        40  1875384928  nvd1  GPT  (894G)
          40        1600     1  efi  (800K)
        1640        1024     2  freebsd-boot  (512K)
        2664        1432        - free -  (716K)
        4096    16785408     3  freebsd-swap  (8.0G)
    16789504  1858594816     4  freebsd-zfs  (886G)
  1875384320         648        - free -  (324K)

There's no need to mount an EFI partition, but there IS a need for the machine to come up after I reboot it :)
 
rihad said:
Uhm, where's the EFI directory?
Click to expand...
It's on the efi filesystem. Which you need to mount with msdosfs(5).

Code: 
root@molly:~ # mount | grep efi
/dev/gpt/efiboot0 on /boot/efi (msdosfs, local)
root@molly:~ # ls -al /boot/efi/
total 19
drwxr-xr-x   1 root  wheel  512 Jan  1  1980 .
drwxr-xr-x  15 root  wheel   71 Apr 11 16:47 ..
drwxr-xr-x   1 root  wheel  512 Jan 30 16:37 EFI
root@molly:~ # ls -al /boot/efi/EFI/
total 3
drwxr-xr-x  1 root  wheel  512 Jan 30 16:37 .
drwxr-xr-x  1 root  wheel  512 Jan  1  1980 ..
drwxr-xr-x  1 root  wheel  512 Jan 30 16:37 BOOT
root@molly:~ # ls -al /boot/efi/EFI/BOOT/
total 1752
drwxr-xr-x  1 root  wheel     512 Jan 30 16:37 .
drwxr-xr-x  1 root  wheel     512 Jan 30 16:37 ..
-r-xr-xr-x  1 root  wheel  896000 Mar 20 21:28 bootx64.efi

rihad said:
There's no need to mount an EFI partition
Click to expand...
You need to mount it, at least temporarily, or else you can't write loader.efi to it. From 13.0 onward there is no *.efifat file any more. It was rather problematic and assumed it was a fairly small partition. It also didn't account for other loaders that might be on it.
 
This is a 11.4 FreeBSD freebsd-update'd to 13.0. The first kernel reboot went fine. Which means there's no need for this image to exist. It isn't anywhere on disk. The /boot/efi directory does exist, but it is completely empty.

Code: 
[rihad@titan ~]$ mount|fgrep efi
[rihad@titan ~]$

Does all this mean that 13.0 didn't need the EFI stuff to boot in our case?
Otherwise, if zpool upgrade messed everything up and re-initializing EFI is required, what would this command have to be turned into and run?

gpart bootcode -p /boot/boot1.efifat -i 1 nvd0

Is there any hope that this was enough?
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0
 
rihad said:
Does all this mean that 13.0 didn't need the EFI stuff to boot in our case?
Click to expand...
It's not FreeBSD that needs it, it's your machine. If you UEFI boot the machine then you need the efi partition and the correct bootx64.efi file. Apparently your machine uses CSM boot (aka old school BIOS boot). Still a good idea to fix the UEFI loader in case you decide to switch from CSM to UEFI boot.
 
Uhm, since originally the machine (running 11.4) had both efi & freebsd-boot partitions, is there any chance that 13.0 booted from the efi partition previously initialized by 11.x?
Code: 
=>        40  1875384928  nvd0  GPT  (894G)
          40        1600     1  efi  (800K)
        1640        1024     2  freebsd-boot  (512K)
        2664        1432        - free -  (716K)
        4096    16785408     3  freebsd-swap  (8.0G)
    16789504  1858594816     4  freebsd-zfs  (886G)
  1875384320         648        - free -  (324K)

Please see the screenshot in this thread's first post - there's BSOD saying rEFInd, which kind of says it attempted to boot from EFI (?)
 
rihad said:
Uhm, since originally the machine (running 11.4) had both efi & freebsd-boot partitions, is there any chance that 13.0 booted from the efi partition previously initialized by 11.x?
Click to expand...
No, your machine doesn't suddenly change the way it boots (this is determined in the BIOS/UEFI settings, not the OS). You can use the machdep.bootmethod to see how your system actually booted:
Code: 
root@molly:~ # sysctl machdep.bootmethod
machdep.bootmethod: UEFI
Code: 
root@hosaka:~ # sysctl machdep.bootmethod
machdep.bootmethod: BIOS
 
You must log in or register to reply here.
Back
Top