ZFS ZFS boot problem after upgrade to 13.0

I have a Gen8 HP Microserver with 4x 3TB SATA disks attached via AHCI, in a 4 way mirror. I *think* since I upgraded to 13.0 from 12.2 I've been unable to boot from them. I am able to boot successfully from a USB stick with the live image installed. I've reinstalled the 13.0 bootcode on every disk, and I've scrubbed and even resilvered every disk as well (switching between devices and gpt partition labels assembling the pool) . None of them are showing SMART failures.

My gut is telling me there's a bug in gptzfsboot but I'm wondering if there's anything else I can try? I can of course customise a USB sticks loader.conf to automatically boot the system, but that feels like a cop out.

Any help much appreciated.
 

Attachments

  • boot_crash.png
    boot_crash.png
    81.6 KB · Views: 181
  • lsdev.png
    lsdev.png
    56.3 KB · Views: 205
  • preboot.png
    preboot.png
    63.6 KB · Views: 187
  • zfs_get_canmount_zpool_get_bootfs.png
    zfs_get_canmount_zpool_get_bootfs.png
    73.4 KB · Views: 202
Your disk0 (second picture) seems to be the one you're booting from? That's UFS, not ZFS. To boot UFS you need gptboot(8). Or is that a memory stick you booted from?
 
It's the memory stick.

I should add I've also shuffled the disks around in the bays to see whether any of them are a root cause, as well as wiping one and installing a base system on to no avail.

I should probably try reinstalling 12.2 again...
 
You never did a zpool upgrade on those pools when you upgraded to 13.0? Then you should still be able to access it with 12.2.
 
It's fully upgraded, first thing I tried.
That was the wrong thing to do though. Now you won't be able to access it from 12.2 any more. Never upgrade your pools until you're dead certain you don't need to access it from an older FreeBSD version any more.
 
you can either boot from the usb stick with a custom loader.conf so you don't have to type anything
hack one or more swap partitions into freebsd ufs eventualy+swap, install ufs bootcode in the freebsd-boot partitions
copy /boot/loader* and all its deps in the new ufs partitions and use a modified loader.conf to boot with zfs (as you typed above)
 
That was the wrong thing to do though. Now you won't be able to access it from 12.2 any more. Never upgrade your pools until you're dead certain you don't need to access it from an older FreeBSD version any more.
Good to know for the future.


you can either boot from the usb stick with a custom loader.conf so you don't have to type anything
hack one or more swap partitions into freebsd ufs eventualy+swap, install ufs bootcode in the freebsd-boot partitions
copy /boot/loader* and all its deps in the new ufs partitions and use a modified loader.conf to boot with zfs (as you typed above)

Ooh, I like your idea.

I think I'll pull the drives and do a trial install of 13.0 on a fresh pair to see if there's a genuine problem that needs reporting first - I've got identical spares.

If so, test the UFS conversion on those before breaking the real mirror, converting it and resilvering. I've got backups of all the important stuff, just prefer to preserve the non-important stuff if possible.

Thank you everyone, I'll report back when I've given it some work.
 
Haven't had time to pull drives etc, but I've been able to work out how to convert the system to UFS bootcode after it's been booted from the USB stick. It's now booting perfectly from the disks again. Interestingly I still get zio_read_error: 5, and gptboot is reporting an invalid backup header (but gpart list shows no issues whatsoever with corrupt tables).
you can either boot from the usb stick with a custom loader.conf so you don't have to type anything
hack one or more swap partitions into freebsd ufs eventualy+swap, install ufs bootcode in the freebsd-boot partitions
copy /boot/loader* and all its deps in the new ufs partitions and use a modified loader.conf to boot with zfs (as you typed above)
Commands below.
Code:
# Update loader.conf to point to zfs root
echo vfs.root.mountfrom=\"zfs:zroot/ROOT/default\" >> /boot/loader.conf
# Repeat below for each disk
# Delete swap partition
swapoff /dev/ada0p2
gpart delete -i 2 ada0
# Create UFS partition
gpart add -t freebsd-ufs -s 512M ada0
# Create (smaller) swap partition
gpart add -t freebsd-swap ada0
swapon /dev/ada0p4
# Install gptboot
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
# Create UFS filesystem
newfs /dev/ada0p2
# Mount new filesystem
mount /dev/ada0p2 /mnt
# Copy boot files
cp -R /boot /mnt
# Unmount new filesystem
umount /mnt

Can you show the output of “gpart list da1” - basically any one of the disks that contain zfs.
May not be useful now I've converted the system, but here it is.
Code:
$ gpart list ada0
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,95c41bcd-5778-11eb-9629-d0bf9c458734,0x28,0x400)
   rawuuid: 95c41bcd-5778-11eb-9629-d0bf9c458734
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 1063
   start: 40
2. Name: ada0p2
   Mediasize: 536870912 (512M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r0w0e0
   efimedia: HD(2,GPT,a4e60e7b-ef1a-11eb-acad-d0bf9c458734,0x428,0x100000)
   rawuuid: a4e60e7b-ef1a-11eb-acad-d0bf9c458734
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 536870912
   offset: 544768
   type: freebsd-ufs
   index: 2
   end: 1049639
   start: 1064
3. Name: ada0p3
   Mediasize: 2998443966464 (2.7T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2148532224
   Mode: r1w1e1
   efimedia: HD(3,GPT,95db10a4-5778-11eb-9629-d0bf9c458734,0x400800,0x15d109800)
   rawuuid: 95db10a4-5778-11eb-9629-d0bf9c458734
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998443966464
   offset: 2148532224
   type: freebsd-zfs
   index: 3
   end: 5860532223
   start: 4196352
4. Name: ada0p4
   Mediasize: 1611116544 (1.5G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 537415680
   Mode: r1w1e0
   efimedia: HD(4,GPT,af358d7e-ef1a-11eb-acad-d0bf9c458734,0x100428,0x3003d8)
   rawuuid: af358d7e-ef1a-11eb-acad-d0bf9c458734
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1611116544
   offset: 537415680
   type: freebsd-swap
   index: 4
   end: 4196351
   start: 1049640
Consumers:
1. Name: ada0
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r2w2e3
 
Back
Top