Solved Unable to boot after fixing root-on-zfs layout

hunter0one

hunter0one

I have a server that is still on 15.0-RELEASE and I was going to update to 15.1-RELEASE following the instructions provided here, starting with creating a backup with bectl. Bectl did not work, which I came to learn had to do with the layout of my root zpool. For some reason after installation the plain "root" dataset was used for the / mountpoint rather than root/ROOT/default, which did not exist.

Anyway, I have managed to fix the layout by creating root/ROOT and root/ROOT/default filesystems and moved all data to root/ROOT/default by using the live CD but I end up with a screen like this after rebooting (the computer is a Dell Vostro 200):

Code: 
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2

The system was using GPT but with BIOS and worked so it's laid out like so (I do not have direct access to the system to easily copy and paste it so I am paraphrasing here):

freebsd-swap on ada0p1
freebsd-boot on ada0p2
freebsd-zfs on ada0p3

Also /boot/gptzfsboot, /boot/pmbr, and /boot/zfs/zpool.cache all exist.

zpool get bootfs root shows root/ROOT/default as the location used for bootfs.

Here is how I was attempting to restore boot capabilities:

Code: 
# gpart bootcode -b /boot/pmbr ada0
# gpart bootcode -p /boot/gptzfsboot -i 2 ada0

Zpool status reports the disk is healthy and I can chroot into the system fine. I would prefer not to reinstall as 1. It seems perfectly fine if boot can be restored and 2. This teaches me more about FreeBSD. Also I have some programs set up on the root/ROOT/default dataset I would prefer not to lose. Thanks.
 
Where is it stuck if you try to boot? mountroot prompt?

And what's bootfs set to on the pool? Assuming you moved everything correctly to zroot/ROOT/default, bootfs should be pointing at that. Perhaps that's still set to zroot/?
 
SirDice said:
Where is it stuck if you try to boot? mountroot prompt?

And what's bootfs set to on the pool? Assuming you moved everything correctly to zroot/ROOT/default, bootfs should be pointing at that. Perhaps that's still set to zroot/?
Click to expand...
Its showing what I believe is a BIOS prompt when I try to boot that looks like the following:

Code: 
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2

So I dont think its getting very far. The other two drives are another hard drive I have with a separate zpool and the USB with FreeBSD. Bootfs is set to root/ROOT/default. Does the pool have to be named zroot specifically?
 
hunter0one said:
I believe is a BIOS prompt
Click to expand...
No, I believe that's code from the PMBR.

hunter0one said:
freebsd-swap on ada0p1
freebsd-boot on ada0p2
Click to expand...
How big is that swap partition? It shouldn't matter if freebsd-boot is the second partition, but it might be too far out on the disk? Not sure what these limitations are. Any chance you can delete the swap and freebsd-boot partitions and swap them around. putting freebsd-boot first? Make sure it's 512K, don't make it any larger or smaller. Then write the bootcodes again, keep in mind the index will be 1 now.

hunter0one said:
Does the pool have to be named zroot specifically?
Click to expand...
No. As long as the naming is consistent. It can be named anything you want. It's common to be zroot because that's what the installer normally creates by default.
 
SirDice said:
How big is that swap partition? It shouldn't matter if freebsd-boot is the second partition, but it might be too far out on the disk? Not sure what these limitations are
Click to expand...
The swap partition is 4G, the freebsd-boot is 512K and the zfs partition is 145G.

I can try that but it worked before I attempted to fix the layout and the entire zpool is only using 23.4G of the 145G partition.

EDIT: I have moved freebsd-boot to index 1 (still 512K) and freebsd-swap to index 2 but I end up stuck at the PMBR still.
 
SirDice said:
No, I believe that's code from the PMBR.
Click to expand...
The message "BIOS disk is..." comes from gptzfsboot, more precisely from /i386/libi386/biosdisk.c.
I don't think the problem comes from pmbr side. It just looks for a freebsd-boot partition, loads it and jumps to its code. If it wasn't find that partition, the message would be explicit.
 
Emrion said:
The message "BIOS disk is..." comes from gptzfsboot
Click to expand...
Wasn't exactly sure where the divide was, but I knew for certain it was FreeBSD not the BIOS that showed these messages. At least something was booting just not sure how far up the chain.
 
Don't you have an error message after "BIOS disk is..."? If yes, we need to know it.
Also, it would be interesting to have the zfs list of your root pool.
 
SirDice said:
How big is that swap partition? It shouldn't matter if freebsd-boot is the second partition, but it might be too far out on the disk? Not sure what these limitations are.
Click to expand...
That was my thought as well. I seem to remember (perhaps wrongly), that there was a limitation on how far the "freebsd-boot" partition can be apart from the beginning of the disk.

A quick test in a bhyve(8) VM shows, the system boots just fine, with the "freebsd-boot" partiton at the end of a (virtual, dynamically allocated) 2TB disk.
Rich (BB code): 
root@test:~ # uname -a
FreeBSD test.home 15.0-RELEASE-p9 FreeBSD 15.0-RELEASE-p9 releng/15.0-n281048-6d536196f1bd GENERIC amd64

root@test:~ # gpart show -p
=>        40  4194303920    nda0  GPT  (2.0T)
          40      532480  nda0p1  efi  (260M)
      532520        2008          - free -  (1.0M)
      534528     4194304  nda0p3  freebsd-swap  (2.0G)
     4728832  4189573128  nda0p4  freebsd-zfs  (2.0T)
  4194301960        1024  nda0p5  freebsd-boot  (512K)
  4194302984         976          - free -  (488K)

root@test:~ # sysctl machdep.bootmethod
machdep.bootmethod: BIOS
 
hunter0one, please show us:
Code: 
zpool history
zpool get  bootfs
zfs list

A test setup in a VM (manually partitioned disk, manually bootcode installed), with root file system on "zroot/ROOT/default" boots just fine:
Rich (BB code): 
root@test2:~ # gpart show -p
=>       40  524287920    ada0  GPT  (250G)
         40    8388608  ada0p1  freebsd-swap  (4.0G)
    8388648       1024  ada0p2  freebsd-boot  (512K)
    8389672  515898288  ada0p3  freebsd-zfs  (246G)

root@test2:~ # zpool history
History for 'zroot':
2026-08-04.11:26:02 zpool create -o ashift=12 -o altroot=/mnt -O atime=off -m none zroot nda0p3
2026-08-04.11:26:38 zfs create -o mountpoint=none zroot/ROOT
2026-08-04.11:26:48 zfs create -o mountpoint=/ zroot/ROOT/default
2026-08-04.11:27:51 zpool set bootfs=zroot/ROOT/default zroot
2026-08-04.11:28:35 zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot


root@test2:~ # zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                397M   236G    96K  none
zroot/ROOT           396M   236G    96K  none
zroot/ROOT/default   396M   236G   396M  /
 
This may be bitten by limitaitons of legacy BIOS.

Boot codes / loader of FreeBSD uses BIOS calls to read anything from disks on legacy BIOS boots. So limitations of legacy BIOS matters, and everything boot codes / loader need to read must be within the area BIOS can access.

On the other hand, Boot codes / loader of FreeBSD uses boottime services of UEFI firmwares to read anything from disks on UEFI boots, and even recent PCs (not 100% sure about enterprize-class servers, though) wouldn't hit its limitations.

And once loader passes control to loaded kernel, FreeBSD drivers for disk controllers (basically) directly control read/writes. So even if installer image can sanely finds partitions / pools and mount/access to it, local boots from installed FreeBSD can fail if the installation hits the above-mentioned limitations.
 
T-Aoki said:
This may be bitten by limitaitons of legacy BIOS.
Click to expand...
Maybe be on very ancient machines?

pmbr uses Extended Read Drive Parameters from BIOS Int13h (function 0x48) and Extended Read Sectors From Drive (function 0x42). See here: https://en.wikipedia.org/wiki/INT_13H

It expects answers from the BIOS with a 64 bits integer for the number or the location of a sector (parameter absolute sector). If BIOS can't answer to these function calls, pmbr emits an I/O error or something like that.

Back in time, there were several errors concerning 64 bits arithmetic in pmbr, but they have been solved since.

233180 – Several errors in pmbr: 64-bits arithmetics and some others

bugs.freebsd.org bugs.freebsd.org

So, Actually, pmbr can see freebsd-boot partitions, even after 2 TiB.

But, that doesn't really help the Op.
 
The live CD you used to write the bootloaders, how old is that? Although if it was too old I would expect it to fail to load the pool entirely. But it looks like it's stuck somewhere in stage 2 and never gets to the point where it's able to read/start loader(8) (stage 3).

docs.freebsd.org

Chapter 15. The FreeBSD Booting Process

An introduction to the FreeBSD Booting Process, demonstrates how to customize the FreeBSD boot process, including everything that happens until the FreeBSD kernel has started, probed for devices, and started init
docs.freebsd.org docs.freebsd.org
 
Emrion said:
Don't you have an error message after "BIOS disk is..."? If yes, we need to know it.
Also, it would be interesting to have the zfs list of your root pool.
Click to expand...
No, I would have been sure to mention it. There is a difference if I try booting the other hard drive (the one not containing FreeBSD) though: the same code appears and underneath it (next in line) an error prints that it has no bootable partition.

The PC is indeed a Dell Vostro 200 with the only difference being it has 8 GB of RAM and the processor was upgraded to the Core 2 Duo E8600. No UEFI but I have been able to use GPT BIOS successfully for a long time now.
T-Daemon said:
hunter0one, please show us:
Code: 
zpool history
zpool get  bootfs
zfs list
Click to expand...
Here (didn't realize it would appear, but there's a lot of setting the mountpoints so I could mount it in the live environment and also exporting it when I reboot):

Code: 
# zpool history
History for 'root':
2026-05-08.14:08:26 zpool create -f -m none -o cachefile=/tmp/bsdinstall_boot/zfs/zpool.cache -O checksum=fletcher4 root /dev
/ada0p3
2026-05-09.15:01:42 zfs create -o mountpoint=/usr/local/jails root/jails
2026-05-09.15:01:57 zfs create root/jails/media
2026-05-09.15:02:01 zfs create root/jails/templates
2026-05-09.15:02:05 zfs create root/jails/containers
2026-08-03.16:57:12 zfs snapshot root@default
2026-08-03.16:57:35 zfs create root/ROOT
2026-08-03.16:59:49 zfs destroy root/ROOT
2026-08-03.17:13:35 zfs snapshot -r root@before-be-migration
2026-08-03.17:18:51 zpool import -f root
2026-08-03.17:20:27 zpool export root
2026-08-03.17:20:44 zpool import -N -f root
2026-08-03.17:26:57 zfs set mountpoint=/tmp/oldroot root
2026-08-03.17:28:19 zfs create -o mountpoint=none root/ROOT
2026-08-03.17:28:37 zfs create -o mountpoint=/tmp/newroot root/ROOT/default
2026-08-03.18:13:13 zfs set mountpoint=/ root/ROOT/default
2026-08-03.18:16:36 zpool import -N -f root
2026-08-03.18:17:08 zpool set bootfs=root/ROOT/default root
2026-08-03.18:18:32 zfs set mountpoint=/tmp/newroot root/ROOT/default
2026-08-03.18:20:05 zpool set cachefile=/tmp/newroot/boot/zfs/zpool.cache root
2026-08-03.18:22:31 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.18:23:17 zfs set mountpoint=/tmp/newroot root/ROOT/default
2026-08-03.18:24:36 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.18:26:35 zfs set -u mountpoint=none root
2026-08-03.18:27:26 zpool export root
2026-08-03.18:36:00 zpool import -N -f root
2026-08-03.18:38:36 zfs set mountpoint=/tmp/mnt root/ROOT/default
2026-08-03.18:40:55 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.18:42:59 zpool export root
2026-08-03.18:46:10 zpool import -N -f root
2026-08-03.18:48:17 zfs set mountpoint=/tmp/mnt root/ROOT/default
2026-08-03.19:12:13 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.19:12:31 zpool export root
2026-08-03.19:16:57 zpool import -N -f root
2026-08-03.19:17:05 zfs set mountpoint=/tmp/mnt root/ROOT/default
2026-08-03.19:25:03 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.19:25:27 zpool export -f root
2026-08-03.19:31:23 zpool import -N root
2026-08-03.19:31:42 zfs set mountpoint=/mnt root/ROOT/default
2026-08-03.19:33:54 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.19:39:01 zpool import -N root
2026-08-03.19:39:16 zfs set mountpoint=/mnt root/ROOT/default
2026-08-03.19:46:44 zfs set canmount=noauto root/ROOT/default
2026-08-03.19:47:09 zfs set canmount=off root/ROOT
2026-08-03.19:49:24 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.19:49:38 zpool export -f root
2026-08-03.21:56:38 zpool import -N root
2026-08-03.21:57:18 zfs set mountpoint=/mnt root
2026-08-03.21:58:33 zfs set mountpoint=none root
2026-08-03.22:14:21 zfs set mountpoint=/mnt root/ROOT/default
2026-08-03.22:19:43 zfs set -u mountpoint=/ root/ROOT/default
2026-08-03.22:26:22 zpool export root
2026-08-04.13:14:42 zpool import -N root

Code: 
# zpool get bootfs
NAME  PROPERTY  VALUE              SOURCE
root  bootfs    root/ROOT/default  local

Code: 
# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
root                   23.4G   116G  11.0G  none
root/ROOT              10.9G   116G    24K  none
root/ROOT/default      10.9G   116G  10.9G  /
root/jails             1.26G   116G    24K  /usr/local/jails
root/jails/containers  1.11G   116G  1.11G  /usr/local/jails/containers
root/jails/media        158M   116G   158M  /usr/local/jails/media
root/jails/templates     37K   116G    24K  /usr/local/jails/templates

SirDice said:
The live CD you used to write the bootloaders, how old is that? Although if it was too old I would expect it to fail to load the pool entirely. But it looks like it's stuck somewhere in stage 2 and never gets to the point where it's able to read/start loader(8) (stage 3).

docs.freebsd.org

Chapter 15. The FreeBSD Booting Process

An introduction to the FreeBSD Booting Process, demonstrates how to customize the FreeBSD boot process, including everything that happens until the FreeBSD kernel has started, probed for devices, and started init
docs.freebsd.org docs.freebsd.org
Click to expand...
It's the FreeBSD 15.0-RELEASE DVD ISO.
 
There is a problem with your first dataset root. Well, I don't see how it could be related to the boot.

Normaly, when you have zroot or whatever and its mountpoint is /zroot or /whatever.
You see the problem here? /root already exists as the home of the root user. Ok, it's funny, but nothing prohibits to choose a different mountpoint than the zpool name and, actually, root dataset doesn't have a mountpoint in your design. That said, this name is a bad idea.
 
Emrion said:
There is a problem with your first dataset root. Well, I don't see how it could be related to the boot.

Normaly, when you have zroot or whatever and its mountpoint is /zroot or /whatever.
You see the problem here? /root already exists as the home of the root user. Ok, it's funny, but nothing prohibits to choose a different mountpoint than the zpool name and, actually, root dataset doesn't have a mountpoint in your design. That said, this name is a bad idea.
Click to expand...
I agree, I would have wanted it as zroot, but I dont remember ever explicitly making it root. It was just like this after installation. The "root" dataset itself isnt set to a mount point anyway. I can change it to zroot.

EDIT: Its now zroot. As expected, it continues to get stuck before the bootloader.
 
Emrion said:
You have to modify the bootfs property in consequence.

Something tells me it's not that complex. You have several disks. Can you unplug all of them except the one where you have your zpool?
Click to expand...
Yes, that did it! Just the USB (which contained Ventoy and had the FreeBSD 15.0 live DVD) had to be unplugged. It now boots into the migrated system. At first it was going straight to mountroot but it was because I still had the mountpoint for zroot/ROOT/default set to /mnt for the live DVD and also I had to change root in /etc/fstab to zroot/ROOT/default. Now its able to boot and I can use bectl because it has the correct layout. I am now upgrading the system to 15.1-RELEASE, after making a bootable snapshot.

Thanks everyone. I will mark this as solved, though it's interesting why having an extra device such as the USB trips up gptzfsboot.
 
Cool it works now.

gptzfsboot will try to discover a zpool anywhere, in each disk and in each partition. So, even the more little bug may trigger depending on the content of a given disk. A software should never hang, so it's a bug.
 
You must log in or register to reply here.
Back
Top