zfs root, gptzfsboot problem

jrk · Apr 6, 2010

Hello,

following the instructions from the wiki, i have installed FreeBSD on a ZFS volume on my harddisk. This was quite straitforward and the installation on the volume is ok, at least i can access it without problems from linux with zfs-fuse.

The problem is, that i cannot boot my installation. So here is my installation in detail:
One 300gb disk, with GPT partitions as follows.
1) 128KB for embedded grub
2) ~150GB for Linux LVM
3) 64KB freebsd-boot (gptzfsboot)
4) 16GB freebsd-swap
5) ~rest freebsd-zfs

gpart -p gptzfsboot -i3 ad4 went well (ad4 is my disk as reported from the bios)
I intended to chainload pmbr with grub which in turn loads gptzfsboot from disk0p3 (== ad4p3 == hd0,3).
With gptzfsboot it should be no problem to load the loader from the zfs root volume.

So here is were the trouble begins. Pmbr seems to load fine, and i think(!) it loads gptzfsboot as well. I'm not quite sure about this but i definitely know that it's not the loader. I get a boot prompt were i can type "lsdev" for example. This will show me the partitions like above:
disk0p1 ..
disk0p2 ..
disk0p3 BOOT
disk0p4 SWAP
disk0p5 ZFS

There is also a line about zfs devices but none shows up (and i have two zfs devices in this machine, one dedicated raidz and the volumes on ad4).

I should mention that i accidentally installed pmbr to the mbr thus overwriting grub. The result was the same just like after restoring grub and chainloading pmbr.

Any help would be appreciated. Having FreeBSD on my harddisk and being able to load the kernel is frustrating. Are there any partition boundaries important to gptzfsboot (e.g. does it have to reside somewhere in first gb?).

Thanks and regards!

knarf · Apr 6, 2010

Are you using 7.3-RELEASE? Did you read ERRATA?

zfsloader is missing a little feature: ZFS.

But you "see a line about zfs devices", so your problem seems to be a different one.

jrk · Apr 6, 2010

Thanks for your answer.

knarf said:
Are you using 7.3-RELEASE?

No, 8.0-RELEASE as usb image. I guess there might be a problem with the freebsd-boot partition being somewhere in the middle of the disk (around 150gb aka a few gazillion blocks..).
But that's just a guess, maybe there is something completely different lacking.

jrk · Apr 7, 2010

Ok.. now this is interesting.

As i wrote above there is a second zfs volume configured for this computer. It's a raidz consisting of 3 physical drives. When i leave it in an imported state (i.e. do not export it) gptzfsboot or the loader tries to load the kernel from this volume.

But i can import/export the other zfs volume with my freebsd installation whatever i like, i never see it at the loader prompt.

The raidz volume (it's called data) has no bootfs option and a fixed mountpoint (/data). The system volume (sys) has the bootfs option set (bootfs=sys) and mountpoint=legacy.

Any ideas whats going here?

knarf · Apr 8, 2010

gptzfsboot or the loader tries to load the kernel from this volume.

Or the loader? The loader loads loader.conf (and loader.conf.local) and there you have vfs.root.mountfrom="zfs:sys", so the loader will try to load /boot/kernel/kernel from your zfs called sys. If sys is not there it should fail - and not try to load from zfs:data.

I have nearly the same configuration here (two zpools, booting from zroot, data is on zdata) and it's working fine. Okay, no Linux parition, but as you can run lsdev at the boot prompt, it should work for you too.

Did you rebuild the loader with LOADER_ZFS_SUPPORT=YES as written in the instructions for GTPZFSBoot? This step is needed for 8.0-RELEASE.

Knarf

jrk · Apr 8, 2010

knarf said:
Or the loader? The loader loads loader.conf (and loader.conf.local) and there you have vfs.root.mountfrom="zfs:sys", so the loader will try to load /boot/kernel/kernel from your zfs called sys. If sys is not there it should fail - and not try to load from zfs:data.

There are some new things i figured out. When i set the bootfs option (zpool set bootfs=sys sys) i get stuck at the boot prompt where i can lsdev etc. If i unset the bootfs option i'm presented with a more primitive prompt with (almost) the same problem as described here. Just that it says 'can't find root filesystem giving up on object set'. If i run status at the boot: prompt i can see my volume and the slice where it resides on. So for the moment: with bootfs set no go, without bootfs bail out on zfs trying to access the volume (and/or root).

knarf said:
I have nearly the same configuration here (two zpools, booting from zroot, data is on zdata) and it's working fine. Okay, no Linux parition, but as you can run lsdev at the boot prompt, it should work for you too.

Which version are you running currently? Which version did you use for installation?

knarf said:
Did you rebuild the loader with LOADER_ZFS_SUPPORT=YES as written in the instructions for GTPZFSBoot? This step is needed for 8.0-RELEASE.

I think i followed all the steps outlined in the wiki.. there a loader.old file in the /boot directory. Furthermore i ran strings on the new loader and found zfs in several places.

knarf · Apr 8, 2010

jrk said:
There are some new things i figured out. When i set the bootfs option (zpool set bootfs=sys sys) i get stuck at the boot prompt where i can lsdev etc. If i unset the bootfs option i'm presented with a more primitive prompt with (almost) the same problem as described here. Just that it says 'can't find root filesystem giving up on object set'. If i run status at the boot: prompt i can see my volume and the slice where it resides on. So for the moment: with bootfs set no go, without bootfs bail out on zfs trying to access the volume (and/or root).

Strange.

Code:

# zpool get bootfs zroot
NAME   PROPERTY  VALUE   SOURCE
zroot  bootfs    zroot   local
# zpool get bootfs zdata
NAME   PROPERTY  VALUE   SOURCE
zdata  bootfs    -       default

jrk said:
Which version are you running currently? Which version did you use for installation?

Currently 8.0-RELEASE-p2. On two different machines. The one is amd64, the other i386. And, on the i386 I forgot to set the bootfs option and it works fine:

Code:

bkool:/root# zpool get bootfs zroot
NAME   PROPERTY  VALUE   SOURCE
zroot  bootfs    -       default
bkool:/root# zpool get bootfs zdata
NAME   PROPERTY  VALUE   SOURCE
zdata  bootfs    -       default

But now I see a big difference. My second zpool consists of eli devices. So gptzfsboot will not be able to see the second zpool at all.

Another detail: The disks of my second zpool are on a 3ware controller with "BIOS" switched off, so they do not appear as "driveN". So even if they would't be encrypted, gptzfsboot won't see them.

So maybe you should try to get the system running without the data zpool? I still have no idea how gptzfsboot decides which zpool to use. It seems bootfs is used, have a look at /sys/boot/zfs/zfsimpl.c. But it does not seem to work for you as expected.

I have the same setup (without a second zpool) in VirtualBox for 8.0-RELEASE-p2 and 8.0-STABLE. Right now I'm installing the same scenario for 7.3-RELEASE and 7.3-STABLE. It should be easy to add a second zpool to these machines, maybe I'll try that later.

jrk · Apr 9, 2010

Hi, i have some good news!

The system booted yesterday and is now up running.. currently i'm building packages from ports though i fear to reboot. However, i think i know what the problem is/was.

Being frustrated with all that stuff i updated the sys pool from version 13 to version 16 with zfs-fuse from linux just to figure out that it was now absolutely impossible to access it from FreeBSD. Lesson learned: never do this with your data pool.

Now i was forced to setup the ZFS partition again. I used the same usb image for installation but i made some subtle changes to my previous installation. Still following the guide from the wiki i now chose a different partition layout (compare it to my first post on this topic):

1) 128KB for embedded grub
2) ~150GB for Linux LVM
3) 64KB freebsd-boot (gptzfsboot)
4) 124GB freebsd-zfs
5) 16GB freebsd-swap

Moving the swap partition to the end of the disk and having the zfs partition directly follow after the boot partition was more of a intuitive decision rather than knowledge. I guessed that either the blocks of the zfs partition were to far off at the end of the disk or that gptzfsboot just looks at the next partition (which was swap previously and thus did not contain a system..).

Furthermore after building the loader from /usr/src in the chroot i installed gptzfsboot that newly built src. I should have md5-summed that one and the one from the usb image but i forgot.

These are the two major changes i made (and that i can remember off). Personally i suspect the first one to be guilty of the non-boot misery. First there are those block error messages as seen on the mailing list and second having missing blocks due to LBA adressing problems sounds quite reasonably to me.

Knarf, thanks a lot for your help and input. I really appreciate that!

If you or someone else wants to investigate this further and needs to know more details, just ask, i'll be glad to help.

knarf said:
But now I see a big difference. My second zpool consists of eli devices. So gptzfsboot will not be able to see the second zpool at all.

Another detail: The disks of my second zpool are on a 3ware controller with "BIOS" switched off, so they do not appear as "driveN". So even if they would't be encrypted, gptzfsboot won't see them.

My raidz data pool consists of three disks connected to the onboard sata (ahci) controller. When i do not export it on shutdown/reboot it will show up, but since the sys pool is now discovered correctly the bootfs property would set in i think.

knarf said:
So maybe you should try to get the system running without the data zpool? I still have no idea how gptzfsboot decides which zpool to use. It seems bootfs is used, have a look at /sys/boot/zfs/zfsimpl.c. But it does not seem to work for you as expected.

I think i was mislead for blaming bootfs. There were some side-effects pointing at it but the real problem was something different.