Problem while upgrading to 15.1-RELEASE with zfs boot

E

edxfs

Hi everyone

I have had a strange problem while upgrading to 15.1-RELEASE
The system would still stick to boot 15.0p9, even after all the upgrade procedures went fine.

# freebsd-update -r 15.1-RELEASE upgrade
# freebsd-update install
# reboot
# freebsd-update install
...

I have a zoo of freebsd machines,
It worked like a charme on the other ones (zfs/ffs/amd64/aarch64)
But on this machine, it did fail.

The upgrade process looked like it did work without flaws,
but after reboot it did again boot into 15.0p9

That machine runs on zfs root,
It was cleanly installed from 15.0/zfs just two months ago,
updated through 15.0p9 without any problems.

I have beadm installed,
Maybe I deleted the default bootfs in a cleanup script?
I will now research this.

I hope, that I am the only one to encounter this problem.
This might be specific to my setup
(while my setup is intentionally really bare bones from scratch)

I then tried to activate one of the recent boot environments (from the upgrade)
# beadm activate ...
# reboot
which put the machine in limbo

It boots with a lot of weird messages.
What the is zfs: can not read MOS?

It comes up to the login prompt,
but I can not even log in as root.

Let's select another boot environment in the boot menu.
This did not work for me.
Even the boot menu display to select another boot environment was broken.
It did only show the current boot environment,
and the display of the options was broken.

Maybe something got really broke on this machine?
Or maybe I did something completely wrong?
Or maybe it's a bug.

I will now try to get these boots to get up running.
This is why we have zfs: nothing get's lost!

I will now boot the machine to single user mode.
And then set zfs to boot from an older boot environment.

I think, this procedure should then be added to the handbook.
- How to mount zfs r/w from single user mode
- How to remount /usr/local from zfs in single user mode
- How to list and set the boot environment from single user mode

Or maybe, we need a new chapter in the handbook
how to survive ... with zfs
 
Also check if you don't have vfs.root.mountfrom="zfs:zroot/ROOT/default" specified in /boot/loader.conf, like I stupidly had.
 
It's difficult to answer as there are many emotions in the text and so few informations.
The MOS error thing makes me think about a damaged zpool, but it could be almost anything else.
 
edxfs What is your specific problem? You installed 15.1 but fall back to 15.0 p9? Then you fiddle around with your BEs without success?

Seems you miss 15.0 p10. Start your 15.0 p9 (I assume a normal boot without using any number during boot) and do:
Code: 
# freebsd-update fetch
# freebsd-update install
# shutdown -r now

After reboot you should be at 15.0 p10.
Code: 
uname -r
freebsd-version -kur

Then check your snapshots and BEs:
Code: 
zfs list -t snapshot
bectl list
I would recommend to create a new snapshot / BE to have a save point to restart. Delete the rest.

Then wait until tomorrow when 15.1 is released before you do
Code: 
# freebsd-update fetch
# freebsd-update install
# shutdown -r now

# freebsd-update upgrade -r 15.1-RELEASE
# freebsd-update install
# shutdown -r now

# freebsd-update install
# freebsd-update install
# shutdown -r now
 
Solved!

root cause:
- /boot/kernel was a symlink
- the boot menu did select another kernel by default

fix:
- remove the symlink
- mv the symlinked kernel to /boot/kernel
- reboot
- select "kernel" as the kernel in the boot menu

Description:

I try to explain, what went wrong.
But some of the following might not be fully accurate.
Please feel free to add more insights.

First of all, my previous installation was not as pure
as I had believed it to be.

I turned out, I had been experimenting,
with compiling some hand tuned kernels.

I moved the generic kernel to /boot/kernel.GENERIC
I moved the new kernel to /boot/kernel.new
and made a symlink /boot/kernel -> kernel.new
(and a handful of kernel.old[123])
but than changed that again to /boot/kernel -> kernel.GENERIC
So /boot /kernel was now a symlink (but back to the GENERIC kernel)

This all worked fine.
I upgraded to evey 15.0 patches (using freebsd-upgrade)
To my believe it did always run the newly patched GENERIC kernel after reboot.
(I did check that with uname -a)

But thinks went different when upgrading to 15.1-RELEASE
As told before, after reboot uname would still report 15.0p9
While the boot environments looked sane.

I turned out, the boot menu did select an older kernel:
/boot/kernel.old1
which for some reason actually was 15.0p9 (this needs further explanation)
but the upgrade to 15.1 did install the kernel at the expected place:
/boot/kernel -> kernel.GENERIC
(and after getting rid of the symlink, everything worked like a charme again)


Suggestions:

freebsd-update is now a thing of the past,
maybe the upgrade should warn, when /boot/kernel is a symlink?

This is the least intrusive change.

I'm not sure how the boot menu selects a kernel.
As for my experience it does not seem to resolve symlinks?
But it does default to /boot/kernel (if that is not a symlink)
But otherwise the boot menu selects a kernel by some other means
(Please note: I usually boot remote, without a console)

Is the selected kernel persisted anywhere?
Could we do that with zfs?
We already have the zpool:bootfs property,
so the boot can read zpool attributes.

How about zpool:org.freebsd.bootkernel?
(the boot stage would not need to write that zpool attribute, we can do that upon successful boot)

Booting 15.1 with a 15.0 kernel:
- network does not come up
- You can not even login
- login crashes with some weird pipefs errors
- we need to document that, as clear indication of the root cause

But in the end, zfs saved my day.
I was able to reproduce the very same error scenario again.
Selecting the boot environment from the boot menu did not work for me.
I did it manually from single user mode.

The boot menu to select a boot environment looked like broken to me,
but it is intended to look like this. This could be improved.
This needs further explanation in the Handbook.

It works different on OpenIndiana,
where you get a list of the boot environments,
but in the freebsd boot, you can switch trough the boot environments.
This was not obvious to me.
and again, I believe, it did not even work for me at all

I also want to add to the Handbook,
how to cope with zfs in single user mode.

Obviously /usr/local does not get mounted in single user mode,
therefore beadm was not available.
Maybe we should get beadm into the core system?


--
Big thanks to all of you,
It is a big pleasure to get all your help

You can tune a file system, but you cannot tune a fish.

~eike
 
You must log in or register to reply here.
Back
Top