Upgrade messed up. Can I overwrite with new install, but preserve data?

[rowan194]

Does that rescue media have a kernel.txz? Just extract that to overwrite the botched kernel. It's probably going to be a 11.2 kernel but that's less likely to cause problems on a 11.1 userland than a 12.2 kernel might.

Doesn't have that specific file, although I did previously extract the 11.1 (or perhaps 11.2) kernel.txz from ftp.freebsd.org. No change.

Instead, I copied the contents of /boot from the rescue image (including the empty loader.conf) to the zroot file system, then wrote out the boot code from the rescue image.

To mount pool and change to its / as cwd
zpool import -R /tmp/mnt 13406323999712830393
umount /tmp/mnt
mount -t zfs zroot/ROOT/default /tmp/mnt
cd /tmp/mnt

Then:
mv ./boot ./oldboot
mkdir boot
rsync -av /boot/ ./boot/ # copies from /boot/ to /tmp/mnt/boot/
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
shutdown -r now

Now, unless something is seriously wrong, surely copying the kernel and bootcode of a known working setup would be sufficient to boot far enough that it logs something, but for some reason, it does not seem to get to the point of mounting zroot. I have checked /var/log/messages (when booted under rescue) to see if there are any warnings about hardware, corrupt partitions, and so on - nothing.

Am I doing something wrong with the way I mount it under rescue? Here's zfs mount

NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot               1.24T   528G    96K  /tmp/mnt/zroot
zroot/ROOT          1.24T   528G    96K  none
zroot/ROOT/default  1.24T   528G  1.24T  /tmp/mnt

I do not export the pool before reboot. Should I be doing this? Could it be that it's refusing to auto-mount because it was last accessed by a 'different' system (rescue)?

I've just ordered a new server, so I will try exporting before reboot purely out of curiosity. [edit: Behaviour did not change.]

 

[T-Daemon]

..., but for some reason, it does not seem to get to the point of mounting zroot.

Do you mean it stops at "mountroot>" ?

 

[rowan194]

Do you mean it stops at "mountroot>" ?

I don't have console access, so all I can tell is that it does not seem to be getting to the point of mounting, or if it does mount, doing enough past that to be able to write to /var/log/messages.

 

[T-Daemon]

I don't have console access,

I see.
Why I asked:

I copied the contents of /boot from the rescue image (including the empty loader.conf) to the zroot file system,

If there is a empty /boot/loader.conf ZFS datasets can't be mounted. There has to be a zfs_load="YES" entry to load the zfs.ko kernel module.

 

[rowan194]

If there is a empty /boot/loader.conf ZFS datasets can't be mounted. There has to be a zfs_load="YES" entry to load the zfs.ko kernel module.

Facepalm. Of course, the rescue system is UFS based. I'm making so many silly mistakes or incorrect assumptions with this issue. I'm glad I mentioned /boot/loader.conf was empty, and thank you for picking up that problem. I have added zfs_load="YES" and restarted, but it hasn't come up.

The new server I've just ordered has IPMI, so I can access the console just by loading a link in my browser. If the current server had that, this thread would likely not exist.

 

[SirDice]

The new server I've just ordered has IPMI, so I can access the console just by loading a link in my browser.

If you ever run into a similar issue on that machine, the access to the console though IPMI is going to be a life-saver. It will make fixing this type of issue a breeze.

 

[rowan194]

If you ever run into a similar issue on that machine, the access to the console though IPMI is going to be a life-saver. It will make fixing this type of issue a breeze.

There is one big catch with the new server: FreeBSD doesn't seem to be supported by OVH any more - either for install, or rescue - so I'm going to have to kludge the install by booting into a Linux rescue then downloading and writing a VM image direct to the HDD. If something goes wrong, booting into a FreeBSD rescue will require fooling around with mounting devices via IPMI, which I've not done before. If this wasn't a very simple setup (basically a default install with a user account, rsync, and some firewall rules) I'd probably decide to not take on that risk. At least I'll be able to see the console!

 

[SirDice]

If you have IPMI access you can connect a CD image remotely and actually boot the machine from it. It's terribly slow over the internet but it'll work. I've used this a couple of times to remotely reinstall a server.

 

[sko]

so I'm going to have to kludge the install by booting into a Linux rescue then downloading and writing a VM image direct to the HDD.

I've used mfsBSD for such extreme cases, but as long as you have any halfway recent (i.e. ZFS-aware) FreeBSD or maybe even linux rescue image available, you can always recover by setting the "bootfs" property on the zfs pool to the previous and known-working BE - of course given that you actually created a new BE for the update!

I *highly* recommend that you make yourself familiar with boot environments - especially on remote servers with limited access methods this can be a life-saver.
You can even install the upgrade in a new BE within a jail to see if everything goes well, then reboot into that new and updated BE and just finish the process by upgrading userland/ports/packages.

 

[Cath O'Deray]

… If there is a empty /boot/loader.conf ZFS datasets can't be mounted. There has to be a zfs_load="YES" entry to load the zfs.ko kernel module.

… rescue system is UFS based. …

service zfs onestart

• FreeBSD 12.2-RELEASE-p7 (up-to-date)
• single user mode
• nothing ZFS-related in /boot/loader.conf
• UFS.

Am I missing something? Would some aspect of rescue cause service zfs onestart to not work?

 

[rowan194]

service zfs onestart

• FreeBSD 12.2-RELEASE-p7 (up-to-date)
• single user mode
• nothing ZFS-related in /boot/loader.conf
• UFS.

Am I missing something? Would some aspect of rescue cause service zfs onestart to not work?

The issue is not so much that ZFS doesn't work when using OVH's rescue boot, more that when I copied that kernel+bootcode I omitted the ZFS config in /boot/loader.conf. Having said that, I do note that the UFS based rescue was happy to import and mount the ZFS pool, without complaint, simply by using zpool import - no need to manually start the service, or load via kldload zfs. Things may be different when booting from the pool.