manage boot environment from rescue system?

Hi,

somehow I bricked my server. I created a new BE (for 12.2-p8), mounted it, chrooted into it, updated it - all as I always do. Then I activated it with -t (onetime boot) and rebooted the system. It didn't came up. Then I reset the system but it is still not reachable. So the older BE doesn't work either anymore.

I don't have console access, but I can boot a FreeBSD 13 rescue system.

From there I am able to import the zfs pool and even mount the current BE, but I can't use bectl in order to activate a previous BE (there are 2 more available).

So, how can I manage (or at least switch) boot environments from a rescue system?


Thanks in advance,
Tom
 
Did you try import pool, mount current BE, chroot into that current BE and then try to use bectl to activate the desired BE? Activate, not "one time boot".
 
I can import the pool, mount the BE and chroot into it, but bectl list shows nothing.

Code:
[root@rescue ~]# zpool import -R /mnt 5285261849149729362
[root@rescue ~]# zfs list
NAME                 USED  AVAIL     REFER  MOUNTPOINT
zroot               37.0G  2.60T       88K  /mnt/zroot
zroot/ROOT          5.92G  2.60T       88K  none
zroot/ROOT/default  9.58M  2.60T     1.89G  /mnt
zroot/ROOT/r122      660K  2.60T     3.07G  /mnt
zroot/ROOT/r122-1    720K  2.60T     3.42G  /mnt
zroot/ROOT/r122-6   5.78G  2.60T     3.69G  /mnt/tmp/mnt
zroot/ROOT/r122-p8   133M  2.60T     3.69G  /mnt
(..)
[root@rescue ~]# zfs mount zroot/ROOT/r122-6
[root@rescue ~]# chroot /mnt
[root@rescue /]# bectl list
(no output)

Now I also remember what I exactly did. I created the BE r122-p8, then started freebsd-update. After the install I realised that I was still on r122-6. So I executed 'freebsd-update rollback', chrooted into r122-p8, did the update there, activated it with -t, rebooted and gut stuck.

Meanwhile I also have console access and this is what I get (see attachment). So as can be seen the fallback to the previous BE worked but this is bricked as well.

From rescue I can mount it (see above) and I tried to copy /boot/kernel.old/ to /boot/kernel/ and tried a reboot, but the messages are unchanged.

Also, there's no prompt, so I can't do anything at this point from the console.
 

Attachments

  • befail.png
    befail.png
    136.4 KB · Views: 62
PS: After:
Code:
mount -t devfs devfs /mnt/dev

I can now use bectl, however:

Code:
[root@rescue /]# bectl list
BE      Active Mountpoint Space Created
default -      -          59.6M 2020-10-26 16:49
r122    -      -          161M  2020-11-09 18:23
r122-1  -      -          262M  2020-12-13 13:47
r122-6  NR     /mnt       5.78G 2021-05-21 09:44
r122-p8 -      -          133M  2021-06-26 10:50

[root@rescue /]# bectl activate r122-1
cannot promote 'zroot/ROOT/r122-1': not a cloned filesystem
Did not successfully activate boot environment r122-1

now what?
 
Huh. The last activate worked regardless of the error message. Now I have 12.1 booted:

Code:
[26.Jun 15:24:07] --- [~] ---
root@e3: # bectl list
BE        Active Mountpoint Space Created
default   -      -          59.6M 2020-10-26 16:49
r122      -      -          161M  2020-11-09 18:23
r122-1    -      -          262M  2020-12-13 13:47
r122-6    -      -          5.78G 2021-05-21 09:44
r122-p8   -      -          133M  2021-06-26 10:50
r221clone NR     /          724K  2021-06-26 16:29

[26.Jun 15:24:11] --- [~] ---
root@e3: # uname -a
FreeBSD e3.daemon.de 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC  amd64

So, I am now cloning this, updating it and will try to boot that one.
 
Ok, worked like a charm. So, finally, this was the command which bricked the system:

Code:
freebsd-update rollback
 
Ok, worked like a charm. So, finally, this was the command which bricked the system:

Code:
freebsd-update rollback
Sorry I wasn't replying; wife had some yard work for me to do. Glad you got it working again.

The rollback command, did you do that while you were chrooted into the BE you upgraded in?
I think bectl mucks with some properties of the dataset when you do "activate". I'm not sure what they are, but in theory, you could do those by hand with zpool get/set operations.
beadm is a shell script it may be easy to trace there. bectl is an executable; of course you could look at the source and get what properties are mucked with.
 
For future reference, it's sometimes useful to be aware of the value of the bootfs property of the relevant pool.

Here, for example:

Code:
% zpool get bootfs copperbowl
NAME        PROPERTY  VALUE                                  SOURCE
copperbowl  bootfs    copperbowl/ROOT/n247565-b49ba74deeb-b  local
% bectl list -c creation
BE                    Active Mountpoint Space Created
n246499-097e8701c9f-d -      -          12.4G 2021-05-12 16:33
n247326-2349cda44fe-b -      -          1022M 2021-06-12 09:36
n247326-2349cda44fe-c -      -          218M  2021-06-17 04:15
n247326-2349cda44fe-d -      -          525M  2021-06-19 07:41
n247326-2349cda44fe-e -      -          315M  2021-06-22 19:01
n247326-2349cda44fe-f -      -          45.0M 2021-06-24 11:10
n247565-b49ba74deeb-a -      -          18.6M 2021-06-26 03:06
n247565-b49ba74deeb-b NR     /          66.1G 2021-06-26 05:26
%

In a problem situation, it might suffice to use any system (not necessarily a rescue system) to set the value.

zpool-get(8)
 
The rollback command, did you do that while you were chrooted into the BE you upgraded in?
Nope, I was on the host. That is:

1) @host: bectl create ...
2) @host: freebsd-update fetch
3) @host: freebsd-update install
4) @host: freebsd-update rollback
5) @host: bectl mount ...
6) @host: chroot ...
7) @chroot: freebsd-update fetch+install
8) @host: bectl activate -t ...
8) @host: reboot
9) login to forums.freebsd.org :)

So steps 2+3 were wrong, I forgot to chroot into the new BE. The rollback did then somehow destroy the old AND the new BE. Once I had managed to activate an old BE (which thanks goodness was still there) I could restore everything.

In a problem situation, it might suffice to use any system (not necessarily a rescue system) to set the value.

Ok, and how exactly could I have done this in my situation?
 
  • Like
Reactions: mer
Nope, I was on the host. That is:

1) @host: bectl create ...
2) @host: freebsd-update fetch
3) @host: freebsd-update install
4) @host: freebsd-update rollback
5) @host: bectl mount ...
6) @host: chroot ...
7) @chroot: freebsd-update fetch+install
8) @host: bectl activate -t ...
8) @host: reboot
9) login to forums.freebsd.org :)

So steps 2+3 were wrong, I forgot to chroot into the new BE. The rollback did then somehow destroy the old AND the new BE. Once I had managed to activate an old BE (which thanks goodness was still there) I could restore everything.



Ok, and how exactly could I have done this in my situation?
This exact sequence is important to understanding "how you go mucked up". :)
I always have a cheat sheet around and triple check before hitting Enter.
 
So looking at beadm (pkg install beadm) for the activate command, there is a bunch of stuff going on. Mucks with canmount, mountpoint, bootfs, the sysctl "vfs.root.mountfrom" so a lot of stuff, but could be done :)
 
Back
Top