bectl & upgrade with full rollback ability

Selin

Member

Reaction score: 9
Messages: 36

Hi

I'd like to figure out how to do an upgrade with the ability to rollback fully if anything goes wrong.
Taking into account the info from the following topics:
Initial system state
Code:
> uname -a
FreeBSD test 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

> freebsd-version -kru
12.1-RELEASE
12.1-RELEASE
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed

I do the following:
  1. zfs snap -r zroot@original
  2. bectl create -r new
  3. bectl mount new
  4. chroot {path to mounted "new" snapshot}
  5. mount -t devfs devfs /dev
  6. rm -rf /var/db/freebsd-update
  7. mkdir /var/db/freebsd-update
  8. freebsd-update upgrade -r {NEW_VERSION}
  9. freebsd-update install
  10. bectl activate new
  11. reboot
  12. What I have at this point
    Code:
    > uname -a
    FreeBSD test 13.0-RC3 FreeBSD 13.0-RC3 #0 releng/13.0-n244696-8f731a397ad: Fri Mar 19 04:00:20 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
    
    > freebsd-version -kru
    13.0-RC3
    13.0-RC3
    12.1-RELEASE
    
    > pkg info
    dialog4ports-0.1.6             Console Interface to configure ports
    diffutils-3.7                  GNU differential compare utilities
    gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
    gettext-tools-0.20.1_1         GNU gettext development and translation tools
    glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
    gmake-4.2.1_3                  GNU version of 'make' utility
    indexinfo-0.3.1                Utility to regenerate the GNU info page index
    libffi-3.2.1_3                 Foreign Function Interface
    libiconv-1.14_11               Character set conversion library
    libsigsegv-2.12                Handling page faults in user mode
    libtextstyle-0.20.1            Text styling library
    mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
    pcre-8.43_2                    Perl Compatible Regular Expressions library
    perl5-5.30.0                   Practical Extraction and Report Language
    pkg-1.12.0                     Package manager
    pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
    python27-2.7.16_1              Interpreted object-oriented programming language
    readline-8.0.0                 Library for editing command lines as they are typed
  13. freebsd-update install
  14. rebuild/update all ports/packages installed
  15. freebsd-update install
  16. gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
  17. reboot
Final System state
Code:
> uname -a
FreeBSD test 13.0-RC3 FreeBSD 13.0-RC3 #0 releng/13.0-n244696-8f731a397ad: Fri Mar 19 04:00:20 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

> freebsd-version -kru
13.0-RC3
13.0-RC3
13.0-RC3

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.21           GNU gettext runtime libraries and programs
gettext-tools-0.21             GNU gettext development and translation tools
glib-2.66.7_1,1                Some useful routines of C programming (current stable version)
gmake-4.3_2                    GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libiconv-1.16                  Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.21              Text styling library
mc-nox11-4.8.26                Midnight Commander, a free Norton Commander Clone
meson-0.57.1                   High performance build system
ninja-1.10.2,2                 Small build system closest in spirit to Make
pcre-8.44                      Perl Compatible Regular Expressions library
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.16.3                     Package manager
pkgconf-1.7.4,1                Utility to help to configure compiler and linker flags
py37-setuptools-44.0.0         Python packages installer
python37-3.7.10                Interpreted object-oriented programming language
readline-8.1.0                 Library for editing command lines as they are typed

Here I have my system upgraded and what to rollback to the initial state (including kernel and versions of all installed ports).
So I'm going to do the following:
  1. zfs snap -r zroot@new
  2. bectl activate default
  3. reboot # here I'm booting from LiveCD
  4. import and mount my zpool
  5. cd {mount dir}
  6. mkdir backup
  7. mv bin boot etc lib libexec rescue sbin usr var backup/
  8. cp -pR {mount dir}/.zfs/snapshot/original/. {mount dir}
  9. gpart bootcode -p {mount dir}/boot/boot1.efifat -i 1 nvd0
Rollback result
Code:
> uname -a
FreeBSD test 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

> freebsd-version -kru
12.1-RELEASE
12.1-RELEASE
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed

Now it seems the system is the same as I had before the upgrade.
Is that correct?
Or I should adjust them somehow?

If some user files were updated in a new system - they can be copied from the /.zfs/snapshot/new/usr/home/{user}

And what about this part of the upgrade/rollback?
gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
Is that correct to update/rollback UEFI loader that way?

Thanks
 

Zirias

Daemon

Reaction score: 1,394
Messages: 2,441

I only had a quick look now, but doing a global snapshot of zroot looks a bit over the top to me. Boot environments already handle the base system. If you also upgrade ports/packages and /usr/local is on a different dataset (which isn't the case in the structure created by the installer), just add a snapshot there as well.

As for things like /var and /usr/home or /home, you should probably think twice whether you want to roll them back at all. Normally, a boot environment will be enough. As long as you don't upgrade your pool (you shouldn't unless you're SURE you don't want to go back), bootcode shouldn't be an issue either, but if you already updated it, just rewrite it from your other BE to go back.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,953
Messages: 38,398

Note that with 13.0 that boot1.efifat file doesn't exist anymore. You need to mount the efi partition with msdosfs(5) and copy /boot/loader.efi to EFI/BOOT/bootx64.efi (FAT is not case-sensitive, so actual case doesn't matter) of that filesystem.

There shouldn't be a need to rollback the EFI loader, it's backwards compatible. You can boot 12.x with the 13.0 loader.efi but not the other way around.
 
OP
S

Selin

Member

Reaction score: 9
Messages: 36

I need to rollback the whole system state: kernel + set of ports installed + all their configs, etc.
Sometimes it might be needed to rollback some application config and data. E.g. DB files might be converted to a new format after DB engine upgrade.
So I decided to do a full backup.

Here is the ZFS structure I have with default installation:
Code:
> zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot               6.24G  6.84G    88K  /zroot
zroot/ROOT          4.00G  6.84G    88K  none
zroot/ROOT/default  2.47G  6.84G  1.89G  /
zroot/tmp            336K  6.84G   120K  /tmp
zroot/usr           2.22G  6.84G    88K  /usr
zroot/usr/home       268K  6.84G   128K  /usr/home
zroot/usr/ports     1.53G  6.84G   880M  /usr/ports
zroot/usr/src        704M  6.84G   704M  /usr/src
zroot/var            932K  6.84G    88K  /var
zroot/var/audit       88K  6.84G    88K  /var/audit
zroot/var/crash       88K  6.84G    88K  /var/crash
zroot/var/log        420K  6.84G   156K  /var/log
zroot/var/mail        88K  6.84G    88K  /var/mail
zroot/var/tmp        160K  6.84G    88K  /var/tmp

bectl -r ... creates snapshot only for the zroot/ROOT/default volume.

What about zroot/usr - isn't it affected during upgrading the system and ports?
 

Zirias

Daemon

Reaction score: 1,394
Messages: 2,441

I wonder how your /usr ended up containing anything, it should normally be set to canmount=no, so everything in /usr (except for the specific datasets you create below) will be stored on your root dataset.

edit: forget the nonsense, looked at the wrong column. Your zroot/usr seems to be empty, as expected.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,953
Messages: 38,398

What about zroot/usr - isn't it affected during upgrading the system and ports?
It's not mounted, it's only there to provide easy access to for /usr/home, /usr/ports and /usr/src. The actual data in /usr/ is stored within zroot/ROOT/default.
Code:
root@molly:~ # zfs get canmount zroot/usr
NAME       PROPERTY  VALUE     SOURCE
zroot/usr  canmount  off       local

/usr/ports doesn't need to be rolled back (or forward), all versions use the same ports tree. /usr/src/ doesn't really matter either, since it's version controlled you can more easily deal with that using svnlite(1) or git(1) (just switch branches).
 
OP
S

Selin

Member

Reaction score: 9
Messages: 36

Thanks everybody for advices.

Updated my notes to the following:

Upgrade
  1. bectl create -r {new}
  2. bectl mount {new}
  3. chroot {path to mounted "new" snapshot}
  4. mount -t devfs devfs /dev
  5. rm -rf /var/db/freebsd-update
  6. mkdir /var/db/freebsd-update
  7. freebsd-update upgrade -r {NEW_VERSION}
  8. freebsd-update install
  9. bectl activate {new}
  10. reboot
  11. freebsd-update install
  12. rebuild/update all ports/packages installed
  13. freebsd-update install
  14. gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
  15. reboot
  16. check if everything works good because after the next step there will be no ability to rollback
  17. zpool upgrade -a
Rollback
  1. bectl activate {old}
  2. chroot {path to mounted "old" snapshot}
  3. mount -t devfs devfs /dev
  4. gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
  5. reboot
NOTE
In 13.0 the boot1.efifat file doesn't exist anymore.
So instead of gpart bootcode -p /boot/boot1.efifat -i 1 nvd0 need to mount the efi partition with msdosfs(5) and copy /boot/loader.efi to EFI/BOOT/bootx64.efi
 

mtu

Active Member

Reaction score: 110
Messages: 166

Upgrade
  1. bectl create -r {new}
  2. bectl mount {new}
  3. chroot {path to mounted "new" snapshot}
  4. mount -t devfs devfs /dev
  5. rm -rf /var/db/freebsd-update
  6. mkdir /var/db/freebsd-update
  7. freebsd-update upgrade -r {NEW_VERSION}
  8. freebsd-update install
  9. bectl activate {new}
  10. reboot
  11. freebsd-update install
  12. rebuild/update all ports/packages installed
  13. freebsd-update install
  14. gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
  15. reboot
  16. check if everything works good because after the next step there will be no ability to rollback
  17. zpool upgrade -a
Note that you can take as much time as you want for the last step. You might miss out on some cool new features, but otherwise there's no penalty for leaving your zpool at an older version.
 

Mjölnir

Daemon

Reaction score: 1,509
Messages: 2,114

  1. bectl create previous
  2. Perform freebsd-update(8) (on default boot environment)... Of course, the chroot(8) method's advantage is reduced interrupt of service
    EDIT or use freebsd-update -b <mountpoint> /EDIT
  3. reboot(8), run into issues, decide to rollback
"Rollback":
  1. bectl activate previous
  2. reboot
  3. bectl destroy default
  4. bectl rename previous default
Conclusio: freebsd-update rollback is only for UFS, and then only if you don't have 2 root filesystem partitions to switch between (e.g. IIRC nanobsd(8) does that by default).
 

Zirias

Daemon

Reaction score: 1,394
Messages: 2,441

FWIW, if you install from source, you can fully install a new release without the need for chroot:
Code:
bectl create 13rc4
bectl mount 13rc4
cd /usr/src
make DESTDIR=/tmp/be-XXXX installkernel
make DESTDIR=/tmp/be-XXXX installworld
make DESTDIR=/tmp/be-XXXX BATCH_DELETE_OLD_FILES=yes delete-old
bectl umount 13rc4
bectl activate 13rc4

But then, having to use chroot isn't that bad either. When upgrading to a new release, I'd always prepare it in the new BE, not in the currently running one, because otherwise it isn't safe to install the userland without booting with the new kernel first.

Edit: Some support for a destination dir in freebsd-update(8) would be nice!
 

Zirias

Daemon

Reaction score: 1,394
Messages: 2,441

Ha! I didn't even look, cause everyone seems to describe the chroot method. Looks like it isn't necessary after all.
 

grahamperrin

Daemon

Reaction score: 500
Messages: 1,619

The chroot method is appealing for its ability to streamline things, however it's too easy for things to go wrong if the end user follows on-screen instructions.
 
Top