Solved How to move the entire OS to another disk?

Hi,
I want to move the entire OS (GELI+ZFSOnRoot) to another disk.

I had execute these commands to make this happen:
Setup my new disk partition table (similar to old one):
gpart create -s gpt nvd0
gpart add -a 1M -s 260M -t efi -l efiboot1 nvd0
gpart add -a 1M -s 4G -t freebsd-swap -l swap1 nvd0
gpart add -t freebsd-zfs -l zfsroot1 nvd0


Setup EFI Partition:
newfs_msdos -F32 -c 1 /dev/nvd0p1
mount -t msdosfs -o longnames /dev/nvd0p1 /mnt/
sudo mkdir -p /mnt/EFI/BOOT
sudo cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt/


Setup GELI:
geli init -b -g -s 4096 /dev/nvd0p3

Add the new disk to my zfs pool:
zpool attach zroot /dev/ada0p3.eli /dev/nvd0p3.eli

After completion of the resilvering, I removed my old disk.
zpool detach zroot ada0p3.eli

Now I can't boot the system:

GELI Passphrase for disk1p3:

Calculating GELI Decryption Key for disk1p3: 12345 iterations...
Calculating GELI Decryption Key for disk2p3: 12345 iterations...
zio_read error: 5
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot


I ended up to booting from memstick and adding the old disk to my pool. So hopefully my system successfully boots.
Also, I am sure that my disks and ZFS pool are healthy.

Did I miss something?
 
gpart backup ada0 | gpart restore nvd0
dd if=/dev/ada0p1 of=/dev/nvd0p1 bs=1m
I don't use encryption but it seems that geli(8) support backup/restore
 
After completion of the resilvering, I removed my old disk.
zpool remove zroot ada0p3.eli
zpool remove removes the complete vdev, not a provider. This shouldn't have worked but given you an error...
The correct command would have been zpool detach.

See zpool(8)

edit: sorry, forgot about the stupid split-up into hundrets of tiny manpages with 13.0... so for 13.x see zpool-detach(8) and zpool-remove(8)
 
try to rename /boot/zfs/zpool.cache and/or /etc/zfs/zpool.cache
does not work either. I tried to rename both files together.
 

Attachments

  • IMG_20220926_165215.jpg
    IMG_20220926_165215.jpg
    1.8 MB · Views: 104
First some suggestions about improving some actions:
  • When geli(8) initilizing the new disk use command geli init -bg -l 256 -s 4096 <provider>. When not specified -l 256, geli(8) defaults to AES-XTS 128.
  • When attaching the new disk to the pool, use the -w option, to make sure resilver is completed: zpool attach -w pool device new_device

To the issue, it seems like, in this configuration, a later added geli(8) encrypted provider with the GELIBOOT flag set (see geli list and geli(8) init -g option for details), another efi boot partition and root-on-ZFS pool (with the same name), after detaching the old device from a mirror, geli(8) booting enabled on both encrypted root filesystems, the loader gets confused and can't find a bootable partition (this looks like a bug).

Remove from ada0p3 the GELIBOOT flag (and eventually the BOOT flag. If the BOOT flag is removed, that disk won't be prompted for attaching during boot. See geli(8) init -b, -g, configure -B, -G for details):

geli configure -BG ada0p3

The above can be execute before of after detaching ada0p3.eli from the mirror.

Furthermore, on the new disk, if not done already, don't forget to modify the swap line (encrypted or not), pointing to the new device in /etc/fstab. Also make sure /boot/efi points to /dev/nvd0p1.

Last, in addition, the UEFI boot order should to be modified to boot the new system first, eventually remove the 'FreeBSD' boot variable and create a new one. See efibootmgr(8).

Reboot system.
 
First some suggestions about improving some actions:
  • When geli(8) initilizing the new disk use command geli init -bg -l 256 -s 4096 <provider>. When not specified -l 256, geli(8) defaults to AES-XTS 128.
  • When attaching the new disk to the pool, use the -w option, to make sure resilver is completed: zpool attach -w pool device new_device

To the issue, it seems like, in this configuration, a later added geli(8) encrypted provider with the GELIBOOT flag set (see geli list and geli(8) init -g option for details), another efi boot partition and root-on-ZFS pool (with the same name), after detaching the old device from a mirror, geli(8) booting enabled on both encrypted root filesystems, the loader gets confused and can't find a bootable partition (this looks like a bug).

Remove from ada0p3 the GELIBOOT flag (and eventually the BOOT flag. If the BOOT flag is removed, that disk won't be prompted for attaching during boot. See geli(8) init -b, -g, configure -B, -G for details):

geli configure -BG ada0p3

The above can be execute before of after detaching ada0p3.eli from the mirror.

Furthermore, on the new disk, if not done already, don't forget to modify the swap line (encrypted or not), pointing to the new device in /etc/fstab. Also make sure /boot/efi points to /dev/nvd0p1.

Last, in addition, the UEFI boot order should to be modified to boot the new system first, eventually remove the 'FreeBSD' boot variable and create a new one. See efibootmgr(8).

Reboot system.
Removing GELIBOOT flag from the old disk is worked. Thank you :D
It's must be a bug, I think.
 
First some suggestions about improving some actions:
  • When geli(8) initilizing the new disk use command geli init -bg -l 256 -s 4096 <provider>. When not specified -l 256, geli(8) defaults to AES-XTS 128.
I don't know if the situation has changed, but in 2009 Bruce Schneier recommended using AES128 over AES256 because of concerns about the latter's weak key schedule (link).
 
I don't know if the situation has changed, but in 2009 Bruce Schneier recommended using AES128 over AES256 because of concerns about the latter's weak key schedule (link).
The Schneier blog post you linked states:
The attack only breaks 11 rounds of AES-256. Full AES-256 has 14 rounds.

Not sure if FreeBSD's AES-256 implementation uses 14 rounds, but I found in source:

/usr/src/stand/libsa/geli/Makefile.inc
Code:
# Extra stuff for GELI
...
# AES implementation from sys/crypto
.PATH: ${SYSDIR}/crypto/rijndael
.for i in rijndael-alg-fst.c rijndael-api-fst.c rijndael-api.c

/usr/src/sys/crypto/rijndael/rijndael-alg-fst.c
/usr/src/sys/crypto/rijndael/rijndael-api.c
Code:
#include <crypto/rijndael/rijndael.h>

/usr/src/sys/crypto/rijndael/rijndael.h
Code:
#define RIJNDAEL_MAXNR  14

I don't know if that refers to 14 rounds. If one wants to know for sure better ask on mailing lists. Most of the FreeBSD developers are found there, maybe on freebsd-current@.

I suggested AES-XTS 256 because the geli(8) encrypted container for Root-on-ZFS automated installation is initialized with that key length:

/usr/src/usr.sbin/bsdinstall/scripts/zfsboot
Rich (BB code):
GELI_PASSWORD_GELIBOOT_INIT='geli init -bg -e %s -J - -l 256 -s 4096 "%s"'
 
Back
Top