Several days ago I introduced UFS Boot Environments that work great on AMD64 (or 64-bit PC if you prefer). I was interested it it will also work on less powerful devices that ZFS is not always the best idea – ARM based devices. After some testing I found out that after one simple modification the UFS Boot Environments work like a charm on ARM devices.
The Table of Contents is as follows.
There is not suitable TL;DR here – you will have to read it all or not at all this time
I currently do not own 64-bit ARM device … so I thought I will try the qemu(1) emulator and ready to download and use ARM images provided by the FreeBSD project.
First we will install needed packages and fetch the ARM64 (also known as aarch64) images.
host # pkg install -y qemu u-boot-qemu-arm64
host % fetch https://download.freebsd.org/ftp/re.../Latest/FreeBSD-13.0-RC4-arm64-aarch64.raw.xz
host % xz -d FreeBSD-13.0-RC4-arm64-aarch64.raw.xz
We will now increase the image size to add additional boot environment partition.
host % ls -lh FreeBSD-13.0-RC4-arm64-aarch64.raw
-rw-r--r-- 1 vermaden vermaden 5.1G 2021-04-04 12:37 FreeBSD-13.0-RC4-arm64-aarch64.raw
host % truncate -s +9G FreeBSD-13.0-RC4-arm64-aarch64.raw
host % ls -lh FreeBSD-13.0-RC4-arm64-aarch64.raw
-rw-r--r-- 1 vermaden vermaden 15G 2021-04-04 12:38 FreeBSD-13.0-RC4-arm64-aarch64.raw
Using qemu(1) emulator we can boot using either UEFI or U-BOOT option. We will test both as some ARM devices use UEFI and some (like Raspberry Pi devices) use U-BOOT mode.
host % export VMDISK=FreeBSD-13.0-RC4-arm64-aarch64.raw
// UEFI
host % qemu-system-aarch64 \
-m 4096M \
-cpu cortex-a57 \
-M virt \
-bios edk2-aarch64-code.fd \
-serial telnet::4444,server \
-nographic \
-drive if=none,file=${VMDISK},format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-device virtio-net-device,netdev=net0 \
-netdev user,id=net0
// U-BOOT
host % qemu-system-aarch64 \
-m 4096M \
-cpu cortex-a57 \
-M virt \
-bios /usr/local/share/u-boot/u-boot-qemu-arm64/u-boot.bin \
-serial telnet::4444,server \
-nographic \
-drive if=none,file=${VMDISK},format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-device virtio-net-device,netdev=net0 \
-netdev user,id=net0
After starting the qemu(1) process it will display the following information.
(...)
QEMU 5.0.1 monitor - type 'help' for more information
(qemu) qemu-system-aarch64: -serial telnet::4444,server: info: QEMU waiting for connection on: disconnected:telnet::::4444,server
We can now use telnet(1) to connect to our serial console on emulated ARM64 system. We will add additional freebsd-ufs partition for our second boot environment.
host % telnet localhost 4444
(...)
login: root
ARM # pkg install -y lsblk
ARM # lsblk
DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT
vtbd0 0:62 14G GPT - -
vtbd0p1 0:63 33M efi gpt/efiesp /boot/efi
vtbd0p2 0:64 1.0G freebsd-swap gpt/swapfs -
vtbd0p3 0:65 4.0G freebsd-ufs ufs/rootfs /
ARM # gpart show
=> 3 10552344 vtbd0 GPT (14G) [CORRUPT]
3 66584 1 efi (33M)
66587 2097152 2 freebsd-swap (1.0G)
2163739 8388608 3 freebsd-ufs (4.0G)
ARM # gpart recover vtbd0
vtbd0 recovered
ARM # gpart add -s 4G -t freebsd-ufs vtbd0
vtbd0p4 added
ARM # gpart show
=> 3 29426709 vtbd0 GPT (14G)
3 66584 1 efi (33M)
66587 2097152 2 freebsd-swap (1.0G)
2163739 8388608 3 freebsd-ufs (4.0G)
10552347 8388608 4 freebsd-ufs (4.0G)
18940955 10485757 - free - (5.0G)
We will now make some manual preparations for ufsbe.sh to work.
For example the FreeBSD images come with GPT labels used in /etc/fstab file which are currently not supported by UFS Boot Environments so we will modify the /etc/fstab file to mount root filesystem from raw devices and partitions.
ARM # mkdir -p /ufsbe/3 /ufsbe/4
ARM # cat /etc/fstab
# Custom /etc/fstab for FreeBSD VM images
/dev/gpt/rootfs / ufs rw 1 1
/dev/gpt/efiesp /boot/efi msdosfs rw 2 2
/dev/gpt/swapfs none swap sw 0 0
ARM # vi /etc/fstab
ARM # cat /etc/fstab
# Custom /etc/fstab for FreeBSD VM images
/dev/vtbd0p3 / ufs rw 1 1
/dev/vtbd0p4 /ufsbe/4 ufs rw 1 1
/dev/gpt/efiesp /boot/efi msdosfs rw 2 2
/dev/gpt/swapfs none swap sw 0 0
ARM # newfs /dev/vtbd0p4
/dev/vtbd0p4: 4096.0MB (8388608 sectors) block size 32768, fragment size 4096
using 7 cylinder groups of 625.22MB, 20007 blks, 80128 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880
We now have second boot environment ready and /etc/fstab file modified to boot from raw devices instead of GPT labels. We will now reboot(8) to make these changes apply.
ARM # reboot
We will not fetch the ufsbe.sh command and finish the setup process.
ARM # fetch https://raw.githubusercontent.com/vermaden/ufsbe/main/ufsbe.sh
ARM # chmod +x ./ufsbe.sh
ARM # ./ufsbe.sh
NOPE: did not found boot environment setup with 'ufsbe' label
INFO: setup each boot environment partition with appropriate label
HELP: list all 'freebsd-ufs' partitions type:
# gpart show -p | grep freebsd-ufs
2098216 33554432 ada0p3 freebsd-ufs [bootme] (16G)
35652648 33554432 ada0p4 freebsd-ufs (16G)
69207080 33554432 ada0p5 freebsd-ufs (16G)
HELP: to setup partitions 3/4/5 as boot environments type:
# gpart modify -i 3 -l ufsbe/3 ada0
# gpart modify -i 4 -l ufsbe/4 ada0
# gpart modify -i 5 -l ufsbe/5 ada0
ARM # gpart show
=> 3 29426709 vtbd0 GPT (14G)
3 66584 1 efi (33M)
66587 2097152 2 freebsd-swap (1.0G)
2163739 8388608 3 freebsd-ufs (4.0G)
10552347 8388608 4 freebsd-ufs (4.0G)
18940955 10485757 - free - (5.0G)
ARM # gpart modify -i 3 -l ufsbe/3 vtbd0
vtbd0p3 modified
ARM # gpart modify -i 4 -l ufsbe/4 vtbd0
vtbd0p4 modified
ARM # ./ufsbe.sh
INFO: flag 'bootme' successfully set on / filesystem
usage:
ufsbe.sh (l)ist
ufsbe.sh (a)ctivate
ufsbe.sh (s)ync
The UFS Boot Environments are now properly deployed on this ARM64 test system.
At the first try I was not able to use UFS Boot Environments as the bootme flag was ignored.
I then submitted a FreeBSD bug – 254764 – GPT ‘bootme’ flag is not respected on AARCH64 – to make sure I am doing everything well on my side. As it turns out the bootme flag is a FreeBSD specific extension and nobody else uses it. The needed fix is to copy /boot/gptboot.efi in place of bootaa64.efi file.
Lets now make that fix.
ARM # cp /boot/gptboot.efi /boot/efi/EFI/BOOT/bootaa64.efi
We will now synchronize boot environments 3 and 4 and then reboot into the 4 boot environments.
ARM # ./ufsbe.sh list
PROVIDER LABEL ACTIVE
vtbd0p3 ufsbe/3 NR
vtbd0p4 ufsbe/4 -
ARM # ./ufsbe.sh sync 3 4
INFO: syncing '3' (source) => '4' (target) boot environments ...
INFO: boot environments '3' (source) => '4' (target) synced
ARM # ./ufsbe.sh activate 4
INFO: boot environment '4' now activated
ARM # reboot
After the reboot the currently active boot environment is 4. It means that UFS Boot Environments work properly on ARM devices.
ARM # ./ufsbe.sh list
PROVIDER LABEL ACTIVE
vtbd0p3 ufsbe/3 -
vtbd0p4 ufsbe/4 NR
ARM # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/vtbd0p4 3.9G 2.6G 935M 74% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/vtbd0p3 3.9G 2.6G 932M 74% /ufsbe/3
/dev/gpt/efiesp 32M 1.3M 31M 4% /boot/efi
I have tested both U-BOOT and UEFI boot modes and they both allow to use UFS Boot Environments.
EOF
Continue reading...
The Table of Contents is as follows.
- ARM Testing
- Setup UFS Boot Environments
- Needed Fix to Make FreeBSD bootme Flags Work
- Reboot into Other Boot Environment Test
There is not suitable TL;DR here – you will have to read it all or not at all this time
ARM Testing
I currently do not own 64-bit ARM device … so I thought I will try the qemu(1) emulator and ready to download and use ARM images provided by the FreeBSD project.
First we will install needed packages and fetch the ARM64 (also known as aarch64) images.
host # pkg install -y qemu u-boot-qemu-arm64
host % fetch https://download.freebsd.org/ftp/re.../Latest/FreeBSD-13.0-RC4-arm64-aarch64.raw.xz
host % xz -d FreeBSD-13.0-RC4-arm64-aarch64.raw.xz
We will now increase the image size to add additional boot environment partition.
host % ls -lh FreeBSD-13.0-RC4-arm64-aarch64.raw
-rw-r--r-- 1 vermaden vermaden 5.1G 2021-04-04 12:37 FreeBSD-13.0-RC4-arm64-aarch64.raw
host % truncate -s +9G FreeBSD-13.0-RC4-arm64-aarch64.raw
host % ls -lh FreeBSD-13.0-RC4-arm64-aarch64.raw
-rw-r--r-- 1 vermaden vermaden 15G 2021-04-04 12:38 FreeBSD-13.0-RC4-arm64-aarch64.raw
Using qemu(1) emulator we can boot using either UEFI or U-BOOT option. We will test both as some ARM devices use UEFI and some (like Raspberry Pi devices) use U-BOOT mode.
host % export VMDISK=FreeBSD-13.0-RC4-arm64-aarch64.raw
// UEFI
host % qemu-system-aarch64 \
-m 4096M \
-cpu cortex-a57 \
-M virt \
-bios edk2-aarch64-code.fd \
-serial telnet::4444,server \
-nographic \
-drive if=none,file=${VMDISK},format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-device virtio-net-device,netdev=net0 \
-netdev user,id=net0
// U-BOOT
host % qemu-system-aarch64 \
-m 4096M \
-cpu cortex-a57 \
-M virt \
-bios /usr/local/share/u-boot/u-boot-qemu-arm64/u-boot.bin \
-serial telnet::4444,server \
-nographic \
-drive if=none,file=${VMDISK},format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-device virtio-net-device,netdev=net0 \
-netdev user,id=net0
After starting the qemu(1) process it will display the following information.
(...)
QEMU 5.0.1 monitor - type 'help' for more information
(qemu) qemu-system-aarch64: -serial telnet::4444,server: info: QEMU waiting for connection on: disconnected:telnet::::4444,server
We can now use telnet(1) to connect to our serial console on emulated ARM64 system. We will add additional freebsd-ufs partition for our second boot environment.
host % telnet localhost 4444
(...)
login: root
ARM # pkg install -y lsblk
ARM # lsblk
DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT
vtbd0 0:62 14G GPT - -
vtbd0p1 0:63 33M efi gpt/efiesp /boot/efi
vtbd0p2 0:64 1.0G freebsd-swap gpt/swapfs -
vtbd0p3 0:65 4.0G freebsd-ufs ufs/rootfs /
ARM # gpart show
=> 3 10552344 vtbd0 GPT (14G) [CORRUPT]
3 66584 1 efi (33M)
66587 2097152 2 freebsd-swap (1.0G)
2163739 8388608 3 freebsd-ufs (4.0G)
ARM # gpart recover vtbd0
vtbd0 recovered
ARM # gpart add -s 4G -t freebsd-ufs vtbd0
vtbd0p4 added
ARM # gpart show
=> 3 29426709 vtbd0 GPT (14G)
3 66584 1 efi (33M)
66587 2097152 2 freebsd-swap (1.0G)
2163739 8388608 3 freebsd-ufs (4.0G)
10552347 8388608 4 freebsd-ufs (4.0G)
18940955 10485757 - free - (5.0G)
We will now make some manual preparations for ufsbe.sh to work.
For example the FreeBSD images come with GPT labels used in /etc/fstab file which are currently not supported by UFS Boot Environments so we will modify the /etc/fstab file to mount root filesystem from raw devices and partitions.
ARM # mkdir -p /ufsbe/3 /ufsbe/4
ARM # cat /etc/fstab
# Custom /etc/fstab for FreeBSD VM images
/dev/gpt/rootfs / ufs rw 1 1
/dev/gpt/efiesp /boot/efi msdosfs rw 2 2
/dev/gpt/swapfs none swap sw 0 0
ARM # vi /etc/fstab
ARM # cat /etc/fstab
# Custom /etc/fstab for FreeBSD VM images
/dev/vtbd0p3 / ufs rw 1 1
/dev/vtbd0p4 /ufsbe/4 ufs rw 1 1
/dev/gpt/efiesp /boot/efi msdosfs rw 2 2
/dev/gpt/swapfs none swap sw 0 0
ARM # newfs /dev/vtbd0p4
/dev/vtbd0p4: 4096.0MB (8388608 sectors) block size 32768, fragment size 4096
using 7 cylinder groups of 625.22MB, 20007 blks, 80128 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880
We now have second boot environment ready and /etc/fstab file modified to boot from raw devices instead of GPT labels. We will now reboot(8) to make these changes apply.
ARM # reboot
Setup UFS Boot Environments
We will not fetch the ufsbe.sh command and finish the setup process.
ARM # fetch https://raw.githubusercontent.com/vermaden/ufsbe/main/ufsbe.sh
ARM # chmod +x ./ufsbe.sh
ARM # ./ufsbe.sh
NOPE: did not found boot environment setup with 'ufsbe' label
INFO: setup each boot environment partition with appropriate label
HELP: list all 'freebsd-ufs' partitions type:
# gpart show -p | grep freebsd-ufs
2098216 33554432 ada0p3 freebsd-ufs [bootme] (16G)
35652648 33554432 ada0p4 freebsd-ufs (16G)
69207080 33554432 ada0p5 freebsd-ufs (16G)
HELP: to setup partitions 3/4/5 as boot environments type:
# gpart modify -i 3 -l ufsbe/3 ada0
# gpart modify -i 4 -l ufsbe/4 ada0
# gpart modify -i 5 -l ufsbe/5 ada0
ARM # gpart show
=> 3 29426709 vtbd0 GPT (14G)
3 66584 1 efi (33M)
66587 2097152 2 freebsd-swap (1.0G)
2163739 8388608 3 freebsd-ufs (4.0G)
10552347 8388608 4 freebsd-ufs (4.0G)
18940955 10485757 - free - (5.0G)
ARM # gpart modify -i 3 -l ufsbe/3 vtbd0
vtbd0p3 modified
ARM # gpart modify -i 4 -l ufsbe/4 vtbd0
vtbd0p4 modified
ARM # ./ufsbe.sh
INFO: flag 'bootme' successfully set on / filesystem
usage:
ufsbe.sh (l)ist
ufsbe.sh (a)ctivate
ufsbe.sh (s)ync
The UFS Boot Environments are now properly deployed on this ARM64 test system.
Needed Fix to Make FreeBSD bootme Flags Work
At the first try I was not able to use UFS Boot Environments as the bootme flag was ignored.
I then submitted a FreeBSD bug – 254764 – GPT ‘bootme’ flag is not respected on AARCH64 – to make sure I am doing everything well on my side. As it turns out the bootme flag is a FreeBSD specific extension and nobody else uses it. The needed fix is to copy /boot/gptboot.efi in place of bootaa64.efi file.
Lets now make that fix.
ARM # cp /boot/gptboot.efi /boot/efi/EFI/BOOT/bootaa64.efi
Reboot into Other Boot Environment Test
We will now synchronize boot environments 3 and 4 and then reboot into the 4 boot environments.
ARM # ./ufsbe.sh list
PROVIDER LABEL ACTIVE
vtbd0p3 ufsbe/3 NR
vtbd0p4 ufsbe/4 -
ARM # ./ufsbe.sh sync 3 4
INFO: syncing '3' (source) => '4' (target) boot environments ...
INFO: boot environments '3' (source) => '4' (target) synced
ARM # ./ufsbe.sh activate 4
INFO: boot environment '4' now activated
ARM # reboot
After the reboot the currently active boot environment is 4. It means that UFS Boot Environments work properly on ARM devices.
ARM # ./ufsbe.sh list
PROVIDER LABEL ACTIVE
vtbd0p3 ufsbe/3 -
vtbd0p4 ufsbe/4 NR
ARM # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/vtbd0p4 3.9G 2.6G 935M 74% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/vtbd0p3 3.9G 2.6G 932M 74% /ufsbe/3
/dev/gpt/efiesp 32M 1.3M 31M 4% /boot/efi
I have tested both U-BOOT and UEFI boot modes and they both allow to use UFS Boot Environments.
EOF
Continue reading...