Solved Manual install to mirrored zfs

[Paul Floyd]

I'd like to install to a slightly custom partitioning, with zroot not using all of the free disk space as happens when doing a guided root-on-zfs install. I can't see any way to do this with Manual Disk Setup. So that leaves Shell. I've been practicing on VirtualBox.

EDIT: It looks like I've cracked this. What I did was run the guided root-on-zfs install. I then saved /tmp/bsdinstall_log to a different computer. After that I modified my setup script based on the bsdinstall_log. I kept my modified gpart commands, and used the same zfs/zpool commands as done by bsdinstall. Obviously you need to change the disk IDs and sizes to match your situation. TAKE CARE! gpart destroy does what it says.

I've seen lots of scripts to do this.
[snipped]

What I've been doing

1. boot installer, setup keyboard etc
2. drop to shell for partitioning
3. start network
4. run the script below
5. stop network
6. exit the shell and continue installer, reboot

zfs-setup.sh:

#!/bin/sh

msg() {
    echo -e "\033[1;32m >> \033[1;37m$1\033[0m..."
}

msg "Destroying GPTs"
gpart destroy -F ada0
gpart destroy -F ada1

msg "Creating GPTs"
gpart create -s gpt ada0
gpart create -s gpt ada1

msg "Creating partitions"
gpart add -s 512K -a 4k -t freebsd-boot ada0
gpart add -s 1G -a 4k -t freebsd-swap -l swap0 ada0
gpart add -s 10G -a 4k -t freebsd-zfs -l zroot0 ada0
gpart add -a 4k -t freebsd-zfs -l data0 ada0

gpart add -s 512K -a 4k -t freebsd-boot ada1
gpart add -s 1G -a 4k -t freebsd-swap -l swap1 ada1
gpart add -s 10G -a 4k -t freebsd-zfs -l zroot1 ada1
gpart add -a 4k -t freebsd-zfs -l data1 ada1

msg "Install bootcode"
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

msg "Creating zroot"
zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f "zroot" mirror  ada0p3 ada1p3
#zfs set checksum=fletcher4 zroot

msg "Creating zfs dirs"
zfs create -o mountpoint=none "zroot/ROOT"
zfs create -o mountpoint=/ "zroot/ROOT/default"
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off "zroot/tmp"
zfs create -o mountpoint=/usr -o canmount=off "zroot/usr"
zfs create  "zroot/usr/home"
zfs create -o setuid=off "zroot/usr/ports"
zfs create  "zroot/usr/src"
zfs create -o mountpoint=/var -o canmount=off "zroot/var"
zfs create -o exec=off -o setuid=off "zroot/var/audit"
zfs create -o exec=off -o setuid=off "zroot/var/crash"
zfs create -o exec=off -o setuid=off "zroot/var/log"
zfs create -o atime=on "zroot/var/mail"
zfs create -o setuid=off "zroot/var/tmp"

msg "Configuring zroot"
zfs set "mountpoint=/zroot" "zroot"
mkdir -p "/mnt/tmp"
chmod 1777 "/mnt/tmp"
mkdir -p "/mnt/var/tmp"
chmod 1777 "/mnt/var/tmp"
zpool set bootfs="zroot/ROOT/default" "zroot"
mkdir -p "/mnt/boot/zfs"
zpool set cachefile="/mnt/boot/zfs/zpool.cache" "zroot"
zfs set "canmount=noauto" "zroot/ROOT/default"

msg "Populating etc and boot config files"
echo "# Device" "Mountpoint" "FStype" "Options" "Dump" "Pass#" >> "/tmp/bsdinstall_etc/fstab"
echo "/dev/ada0p2" "none" "swap" "sw" "0" "0" >> "/tmp/bsdinstall_etc/fstab"
echo "/dev/ada1p2" "none" "swap" "sw" "0" "0" >> "/tmp/bsdinstall_etc/fstab"
echo "zfs_enable=\"YES\"" >> "/tmp/bsdinstall_etc/rc.conf.zfs"
echo "vfs.zfs.min_auto_ashift=12" >> "/tmp/bsdinstall_etc/sysctl.conf.zfs"

echo "kern.geom.label.disk_ident.enable=\"0\"" >> "/tmp/bsdinstall_boot/loader.conf.zfs"
echo "kern.geom.label.gptid.enable=\"0\"" >> "/tmp/bsdinstall_boot/loader.conf.zfs"

 

[mrtuple]

Well, one good turn deserves another. The OPs overview and script gave me a good head start at a custom ZFS install for two Dell 7440s with a "mini" 128GB drive and a larger standard SATA 466GB drive. But looking through the /tmp/bsdinstall_log made my eyes go buggy so I whipped up this script:

#!/bin/sh

# need an FSTAB_FMT for printf cmds
echo FSTAB_FMT="'%-23s %-15s %-7s %-15s %-7s %-7s\n'"

# grab just the zfs_create cmds, ignore known commentary lines
# (so hopefully we grab all install cmds run) and 
# take out the first two colon terminated fields

##
## known commentary
## :>
## retval=...
## lines with words beginning capitalized
## lines with param=[value] --- probably a consistent debug msg of installer
##
grep -E -e '(zfs_create_[a-z].*: )' ${1:-/var/log/bsdinstall_log} |\
        grep -Ev -e ': (retval=[[:digit:]]|[A-Z][A-Za-z]| *:>|.*[a-z]=\[.+\])' |\
        sed 's/[^:]*:[^:]*://'

1. I booted from a 12.2 thumb drive,
2. Went through a whole ZFS mirrored install, when the install was done
3. I slapped a second thumb drive in and jumped down the chroot shell rabbit hole provided before reboot,
4. I mounted the second thumb drive ( mkdir /tmp/k; mount /dev/da1 /tmp/k) which I had already prepared with bsdinstall_log-to-customzfs.sh (above),
5. then: sh ./bsdinstall_log-to-customzfs.sh > e3-customzfs.sh
6. I edited e3-customzfs.sh so that I could have a redundant system filespace (zroot). I also wanted a local filespace pool (zdata) for home and local datasets but I set that up in e3-post-install.sh. Both of these are attached, it is easy to see the changes I made to the first because bsdinstall_log-to-customzfs.sh naturally creates one-space indented lines --- the ones I edited or added begin on column 1.
7. I unmounted the thumb drive umount /tmp/k and rebooted back into the install media,
8. At the partitioning options, I slid into the shell, mounted my e3-*.sh containing drive, and ran sh ./e3-customzfs.sh,
9. after exit-ing the install plowed through.
10. I rebooted into the new system, ran my e3-post-install.sh and case closed

Of course it wasn't this straight forward --- I practiced a lot in a virtual machine, and then had to suffer through the whole MBR/GPT/UEFI fiasco just to get a basic install working. But once the BIOS and essential install stuff got worked out the recipe above worked smoothly for two laptops in a row. I wouldn't have even known how to start if the OP hadn't taken the time to share is approach, so thanks Mr. Floyd.

 

[mefizto]

Hi mrtuple,

I just noticed your script. As you can see from the following thread: https://forums.freebsd.org/threads/almost-there-booting-from-usb-flash-drive-running-from-ssd.79457/, I am working (so far unsuccessfully) on a slightly different issue, but so far without luck. Can you please let me know how to save the bdsinstll log?

Kindest regards,

M