Solved Modifying ada0 ZFS /boot/loader.conf booting from rescue USB

I have a playground FreeBSD system, single disk, 12.1R/amd64 installed on ZFS.
Two GPT partitions on ada0. An 512k freebsd-boot, followed by a zfs partition covering the rest of the disk.
Absolutely no valuable data on disk. It really is just my playground.
Here is what mount shows on the running system:
Code:
kzp/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
kzp/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp on /kzp (zfs, local, noatime, nfsv4acls)
kzp/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
kzp/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/mail on /var/mail (zfs, local, nfsv4acls)
kzp/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
kzp/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
linprocfs on /compat/linux/proc (linprocfs, local)
linsysfs on /compat/linux/sys (linsysfs, local)
tmpfs on /compat/linux/dev/shm (tmpfs, local)

I wish to modify the /boot/loader.conf file of this system, while the actual system is NOT running. So I booted the computer hosting this system off of the standard FreeBSD 12.1R/amd64 installer USB stick and entered its Live system.
Then attempted to mount my ZFS from the HDD.

Code:
root@:/ # cd /var/tmp

root@:/var/tmp # mkdir SATAHD

root@:/var/tmp # mount

/dev/ufs/FreeBSD_Install on / (ufs, local, noatime, read-only)
devfs on /dev (devfs, local, multilabel)
tmpfs on /var (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
root@:/var/tmp # zpool status

no pools available
root@:/var/tmp # zpool import

   pool: kzp
     id: 3953766718312634956
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

    kzp         ONLINE
      ada0p4    ONLINE

root@:/var/tmp # zpool import -f -R /var/tmp/SATAHD kzp

root@:/var/tmp # zfs list

NAME               USED  AVAIL  REFER  MOUNTPOINT
kzp               7.17G   134G    88K  /var/tmp/SATAHD/kzp
kzp/ROOT          5.07G   134G    88K  none
kzp/ROOT/default  5.07G   134G  5.07G  /var/tmp/SATAHD
kzp/tmp            112K   134G   112K  /var/tmp/SATAHD/tmp
kzp/usr           2.09G   134G    88K  /var/tmp/SATAHD/usr
kzp/usr/home      43.7M   134G  43.7M  /var/tmp/SATAHD/usr/home
kzp/usr/ports      707M   134G   707M  /var/tmp/SATAHD/usr/ports
kzp/usr/src       1.36G   134G  1.36G  /var/tmp/SATAHD/usr/src
kzp/var            788K   134G    88K  /var/tmp/SATAHD/var
kzp/var/audit       88K   134G    88K  /var/tmp/SATAHD/var/audit
kzp/var/crash       88K   134G    88K  /var/tmp/SATAHD/var/crash
kzp/var/log        324K   134G   324K  /var/tmp/SATAHD/var/log
kzp/var/mail       112K   134G   112K  /var/tmp/SATAHD/var/mail
kzp/var/tmp         88K   134G    88K  /var/tmp/SATAHD/var/tmp

root@:/var/tmp # mount

/dev/ufs/FreeBSD_Install on / (ufs, local, noatime, read-only)
devfs on /dev (devfs, local, multilabel)
tmpfs on /var (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
kzp/var/crash on /var/tmp/SATAHD/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/log on /var/tmp/SATAHD/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/tmp on /var/tmp/SATAHD/var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp on /var/tmp/SATAHD/kzp (zfs, local, noatime, nfsv4acls)
kzp/var/audit on /var/tmp/SATAHD/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/usr/home on /var/tmp/SATAHD/usr/home (zfs, local, noatime, nfsv4acls)
kzp/tmp on /var/tmp/SATAHD/tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp/var/mail on /var/tmp/SATAHD/var/mail (zfs, local, nfsv4acls)
kzp/usr/src on /var/tmp/SATAHD/usr/src (zfs, local, noatime, nfsv4acls)
kzp/usr/ports on /var/tmp/SATAHD/usr/ports (zfs, local, noatime, nosuid, nfsv4acls)

root@:/var/tmp # cd SATAHD

root@:/var/tmp/SATAHD # pwd

/var/tmp/SATAHD
root@:/var/tmp/SATAHD # ls -la

total 25
drwxr-xr-x  6 root  wheel  256 Sep 20 22:35 .
drwxrwxrwt  5 root  wheel  320 Sep 20 22:33 ..
drwxr-xr-x  2 root  wheel    2 Jun  8 20:19 kzp
drwxrwxrwt  6 root  wheel    6 Sep 19 21:47 tmp
drwxr-xr-x  5 root  wheel  192 Sep 20 22:35 usr
drwxr-xr-x  7 root  wheel  320 Sep 20 22:35 var
root@:/var/tmp/SATAHD # cd kzp

root@:/var/tmp/SATAHD/kzp # ls -la

total 1
drwxr-xr-x  2 root  wheel    2 Jun  8 20:19 .
drwxr-xr-x  6 root  wheel  256 Sep 20 22:35 ..
root@:/var/tmp/SATAHD/kzp #

I expected to find my loader.conf file as /var/tmp/SATAHD/boot/loader.conf, but there is no boot directory in /var/tmp/SATAHD.
So, what do I do wrong? What did I miss? Where is the boot directory? How do I get to the /boot/loader.conf file of the harddisk's ZFS?
 
It should be on the freebsd-boot partition. Mount that one.
This seems to contradict my existing knowledge. I was under the impression that the freebsd-boot partition cannot be accessed, it has no filesystem, the only thing it contains is the bootloader program code written directly into those storage blocks occupied by the slice. Therefore, my loader.conf cannot actually reside on that partition. Also, the contents of my /boot/ directory could not fit into the 512k space this partition covers.
But, I am open to learn something new. So, how would you mount the freebsd-boot partition? As what FileSystem? What value do you give to mount's -t argument?
 
I thought that there is raw bootcode on the freebsd-boot partition (first two stages of the boot process). But then there follows a file system which contains the loader() (third stage).

I have an efi partition there and I can mount it as msdosfs. But I am not sure about freebsd-boot. ;D
 
I expected to find my loader.conf file as /var/tmp/SATAHD/boot/loader.conf, but there is no boot directory in /var/tmp/SATAHD.
So, what do I do wrong? What did I miss? Where is the boot directory? How do I get to the /boot/loader.conf file of the harddisk's ZFS?
After root@:/var/tmp # zpool import -f -R /var/tmp/SATAHD kzp run zfs mount kzp/ROOT/default.
 
What does gpart show tell? Could /boot reside in another partition?
Yes it could. I investigated that possibility myself. But in my current circumstances, I believe it does not reside on another partition.
Code:
root@demobsd:~ # gpart show
=>       40  312581728  ada0  GPT  (149G)
         40     409600     1  efi  (200M)
     409640       1024     2  freebsd-boot  (512K)
     410664        984        - free -  (492K)
     411648    4194304     3  freebsd-swap  (2.0G)
    4605952  307974144     4  freebsd-zfs  (147G)
  312580096       1672        - free -  (836K)
This is exactly the layout what bsdinstall picked to make on my otherwise empty GPT disk when I chose to setup on zfs.
 
After root@:/var/tmp # zpool import -f -R /var/tmp/SATAHD kzp run zfs mount kzp/ROOT/default.
That was it. Thank you!
Code:
Script started on Wed Sep 23 12:42:43 2020

root@:/var/tmp # pwd
/var/tmp

root@:/var/tmp # mount
/dev/ufs/FreeBSD_Install on / (ufs, local, noatime, read-only)
devfs on /dev (devfs, local, multilabel)
tmpfs on /var (tmpfs, local)
tmpfs on /tmp (tmpfs, local)

root@:/var/tmp # zpool import
   pool: kzp
     id: 3953766718312634956
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
    the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

    kzp         ONLINE
      ada0p4    ONLINE

root@:/var/tmp # mkdir SATAHD

root@:/var/tmp # ls -la
total 4
drwxrwxrwt   4 root  wheel   192 Sep 23 12:43 .
drwxr-xr-x  24 root  wheel  1408 Sep 23 12:41 ..
drwxr-xr-x   2 root  wheel     0 Sep 23 12:43 SATAHD
drwxrwxrwt   2 root  wheel     0 Sep 23 12:41 vi.recover

root@:/var/tmp # zpool import -f -R /var/tmp/SATAHD kzp

root@:/var/tmp # zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
kzp               7.17G   134G    88K  /var/tmp/SATAHD/kzp
kzp/ROOT          5.07G   134G    88K  none
kzp/ROOT/default  5.07G   134G  5.07G  /var/tmp/SATAHD
kzp/tmp            112K   134G   112K  /var/tmp/SATAHD/tmp
kzp/usr           2.09G   134G    88K  /var/tmp/SATAHD/usr
kzp/usr/home      43.7M   134G  43.7M  /var/tmp/SATAHD/usr/home
kzp/usr/ports      707M   134G   707M  /var/tmp/SATAHD/usr/ports
kzp/usr/src       1.36G   134G  1.36G  /var/tmp/SATAHD/usr/src
kzp/var            792K   134G    88K  /var/tmp/SATAHD/var
kzp/var/audit       88K   134G    88K  /var/tmp/SATAHD/var/audit
kzp/var/crash       88K   134G    88K  /var/tmp/SATAHD/var/crash
kzp/var/log        328K   134G   328K  /var/tmp/SATAHD/var/log
kzp/var/mail       112K   134G   112K  /var/tmp/SATAHD/var/mail
kzp/var/tmp         88K   134G    88K  /var/tmp/SATAHD/var/tmp

root@:/var/tmp # cd SATAHD

root@:/var/tmp/SATAHD # ls -la
total 25
drwxr-xr-x  6 root  wheel  256 Sep 23 12:43 .
drwxrwxrwt  4 root  wheel  192 Sep 23 12:43 ..
drwxr-xr-x  2 root  wheel    2 Jun  8 20:19 kzp
drwxrwxrwt  6 root  wheel    6 Sep 23 10:40 tmp
drwxr-xr-x  5 root  wheel  192 Sep 23 12:43 usr
drwxr-xr-x  7 root  wheel  320 Sep 23 12:43 var

root@:/var/tmp # mount
/dev/ufs/FreeBSD_Install on / (ufs, local, noatime, read-only)
devfs on /dev (devfs, local, multilabel)
tmpfs on /var (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
kzp/var/log on /var/tmp/SATAHD/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/crash on /var/tmp/SATAHD/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp on /var/tmp/SATAHD/kzp (zfs, local, noatime, nfsv4acls)
kzp/var/audit on /var/tmp/SATAHD/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/tmp on /var/tmp/SATAHD/var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp/usr/home on /var/tmp/SATAHD/usr/home (zfs, local, noatime, nfsv4acls)
kzp/tmp on /var/tmp/SATAHD/tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp/var/mail on /var/tmp/SATAHD/var/mail (zfs, local, nfsv4acls)
kzp/usr/src on /var/tmp/SATAHD/usr/src (zfs, local, noatime, nfsv4acls)
kzp/usr/ports on /var/tmp/SATAHD/usr/ports (zfs, local, noatime, nosuid, nfsv4acls)

root@:/var/tmp/SATAHD # zfs mount kzp/ROOT/default

root@:/var/tmp/SATAHD # mount
/dev/ufs/FreeBSD_Install on / (ufs, local, noatime, read-only)
devfs on /dev (devfs, local, multilabel)
tmpfs on /var (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
kzp/var/log on /var/tmp/SATAHD/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/crash on /var/tmp/SATAHD/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp on /var/tmp/SATAHD/kzp (zfs, local, noatime, nfsv4acls)
kzp/var/audit on /var/tmp/SATAHD/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
kzp/var/tmp on /var/tmp/SATAHD/var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp/usr/home on /var/tmp/SATAHD/usr/home (zfs, local, noatime, nfsv4acls)
kzp/tmp on /var/tmp/SATAHD/tmp (zfs, local, noatime, nosuid, nfsv4acls)
kzp/var/mail on /var/tmp/SATAHD/var/mail (zfs, local, nfsv4acls)
kzp/usr/src on /var/tmp/SATAHD/usr/src (zfs, local, noatime, nfsv4acls)
kzp/usr/ports on /var/tmp/SATAHD/usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
kzp/ROOT/default on /var/tmp/SATAHD (zfs, local, noatime, nfsv4acls)

root@:/var/tmp/SATAHD # ls -la
total 109
drwxr-xr-x  20 root  wheel    26 Sep 23 10:40 .
drwxrwxrwt   4 root  wheel   192 Sep 23 12:43 ..
-rw-r--r--   2 root  wheel   951 Nov  1  2019 .cshrc
-rw-r--r--   2 root  wheel   470 Nov  1  2019 .profile
-r--r--r--   1 root  wheel  6181 Nov  1  2019 COPYRIGHT
drwxr-xr-x   2 root  wheel    46 Aug  9 17:38 bin
drwxr-xr-x  10 root  wheel    63 Sep 23 10:40 boot
drwxr-xr-x   3 root  wheel     3 Aug  9 20:14 compat
drwxr-xr-x   2 root  wheel     2 Jun  8 20:19 dev
-rw-------   1 root  wheel  4096 Sep 23 10:40 entropy
drwxr-xr-x  25 root  wheel   109 Aug 14 18:03 etc
lrwxr-xr-x   1 root  wheel     8 Aug  5 11:25 home -> usr/home
drwxr-xr-x   2 root  wheel     2 Jun  8 20:19 kzp
drwxr-xr-x   5 root  wheel    62 Jun 30 13:30 lib
drwxr-xr-x   3 root  wheel     5 Jun  8 20:20 libexec
drwxr-xr-x   2 root  wheel     2 Nov  1  2019 media
drwxr-xr-x   2 root  wheel     2 Nov  1  2019 mnt
drwxr-xr-x   2 root  wheel     2 Nov  1  2019 net
dr-xr-xr-x   2 root  wheel     2 Nov  1  2019 proc
drwxr-xr-x   2 root  wheel   150 Jun 30 13:30 rescue
drwxr-xr-x   6 root  wheel    21 Sep 23 10:40 root
drwxr-xr-x   2 root  wheel   137 Jun 30 13:30 sbin
lrwxr-xr-x   1 root  wheel    11 Nov  1  2019 sys -> usr/src/sys
drwxr-xr-x   2 root  wheel     2 Jun  8 20:19 tmp
drwxr-xr-x  16 root  wheel    16 Jun  8 20:22 usr
drwxr-xr-x  25 root  wheel    25 Sep 23 12:39 var

root@:/var/tmp/SATAHD # cd boot

root@:/var/tmp/SATAHD/boot # ls -l
total 4279
-r--r--r--  1 root  wheel    3554 Nov  1  2019 beastie.4th
-r--r--r--  1 root  wheel    8192 Nov  1  2019 boot
-r--r--r--  1 root  wheel     512 Nov  1  2019 boot0
-r--r--r--  1 root  wheel     512 Nov  1  2019 boot0sio
-r--r--r--  1 root  wheel     512 Nov  1  2019 boot1
-r-xr-xr-x  1 root  wheel   86016 Nov  1  2019 boot1.efi
-r--r--r--  1 root  wheel  819200 Nov  1  2019 boot1.efifat
-r--r--r--  1 root  wheel    7680 Nov  1  2019 boot2
-r--r--r--  1 root  wheel    2126 Nov  1  2019 brand-fbsd.4th
-r--r--r--  1 root  wheel    2806 Nov  1  2019 brand.4th
-r--r--r--  1 root  wheel    1188 Nov  1  2019 cdboot
-r--r--r--  1 root  wheel    6277 Nov  1  2019 check-password.4th
-r--r--r--  1 root  wheel    1867 Nov  1  2019 color.4th
drwxr-xr-x  2 root  wheel       3 Nov  1  2019 defaults
-r--r--r--  1 root  wheel    4053 Nov  1  2019 delay.4th
-r--r--r--  1 root  wheel     829 Nov  1  2019 device.hints
drwxr-xr-x  5 root  wheel       5 Aug  9 17:38 dtb
-r--r--r--  1 root  wheel    1591 Nov  1  2019 efi.4th
-rw-------  1 root  wheel    4096 Sep 23 10:40 entropy
drwxr-xr-x  2 root  wheel       2 Nov  1  2019 firmware
-r--r--r--  1 root  wheel    3695 Nov  1  2019 frames.4th
-r--r--r--  1 root  wheel   59110 Nov  1  2019 gptboot
-r--r--r--  1 root  wheel  105054 Nov  1  2019 gptzfsboot
-r--r--r--  1 root  wheel   14931 Nov  1  2019 isoboot
drwxr-xr-x  2 root  wheel     890 Aug  9 17:38 kernel
drwxr-xr-x  2 root  wheel     888 Aug  9 17:38 kernel.old
-r-xr-xr-x  3 root  wheel  393216 Nov  1  2019 loader
-r--r--r--  1 root  wheel    7425 Nov  1  2019 loader.4th
-rw-r--r--  1 root  wheel     237 Jun 30 12:55 loader.conf
-r-xr-xr-x  2 root  wheel  460800 Jun 30 13:30 loader.efi
-r--r--r--  1 root  wheel     468 Nov  1  2019 loader.rc
-r-xr-xr-x  1 root  wheel  344064 Nov  1  2019 loader_4th
-r-xr-xr-x  1 root  wheel  395776 Jun 30 13:30 loader_4th.efi
-r-xr-xr-x  3 root  wheel  393216 Nov  1  2019 loader_lua
-r-xr-xr-x  2 root  wheel  460800 Jun 30 13:30 loader_lua.efi
-r-xr-xr-x  1 root  wheel  294912 Nov  1  2019 loader_simp
-r-xr-xr-x  1 root  wheel  335360 Jun 30 13:30 loader_simp.efi
-r--r--r--  1 root  wheel    3110 Nov  1  2019 logo-beastie.4th
-r--r--r--  1 root  wheel    2636 Nov  1  2019 logo-beastiebw.4th
-r--r--r--  1 root  wheel    2214 Nov  1  2019 logo-fbsdbw.4th
-r--r--r--  1 root  wheel    2631 Nov  1  2019 logo-orb.4th
-r--r--r--  1 root  wheel    2354 Nov  1  2019 logo-orbbw.4th
drwxr-xr-x  2 root  wheel      17 Nov  1  2019 lua
-r--r--r--  1 root  wheel     512 Nov  1  2019 mbr
-r--r--r--  1 root  wheel    9260 Nov  1  2019 menu-commands.4th
-r--r--r--  1 root  wheel   36016 Nov  1  2019 menu.4th
-r--r--r--  1 root  wheel    6328 Nov  1  2019 menu.rc
-r--r--r--  1 root  wheel   18597 Nov  1  2019 menusets.4th
drwxr-xr-x  2 root  wheel     608 Aug  9 17:38 modules
-r--r--r--  1 root  wheel     512 Nov  1  2019 pmbr
-r--r--r--  1 root  wheel  395264 Nov  1  2019 pxeboot
-r--r--r--  1 root  wheel    2675 Nov  1  2019 screen.4th
-r--r--r--  1 root  wheel    2613 Nov  1  2019 shortcuts.4th
-r--r--r--  1 root  wheel   36282 Nov  1  2019 support.4th
-r--r--r--  2 root  wheel  422408 Nov  1  2019 userboot.so
-r--r--r--  1 root  wheel  350528 Nov  1  2019 userboot_4th.so
-r--r--r--  2 root  wheel  422408 Nov  1  2019 userboot_lua.so
-r--r--r--  1 root  wheel    3065 Nov  1  2019 version.4th
drwxr-xr-x  2 root  wheel       3 Sep 23 10:39 zfs
-r--r--r--  1 root  wheel  262656 Nov  1  2019 zfsboot
-r-xr-xr-x  3 root  wheel  393216 Nov  1  2019 zfsloader

root@:/var/tmp/SATAHD/boot # exit

exit

Script done on Wed Sep 23 12:46:09 2020

This is exactly what I was looking for. Now I can access my loader.conf.
Why this was not mounted automatically together with the rest when I imported the pool still puzzles me. But I can live with it.
Interestingly enough, I checked the list of mounted filesystems before and I must have seen that kzp/ROOT/default is not in the list, yet it did not sink into my brain that it is what I am missing and all I need to try is to mount that manually.
This would have troubled me for some more time and I am not sure I would have eventually spoted the solution by myself.
 
Yes, this issue was very similar to mine and your solution there was exactly what I needed now.
 
Back
Top