Solved Identifying disks for UEFI/efibootmgr/loader.env when ESP and /boot/kernel are on different disks: "efibootmgr -v" versus "geom disk list"

A

avner

I am trying to use efibootmgr to be able to multiboot different versions of freebsd and maybe even some other operating systems. But, I am a bit confused about how to correlate the efibootmgr identifier to the /dev/dax or /dev/gpt/ device names. I have three hard disks ( /dev/da0, /dev/da1, /dev/da2 ) that each have an ESP partition ( EFI partition ) at GPT partition number 13.

I am really close and maybe have a solution but wonder if anyone can explain why the lun ids I am getting from geom disk list is different ( by a value of 2 ) than the lun id's that I see in the efibootmgr -v output?

Example from my system:
Code: 
root@here:/usr/home/me # foreach x ( 0 1 2 )
foreach? echo /dev/da$x :
foreach? geom disk list da$x | grep lunid
foreach? end
/dev/da0 :
   lunid: 5000c50026029eb7
/dev/da1 :
   lunid: 5000c50026398b9b
/dev/da2 :
   lunid: 5000c50026353cef
Code: 
root@here:/usr/home/me # efibootmgr -v
BootCurrent: 0002
Timeout    : 0 seconds
BootOrder  : 0002, 0000, 0005, 0001
+Boot0002* EFI Fixed Disk Boot Device 2 PcieRoot(0x0)/Pci(0x4,0x0)/Pci(0x0,0x0)/SAS(0x[COLOR=rgb(235, 107, 86)]5000c50026353ced[/COLOR],0x0,0x1,SAS,External,Direct,0,0x0)/HD(13,GPT,5adc7cec-217c-11ed-9295-782bcb02e2bc,0x3000028,0x104000)
 Boot0000* EFI Fixed Disk Boot Device 1 PcieRoot(0x0)/Pci(0x4,0x0)/Pci(0x0,0x0)/SAS(0x[COLOR=rgb(147, 101, 184)]5000c50026029eb5[/COLOR],0x0,0x1,SAS,External,Direct,0,0x0)/HD(13,GPT,5b694d58-217c-11ed-9295-782bcb02e2bc,0x3000028,0x104000)
 Boot0005* EFI Fixed Disk Boot Device 3 PcieRoot(0x0)/Pci(0x4,0x0)/Pci(0x0,0x0)/SAS(0x[COLOR=rgb(84, 172, 210)]5000c50026398b99[/COLOR],0x0,0x1,SAS,External,Direct,0,0x0)/HD(13,GPT,5aa830bc-217c-11ed-9295-782bcb02e2bc,0x3000028,0x104000)
 Boot0001* PLDS DVD+/-RW DS-8A5SH                   PcieRoot(0x0)/Pci(0x1f,0x2)/Ata(Primary,Master,0x0)

Unreferenced Variables:
 Boot0004* FreeBSD HD(1,GPT,f5de00f2-0ca9-11ed-85fc-782bcb02e2bc,0x28,0x82000)/File(\efi\freebsd\loader.efi)
root@here:/usr/home/me #
Also, the following also suggests a correlation of the efibootmgr labels to scsi bus target numbers ( inital boot order was 0,2,5 before I modified it ).
Code: 
root@here:/usr/home/me # camcontrol devlist
<SEAGATE ST32000445SS MS02>        at scbus0 target 0 lun 0 (pass0,da0)
<SEAGATE ST32000445SS MS02>        at scbus0 target 2 lun 0 (pass1,da1)
<SEAGATE ST32000445SS MS02>        at scbus0 target 5 lun 0 (pass2,da2)
And, just in case anyone is still reading this too long post we have the following:
Code: 
root@here:/usr/home/me # efibootmgr -E
efibootmgr: Can't convert to unix path
root@here:/usr/home/me # efibootmgr -Ed
PcieRoot(0x0)
root@here:/usr/home/me # efibootmgr -Ep
efibootmgr: Can't convert to unix path
All advice and ideas welcome!
 
Alain De Vos said:
What's the output of
Code: 
geom disk list | egrep "Name|lunid"
Click to expand...
Code: 
root@here:/usr/home/me # geom disk list | egrep "Name|lunid"
1. Name: da0
   lunid: 5000c50026029eb7
1. Name: da1
   lunid: 5000c50026398b9b
1. Name: da2
   lunid: 5000c50026353cef
1. Name: da3
   lunid: 5000c500aea28d9f
1. Name: da4
   lunid: 5000c500aea273bb
1. Name: da5
   lunid: 5000c500aea2835b
1. Name: cd0

Thanks! That was helpful. I'm still curious about why for example the lunid for da0 ends in "eb7" from geom disk list vs "eb5" in efibootmgr -v but must be some trivial difference in how these values are determined between the two programs.
 
If you are looking for a solution to multi boot different versions of FreeBSD from those three disks, assuming on each disk is one OS, then create a boot variable (entry ) for each. The different versions can be booted from the UEFI boot menu.

Alternatively, rEFInd can be used. On a multi disk system rEFInd has no problems booting FreeBSD as it has on a single disk, multi FreeBSD versions system.
 
T-Daemon said:
To what purpose do you want to correlate the efibootmgr identifier to the device names?
Click to expand...
It is a little more complicated since I am using gmirror for root file systems. I have three gmirror partitions reserved for root file system ( one for each OS ). These mirrors in turn consist of 2 partitions each, from 2 of 3 disks ( for example: da0p3-da1p3, da0p4-da2p4, etc ). It was unclear to me how to tell the efi system where to look for the loader and thought I needed to provide the complex hexadecimal EFI device path but now I think I understand that efibootmgr takes care of providing that. My command will be something like the following:
Code: 
mount /dev/da0p13 /mnt   # mount the esp partition on /mnt
cp /boot/loader.efi  /mnt/EFI/freebsd/myloader.efi   # copy the loader.efi to esp
efibootmgr -b 0003 -c -l /mnt/EFI/freebsd/myloader.efi -L "here-is-my-new-OS"   # create new boot method
echo "currdev = /dev/mirror/root01" > /mnt/EFI/freebsd/loader.env   # supply environmental variable
efibootmgr -b 0003 -a   # activate boot method
efibootmgr -b 0003 -n   # set boot method as next-boot
reboot
 
The above did not work. Fortunately, after failing with next-boot method UEFI then tries my standard boot method ( the one set up during installation). One problem, I think, is that UEFI knows nothing about "/dev/mirror/root01" so "currdev = /dev/mirror/root01" must be wrong. Rather currdev seem to want an argument in the form of "diskXpY" but what does this indicate? Truly I do not understand how /EFI/freebsd/loader.efi is able to find the kernel. At some point /boot/loader.conf is parsed because that is where my gmirror module is loaded. I had no loader.env file in my original install and yet somehow loader.efi finds the kernel and finds /dev/dax/loader.conf. What is the magic that makes this happen? Does the installer include host specific information in loader.efi? loader.efi is binary so I can't read it. How does loader.efi know where to find the kernel? Does it search all the disks until it finds a /boot/kernel directory and uses the first such directory it finds? I did RTFM ( and the handbook too ) but cannot find this information there. I have not actually done a new OS installation just playing around to see if I can establish a new boot method (with ESP on a different disk from kernel) which will also boot my existing OS.
 
I did succeed at creating a new boot method using efibootmgr from an esp on a different disk from the one on which the kernel resides. This required setting rootdev=disk4p16 in /EFI/freebsd/loader.env. Apparently $rootdev identifies on which disk the kernel should be loaded from. It turns out that on my machine ( Dell r710 ) UEFI disk4p16 corresponds to /dev/da2p16 as understood by freebsd GEOM after boot on my machine. The UEFI boot process I think believes this to be the disk in enclosure 4 although on my machine this corresponds to the slot labeled, I think, 1 or 2. I was able to figure out the correct disk number to use by watching the console during the boot process. When UEFI does not find the correct disk at first it cycles through each disk and partition until it finds the correct one and then loads the kernel. By watching the console closely you can catch the last currdev tested before boot process continues and that is the correct value to set for rootdev=. Also, it may not be good to enclose the value to the right of the = in quotes.
 
avner said:
Does it search all the disks until it finds a /boot/kernel directory and uses the first such directory it finds?
Click to expand...
Yes. I think that is what UEFI does if UEFI does not find it based on the $rootdev value you have set. This could be a problem if there are different /boot/kernel directories on multiple different hard drives.
 
avner said:
The above did not work.
Click to expand...
avner said:
Rather currdev seem to want an argument in the form of "diskXpY"
Click to expand...
That is correct.

avner said:
I had no loader.env file in my original install and yet somehow loader.efi finds the kernel and finds /dev/dax/loader.conf. What is the magic that makes this happen?
Click to expand...
See loader.efi(8), boot1.efi(8) and loader.conf(5).

avner said:
I did succeed at creating a new boot method using efibootmgr from an esp on a different disk from the one on which the kernel resides. This required setting rootdev=disk4p16 in /EFI/freebsd/loader.env.
Click to expand...
I have 2 test systems set up in VirtualBox:
  • Setup one: 3 disks gmirror, gpart efi, freebsd-swap, 2 x freebsd-ufs , efibootmgr boot order pointing on the first disk of gmirror
  • Setup two: 4 disks, 3 disks gmirror, freebsd-swap, 2 x freebsd-ufs + 1 disk for ESP, efiboogmgr boot order pointing on the ESP disk

Both systems boot OK when currdev=diskXpY set in ESP loader.env (also with rootdev=), not sure why it doesn't work for you.

avner said:
Does it search all the disks until it finds a /boot/kernel directory and uses the first such directory it finds?
Click to expand...
avner said:
Yes. I think that is what UEFI does if UEFI does not find it based on the $rootdev value you have set. This could be a problem if there are different /boot/kernel directories on multiple different hard drives.
Click to expand...
UEFI reads loader.env, sets the loader environment and then loads loader.efi, from there, what root root partition is set as target is booted.


From thread Thread setting-root-filesystem-in-kernel.85611
avner said:
I am using UFS2 and had no success in setting currdev in /EFI/freebsd/loader.env but did have success setting rootdev even though kernel and root file system were on a different HDD than the ESP. I have not tried yet to set either of these variables in /boot/loader.conf. Maybe they would work there.
Click to expand...
If /boot/loader.conf is located on a gmirror root partition, then it won't work.

Also when rootdev would be set on a non-gmirror setup it's ill advised when different FreeBSD branches are installed in a multi OS setup. As I recall you want to run 13.1-RELEASE and -STABLE. If rootdev is set to boot STABLE in the 13.1-RELEASE loader.conf, the loader would boot the 13.1-RELEASE kernel but mount the -STABLE userland, and vice versa.

This won't happen with currdev: loader(8)
Code: 
     currdev   Selects the default device to loader the kernel from.
 
I don’t think it will be a problem to boot multiple versions of FreeBSD once I can pin down the correct x and y for the statement “rootdev=diskxpy” in loader.env. So far I have the values for 3 disks figured out by trial and error. That’s probably enough but I will probably figure out the other 3 disks to satisfy my curiosity. It is interesting to me that the x variable (disk number), at least in my case, has no relationship to geom lunid, mpsutil enclosure value, or physical slot number and certainly not to /dev/dax. Seems like EFI has it’s own special random number algorithm to associate the a rootdev value to a UEFI device path.
 
T-Daemon said:
I have 2 test systems set up in VirtualBox:
  • Setup one: 3 disks gmirror, gpart efi, freebsd-swap, 2 x freebsd-ufs , efibootmgr boot order pointing on the first disk of gmirror
  • Setup two: 4 disks, 3 disks gmirror, freebsd-swap, 2 x freebsd-ufs + 1 disk for ESP, efiboogmgr boot order pointing on the ESP disk

Both systems boot OK when currdev=diskXpY set in ESP loader.env (also with rootdev=), not sure why it doesn't work for you.
Click to expand...
I apologize for misleading you. I had forgotten to include the trailing ":" when setting the value of currdev in loader.env. This makes all the difference. Apparently rootdev is not as fussy about including trailing ":". Now setting either currdev or rootdev to the disk number that correlates to my root file system gives the same favorable result.
Here is the annoying part for me: gpart show says my root fs is on /dev/da1p16, mpsutils states this to be in slot 4, scsi address is 0:2:0, and loader.env wants currdev=disk5p16: and looking at the physical case the disk is in the slot externally labeled #1. I am hoping that if/when I swap out one of the HDDs that this does not cause those currdev settings to change.
 
One final note to anyone that may find this thread. The physical 3.5" HDD slots on my computer are labeled 0-5. The corresponding loader.env diskXpN values for X were: 3,4,5,2,1,0. I had to figure this out experimentally by creating esp partitions on each disc and root partitions on each disk, copying the contents of the existing root partition to the 4 other root partitions. Creating an efibootmgr entries for each esp for example one for /dev/da3. Adjusting the /etc/fstab on the /dev/da3 root file system to mount the /dev/da3 root partition on /. Then test various values for currdev=diskXpN: until the system consistently booted and loaded the correct root file system ( the root file system on /dev/da3 ). I did this for each disk until I was convinced that I had established a concordance between the loader.env currdev values and the OS device descriptions ( eg /dev/da3 ). This took quite a while. Since loader.efi starts searching additional partitions and other disks for a kernel if it does not find it at the set currdev= site it is possible to set the wrong value for currdev= and by chance wind up with the correct root file system mounted. Good luck!
 
T-Daemon said:
The different versions can be booted from the UEFI boot menu.
Click to expand...
If you have boot loader of respective versions/other OS created in ESP during installation, then you can use any one of these by selecting the disk from bios menu list. ( using Fn key of your Bios). No need of other loader in your booting process. In UEFI boot menu you can make default booting disk.
 
T-Daemon said:
The different versions can be booted from the UEFI boot menu.
Click to expand...
gnath said:
If you have boot loader of respective versions/other OS created in ESP during installation, then you can use any one of these by selecting the disk from bios menu list. ( using Fn key of your Bios).
Click to expand...
That's what I meant. I should have said "startup device menu of the UEFI". I didn't use the BIOS term, being the notation for the legacy boot firmware.
 
You must log in or register to reply here.
Back
Top