Non-trivial (or rather trivial) boot question

tarkhil

tarkhil

Hello

I have a server, a bit aged but still able and fit for most of my needs. Trying to speed it up and free two disk hotswap places, I've perusaded server's owner to purchase two NVME PCI cards for boot and ZFS cache.
After installing them, I've (suddenly!) find out that the motherboard cannot boot from PCI NVME, and I'm advised to purchase a new motherboard. And two new CPUs. And new memory. Not right now, sorry.
So I think of the USB stick with loader telling boot2 to load from NVME.
As far as I understand, that should be possible, but I've found nothing on the subject.
Maybe someone has already accomplished something similar?
 
if nvme is not visible in bios you need full kernel on the usb stick
and mount root from nvme
that works for sure and then mount the stick on /boot
 
How often do you boot this server? TrueNAS, which is based on FreeBSD boots quite easily from a USB stick, and usually does not need to reboot for a month or two.
 
just download a minimal install image, write it to a memory stick, mount it and edit its loader.conf
vfs.root.mountfrom="zfs:ntank/boot" or
vfs.root.mountfrom="ufs:/dev/something0p1"
then you can add /dev/da0p2 /boot to your fstab
 
Hi covacat,

thank you for the reply, unfortunately I my understanding of the boot process is insufficient, so it does not help.

Greetings all,

based on the forgoing, I started reading on the boot process in the Handbook. It is rather difficult to connect the description with the different files setting boot variables, but as best understood from the Handbook quote: (emphasis supplied):
Stage two, boot2, is slightly more sophisticated, and understands the FreeBSD file system enough to find file
Click to expand...
that this is performed by gptzfsboot(8), cf.
gptzfsboot tries to find all ZFS pools that are composed of BIOS-visible hard disks or partitions on them
Click to expand...
and
The first pool seen during probing is used as a default boot pool.

The filesystem specified by the bootfs property of the pool is used as a
default boot filesystem. If the bootfs property is not set, then the
root filesystem of the pool is used as the default. loader(8) is loaded
from the boot filesystem.
Click to expand...

The gptzfsboot(8) further states:
gptzfsboot looks for ZFS device labels
on all visible disks and in discovered supported partitions for all sup-
ported partition scheme types. The search starts with the disk from
which gptzfsboot itself was loaded. Other disks are probed in BIOS de-
fined order.
Click to expand...
But, it does not appear to work. Although it appears that the NVMe drive is visible and has a ZFS filesytem on it, gptzfsboot(8) cannot find it. And thee are no options that could be set to help the process mentioned by the gptzfsboot(8).

In a different thread https://forums.freebsd.org/threads/running-os-off-usb-thumb-drive-or.79080/, Mjölnir asserted:
IIRC you can boot from one media, and have rootdev="disk2s1a" in loader.conf(5) to switch to another device.
Click to expand...
However, as I understand, the lader.conf(5) is on the pool/filesystem that the gptzfsboot(8), cannot find.

Any help would be appreciated.

Kindest regards,

M
 
I've done something similar recently and was going to ask if I'd missed a better way.

So my main issue was the Dell R220 doesn't allow booting from NVMe.

I'd used USB UEFI boot to install FreeBSD 13.0 to a single NVME, UFS and that went fine.

But when I unplugged the USB ... no way to get the BIOS to boot from the NVME.

I need to dig out my notes as to what exactly I did for my situation, but this page was the main help:

Load NVMe Driver in UEFI Shell

Months ago I bought an HP Z620 as a Hyper-V server at home. One day, I fond that my 500GB 860EVO SSD was almost full. Considering the factors of price, performance, and capacity, I decided to purchase a 2TB Intel DC P4500 SSD. However, after installing the SSD, I found that it was not recognized...
ntzyz.io ntzyz.io

So I've ended up with a USB stick in the internal USB port and this USB stick is the boot device holding a UEFI shell and a startup.nsh script.

It boots from the USB stick, loads the UEFI shell that uses the startup.nsh to boot FreeBSD from the NVME.

In case the linked page disappears from the internet, I've copied the four lines of the script:
Code: 
load -nc fs0:\NvmExpressDxe-64.efi
connect -r
map -u
fs1:\EFI\BOOT\BOOTX64.EFI
The UEFI shell doesn't have an NVME driver, so line 1 does that. The connect and map commands connect & map to the NVMe device, making it available as fs1 (fs0 being the boot USB device). The fourth line makes it run the EFI code on the NVMe device, and because I'd previously installed FreeBSD on the NVMe, it carried on booting my previous installation.

The reason I was going to post on the forums - what I've done does work, but there are a few quirks (again have to check my notes) - main one that I can recall is that when I use top it shows a battery line as though it thinks I've got it running on a laptop - but it's a Dell R220 server.

My other questions were about how safe it was to boot from a downloaded UEFI shell & NVME driver.

Finally you talk about two NVMe devices - that won't work in my case with the Dell because the BIOS doesn't support bifurcation - so you might have further issues if you are trying to use a card with two NVMe devices on.

I'll dig out my notes so I can explain exactly what I had to do (and where I got the UEFI shell and NVME driver from), but think this approach might work for you as well, depending on the hardware and what exactly you are trying to do.

I've not used the machine yet, just got to the point of it UEFI booting from USB (as an initial stage) and then booting off NVMe - so there might be flaws in the cunning plan that I'm not aware of yet!
 
Hi richardtoohey2,

although your post appears to be directed at the OP, as I have not mentioned two NVMe devices, thank you for posting it. Unfortunately, the motherboard that I have does not support UEFI, but perhaps it may be adopted.

In a meanwhile I found a post that separates the /boot/ and the rest of the system https://freebsdfoundation.blogspot.com/2015/04/from-trenches-tips-tricks-edition.html. Looking at hier(7), perhaps this is the way forward, since the entire /boot/ with all the entities supporting the boot process will be on the boot drive, thus solving the problem that the gptzfsboot(8) has no options for affecting the boot process.

Though I remember reading somewhere that it is not wise to place the /boot/ on its own filesystem, the author is a prominent FreeBSD member, so I will experiment with the approach.

Kindest regards,

M
 
covacat said:
just download a minimal install image, write it to a memory stick, mount it and edit its loader.conf
vfs.root.mountfrom="zfs:ntank/boot" or
vfs.root.mountfrom="ufs:/dev/something0p1"
then you can add /dev/da0p2 /boot to your fstab
Click to expand...
So the idea is to mount /boot from the stick and everyone else from zfs on nvme. And, as far as I recall UNIX mounting process, I'll do just fine while my kernel/modules versions are the same.
And I'll look into mfsbsd scripts because I do like the idea of totally r/o boot stick. Flash without writing access should last forever...
 
beware of flash bit rot (after several years)
happened on a box i know
i think if you update it at every freebsd update is enough to keep the data ok
forums.freebsd.org

USB flash data rot

I have an old bootable stick. After 8 years data has changed slightly, enough to panic at every boot (privileged instruction fault in kernel mode). Changed mobo/memory/cpu and whatnot until i figured it out. no fsck errors, no surface scan errors the stick is only used for booting, no write/no...
forums.freebsd.org forums.freebsd.org
 
You must log in or register to reply here.
Back
Top