Workaround hardware limitation on booting from NVMe

[taiwan740]

TL;DR: how do I install the FreeBSD bootloader on a different drive?

Hi. I've got an old desktop that I'm committed to keep current, it presents some interesting challenges which require some creative thought.

After using SATA SSDs for a while which, due to the age of the motherboard, are being throttled themselves to SATA II, I've since learnt about using PCIe M.2 NVMe boards. OK, yes the NVMe card is still going to be throttled by the PCIe version being 2.0, but allegedly, data transfer speed should increase four fold. I've got bugger-all else planned this weekend and the expansion cards weren't that expensive so I thought I'd give it a go.

The main hurdle is that the motherboard isn't going to recognise the PCIe NVMe drive as a bootable device.

My thinking to work around this is to 'dd' my current installation from the current SATA SSD over to the NVMe card, and then use a small cheap blank SSD in its place, so the storage will be limited to the NVMe card with the FreeBSD 13.2 installation, and the small empty SATA SSD. I can then install some kind of bootloader on the SATA SSD that the computer will boot (in legacy BIOS mode) and then some bootloader can boot the FreeBSD installation on the NVMe card.

I'm pretty sure I could do this by installing grub on the SATA SSD that would detect and then chainload to the FreeBSD bootloader on the NVMe, but ideally I don't want that extra step, and just want the FreeBSD loader to fire up and load the system from the NVMe as if it were all on one drive. Anyone got any thoughts on how the bootloader could be installed and configured on a separate drive?

Alternatively, I suppose I could create a new install on the SATA SSD and just have select partitions with mountpoints like /etc/, /var/ and /home on the NVMe which would likely be just as effective in userland speed enhancements.

By the way, I'm not using ZFS, just UFS. Not got my head around zfspool just yet and that's a different project.

Apologies for all the verbose, it's a feature.

 

[Phishfry]

small empty SATA SSD. I can then install some kind of bootloader on the SATA SSD that the computer will boot (in legacy BIOS mode)

It all sounded good until you said legacy.
I would recommend EFI partition then rest on nvme. add currdev=disk1p2 to loader.conf

Otherwise if not possible you need:

have select partitions with mountpoints like /etc/, /var/ and /home on the NVMe

and /boot/ on sata which is not pretty. It can be done from bsdinstall for UFS though. Just takes some work.

 

[taiwan740]

It all sounded good until you said legacy.

Haha

Thanks for your thoughts.

Yeah, it's an old Mac, so it only supports EFI in 32-bit. It falls back to legacy BIOS boot after a 30 seconds timeout, so computer boot speed isn't really the concern here as it takes forever and I make a hot drink while it's booting anyway, but I'd like the applications and userland to be a bit snappier when it is booted, so maybe just some highly convuluted fstab is the way to go.

Perhaps I'll wait until FreeBSD 14 as it sounds like a clean installation would be easier to set up.

 

[taiwan740]

Phishfry am I correct in thinking that FreeBSD has actually detected the new drive? :O

From dmesg

nvme0: Allocated 32MB host memory buffer
nvd0: <CT500P3PSSD8> NVMe namespace
nvd0: 476940MB (976773168 512 byte sectors)

 

[richardtoohey2]

Don't think this is what you mean, but just in case ... you can also look at a USB-boot that then boots the OS from NVMe:

Non-trivial (or rather trivial) boot question

Hello I have a server, a bit aged but still able and fit for most of my needs. Trying to speed it up and free two disk hotswap places, I've perusaded server's owner to purchase two NVME PCI cards for boot and ZFS cache. After installing them, I've (suddenly!) find out that the motherboard...

forums.freebsd.org

 

[taiwan740]

Don't think this is what you mean, but just in case ... you can also look at a USB-boot that then boots the OS from NVMe:

Non-trivial (or rather trivial) boot question

Hello I have a server, a bit aged but still able and fit for most of my needs. Trying to speed it up and free two disk hotswap places, I've perusaded server's owner to purchase two NVME PCI cards for boot and ZFS cache. After installing them, I've (suddenly!) find out that the motherboard...

forums.freebsd.org

Cheers, yeah I have further hurdles as this machine doesn't support UEFI nor does it allow booting from USB. I could have a scan through and see if I can apply any of the concepts to a SATA drive though.

 

[sko]

For this purpose and/or some smaller appliances I managed to get a few 32GB SATA SSDs from an "new/old stock" sale on ebay - 10 drives for 4EUR each including shipping...
One can find such deals relatively often and ~5-6EUR/piece is pretty common for 32GB drives. Makes no sense IMHO to use slow and unreliable USB thumb drives if you can get "real" disks for the same or less.

If they are only intended for booting from NVMe I often use the remaining space to keep either a current installer image or even a vanilla base install around for troubleshooting. Saved my life once when a remote system wouldn't boot off the NVMe mirror pool after a hardware failure.

 

[covacat]

there is a project in progress that will allow to use a linux kernel to boot freebsd
linux can launch a 64 kernel from 32bit efi so "in theory" you can get there

The FreeBSD Project

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org

so in theory mac->grubefi32->linux 64 kernel->loader.kboot->freebsd kernel

 

[taiwan740]

there is a project in progress that will allow to use a linux kernel to boot freebsd
linux can launch a 64 kernel from 32bit efi so "in theory" you can get there

The FreeBSD Project

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org

so in theory mac->grubefi32->linux 64 kernel->loader.kboot->freebsd kernel

Interesting. I did some experimentation previously, and I couldn't get Linux to boot at all on this machine. I tried Debian and Fedora (the rest are nonsense anyway) and lost the will to live with it. FreeBSD boots and installs using a isomacprogs hack on the ISO file, and then will allow installation with biosboot. It's the only current operating system I've managed to get working comfortably on it.

For this purpose and/or some smaller appliances I managed to get a few 32GB SATA SSDs from an "new/old stock" sale on ebay - 10 drives for 4EUR each including shipping...
One can find such deals relatively often and ~5-6EUR/piece is pretty common for 32GB drives. Makes no sense IMHO to use slow and unreliable USB thumb drives if you can get "real" disks for the same or less.

If they are only intended for booting from NVMe I often use the remaining space to keep either a current installer image or even a vanilla base install around for troubleshooting. Saved my life once when a remote system wouldn't boot off the NVMe mirror pool after a hardware failure.

Yeah I've got an old 128GB SATA SSD kicking about, but to be honest for this purpose, even a 160GB old HDD might do, although it would bug me to think that it has to sit there and spin endlessly doing nothing throughout operation, but then I suppose if energy efficiency was my concern, it's running two Intel CPUs from 2006, so... well...