Other NVMe boot drives with legacy BIOSes without UEFI using loader(8)

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

What I'm considering doing is using a generic two port M.2 drive (SATA and NVMe) as my root and swap device. Two of my systems on the network I'm constructing do not suport UEFI (HP Microserver G7 N54L). The M.2 SATA SSD is going to be connected to the internal systemboard SATA port.

I'm looking at generic dual M.2 cards with one being a M.2 NVMe and the other SATA M.2 with the root file systems in a software RAID1. Obviously you want to run root and swap on a fast device. NVMe makes sense to me as a primary root and swap device. I also want to be able to dual boot to FreeDOS (for diagnostic purposes) and FreeBSD on my servers.

I see this as the following;
  • Loader is on a support legacy BIOS device; either SATA or USB drive.
  • Loader defaults to booting the NVMe M.2 SSD.
  • In the event of the NVMe M.2 device failure, it can be booted to the mirrored M.2 SATA device that's on the same PCIe card. This should make root fs disaster recovery simple as I just replace the M.2 NVMe SSD and dd it from the M.2 SATA SSD or let RAID1 take care of reconstructing the NVMe M.2 SSD.
Has anyone done this with loader(8) that can be put on a bootable USB or a small partition of a M.2 SATA drive and add UEFI functionality to a legacy BIOS? Are there detailed examples of this somewhere? It appears that perhaps loader(8) supports this. If loader(8) doesn't have this functionality yet, what other options are available that people here have had success with? The data on my systems is going to be RAIDZ2 with four 3.5" SATA drives.
 

Phishfry

Beastie's Twin

Reaction score: 2,562
Messages: 5,437

Two of my systems on the network I'm constructing do not suport UEFI
This could be a problem. The BIOS has modules for different functions.
To boot off of NVMe you need to have BIOS support.
From what I assume NVMe will only boot off EFI. On legacy BIOS installations the NVMe is found but not bootable.
I can check this out for you and verify.
From what I remember bsdinstall will recognize the NVMe and act like an installable device but when you reboot after installation it will not boot FreeBSD.
So this is more a BIOS issue rather than loader.
I am pretty sure that wth a EFI install you could create a bootable gmirror with a NVMe and SATA device and install onto it. gmirror is not pickey about devices.
 
OP
L

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

This could be a problem. The BIOS has modules for different functions.
Unfortunately, it's the chicken/egg paradox when you're dealing with older BIOSes; now UEFI is the standard. Intel is slated to abandon legacy BIOS support this year.

To clarify the question is can loader(8) boot UEFI NVMe on non-UEFI BIOSes or does the UEFI support come into play after the kernel is loaded with all modules? There are some UEFI commands in loader(8) which prompted my OP.

Thanks.
 

Phishfry

Beastie's Twin

Reaction score: 2,562
Messages: 5,437

I found one machine with legacy BIOS board booting from NVMe
Gigabyte MX-31

Code:
root@MX31:~ # sysctl machdep.bootmethod
machdep.bootmethod: BIOS
root@MX31:~ # nvmecontrol devlist
 nvme0: THNSN5512GPU7 NVMe TOSHIBA 512GB
    nvme0ns1 (488386MB)
root@MX31:~ # kenv|grep vfs
vfs.root.mountfrom="ufs:/dev/nvd0p2"
 
OP
L

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

Phishfry Excellent. That's what I needed. Are there any caveats I need to be aware of besides verifying those two sysctl MIBs? I don't have the NVMe hardware yet.
 

Phishfry

Beastie's Twin

Reaction score: 2,562
Messages: 5,437

The real caveat is BIOS support. My Gigabyte MX31 has an onboard slot for NVMe so that may play a factor into boot support.
The BIOS has the NVMe module needed for booting.
I have seen people do custom BIOS mods to add the NVMe module to unsupported boards.
BIOS use a system of 'modules' for supporting periphery. See Option ROM support as an example..
With the HP Microserver G7 N54L I really wonder if that NVMe module is in the BIOS.
I seriously doubt it. There are modded BIOS for that box but I don't see any NVMe mods.
 

Phishfry

Beastie's Twin

Reaction score: 2,562
Messages: 5,437

I also noticed that the HP-N54L does not use PCIe 3.0 slots. So you are kind of pissing in the wind.
 
OP
L

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

You just saved me a major headache after looking into it further. NVMe is not backwards compatible. You need PCIe 3.x or later for NVMe in any capacity. I need to find a PCIe 2.0 card that appears to the system as a pair of additional SATA drives that's a dual M.2 carrier.

If you're ever in Vegas; a round is on me. :beer:
 

Phishfry

Beastie's Twin

Reaction score: 2,562
Messages: 5,437

What I meant was with PCIe 2.x you will only see around 700MB/s whereas on PCIe 3.x bus you will get full speed like the Samsung PM983@1900MB/s.
NVMe will work on PCIe 2 bus but at degraded speed.
 
OP
L

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

What I meant was with PCIe 2.x you will only see around 700MB/s whereas on PCIe 3.x bus you will get full speed like the Samsung PM983@1900MB/s.
NVMe will work on PCIe 2 bus but at degraded speed.
I was expecting the degraded speed. If I can get comparable to SATA 3 out of NVMe on a PCIe 2.x slot, I'm ok with it as it's better than booting from USB 2.0 and mirroring it.

Another forum has posted that if you're not running PCIe 3.x you might as well run a multiport SATA 3 card as you'll get better bandwidth and not have to deal with BIOS hacks. option ROMs or loaders. With the real estate constraints of the HP MS G7s, M.2 drives are a more attractive option. The issue with HP MS G7s is the lack of SATA ports and PCIe slots which is why I'm looking at NVMe. I want to run a RAIDZ2 in my pair with four 3.5" SATA drives and have mirrored boot SSDs. The other option would be a SATA splitter off of the systemboard internal SATA port which I'm not thrilled about and probably opens up another can of worms.
 
OP
L

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

Yikes,do you mean port multiplier? Those are junk as far as I am concerned.
Yes. I've heard the same thing which is why I'm not at all thrilled about it.
One of the BIOS hacks I saw allowed you to use the Optical Drive connector for regular SATA drives.
You are correct, sir. That was one of the motivating factors for the BIOS hacking; full SATA II capability on all SATA ports.

A very minor peeve; the hacked BIOS though does contain some modules the G7 does not have such as USB 3.0 and an Atheros AR8132M Ethernet chip. It looks like the BIOS hacker spliced together several AMD Athlon II BIOS modules for maximum configuration control of the CPU and chipset to create a FrankenBIOS that works best in the G7s.
 
OP
L

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

One of the BIOS hacks I saw allowed you to use the Optical Drive connector for regular SATA drives.
Which BIOS were you looking at? I installed one of the hacked BIOSes on mine as it appears as a SATA I (1.5 Gbps) IDE/SATA combo port when FreeBSD 12.1 is booted as an ata controller with SATA 1 capabilities. I'm aware of a few hacked G7 BIOSes. It appears as an IDE controller in the BIOS; not a SATA 2 AHCI port.
 

Phishfry

Beastie's Twin

Reaction score: 2,562
Messages: 5,437

I helped a user brick their laptop with a modified BIOS for their IBM laptop. I told myself I would not do that again.
 
OP
L

LVLouisCyphre

Member

Reaction score: 15
Messages: 72

I helped a user brick their laptop with a modified BIOS for their IBM laptop. I told myself I would not do that again.
I don't blame you. I've had to debrick or resurrect failed upgrades some of them due to a hardware bug. Ask me about the Cisco WS-X2922-XL U8 issue and how I took down a large employer's remote site. Cisco did a production run of them where the clock speed chip for the 100BFX module was running at half clock speed essentially making it a 50BFX module. We thought we had all of them swapped out. I remember being on the CERT for it. Some of them fell through the cracks. It would only link to another module with the same incorrect clock chip. The IOS upgrade over the WAN (frame relay) went perfectly. However, everything downstream of that switch didn't after I reloaded the switch. D'oh!

(*FACEPALM*)

That was a f$ck(8) my life moment. If you've had those moments in the trenches then you've earned your merit badges in the business. We all have them.

However with the HP Microserver G7s, the reward greatly justifies the risk to disable the brain damaged IDE SATA combined mode for the internal (optical) and eSATA ports. I did find the issue, it was poor documentation on the hacked BIOS and exactly what needs to be disabled. Both of my G7s are happily running all SATA ports at full AMD 9x0 Southbridge SATA 2.x capacity of 3 Gbs.

Dinking around with FreeBSD 12.1 and burning in my servers, it appears loader has a (mini) EFI shell which may support the gist of this OP. It justifies spending more into my server garden to see if it can be done when funds are available.
 
Top