Other Server (U.2) NVMe hot-swap support status?

spork

spork

This is honestly just a revisit of a post from 2019 (see below). I've been asking around on IRC, our Supermicro vendor, etc. and just not coming up with any info on how well hot swap works in practice with NVMe drives on a backplane in a server running FreeBSD. I'm not sure if some of the big guys out there just don't speak publicly about their hardware choices in this day and age or if people still find that SATA/SAS are sufficient. My understanding is that NVMe on laptops and desktops where the "drive" is just inserted in a slot on the mainboard works fine, but also nobody really has reason to be hot-swapping those.

And a recap, still same set of questions:

I'm about to buy a few more servers and it seems like NVMe is the way things are going, and the pricing is decent enough. The "U.2" standard seems here to stay and there are plenty of servers with NVMe "hot swap" bays for "U.2" format drives.
...
Anyone with practical experience who can weigh in on whether hot swap exists and more importantly, how well it works?
Click to expand...

Of note that was 12.2, we'd be talking 14.latest here.

Original thread, where the consensus was "doesn't work" and/or "if it does, too soon to use it":

spork

Other Thread 'NVMe hot swap working?'

I'm about to buy a few more servers and it seems like NVMe is the way things are going, and the pricing is decent enough. The "U.2" standard seems here to stay and there are plenty of servers with NVMe "hot swap" bays for "U.2" format drives.

I've done a fair amount of googling and looking through mailing list archives and I'm not really seeing anything definitive on whether or not FreeBSD supports hot swap on NVMe drives or not. I imagine it's a tricky mess - it's not like pulling a drive off the SATA or SAS bus, you're basically removing a PCIe device without powering the host off...
 
Good question. I just missed a deal on a hotswap frame, so I'm still not able to test.

Probably also depends on the specific equipment.
 
spork said:
I'm not sure if some of the big guys out there just don't speak publicly about their hardware choices in this day and age
Click to expand...

That happens. I don't really know why. I would occasionally like to have a talk about some topic from the pro realm, but to no avail.

And a recap, still same set of questions: Of note that was 12.2, we'd be talking 14.latest here. Original thread, where the consensus was "doesn't work" and/or "if it does, too soon to use it"
Click to expand...

Nice question. I think the matter is, does the hardware do ist correctly? I have no hardware to test such, but on the software side, if I get it correctly, the U.2 is just an omnibus, and the actual connection is PCI. The question then is, can we hotswap PCI? Or does the following look any different with U.2?

Code: 
# devinfo
    pcib0
      pci0
        hostb0
        pcib1
          pci1
            nvme0
# devctl detach nvme0
# devinfo
    pcib0
      pci0
        hostb0
        pcib1
          pci1
#

Apr 20 00:09:05 <kern.crit> disp kernel: [427617] nda0: <KBG50ZNV512G KIOXIA HP01AN00 234PH47KQMLK>
Apr 20 00:09:05 <kern.crit> disp kernel: [427617] (nda0:nvme0:0:0:1): Periph destroyed
Apr 20 00:09:05 <kern.crit> disp kernel: [427617] nvme0: detached

I don't see reason why it should not be possible to rip the device out of the system at this point.
But then, how do we get it back? A few unsuccesful ventures, and then:

Code: 
# devctl reset -d  pci1
# devctl
    pcib0
      pci0
        hostb0
        pcib1
          pci1
            nvme0
#

Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nvme0: <Generic NVMe Device> mem 0xf7d00000-0xf7d03fff irq 16 at device 0.0 on pci1
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nvme0: failed to alloc HMB
Apr 20 00:13:54 <kern.crit> disp syslogd: last message repeated 3 times
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nvme0: Allocated 40MB host memory buffer
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nda0 at nvme0 bus 0 scbus3 target 0 lun 1
Apr 20 00:13:54 <kern.crit> disp kernel: [427906] nda0: <KBG50ZNV512G KIOXIA HP01AN00 234PH47KQMLK>

Seems to work:

Code: 
# dd if=/dev/nda0 bs=4096
^C
1613+0 records in
1613+0 records out
6606848 bytes transferred in 0.829604 secs (7963861 bytes/sec)

So, what else do we need?
 
spork, U.2 hot-plug is essentially NVMe hot-plug which is essentially PCIe hot-plug.
It should work on FreeBSD as long as the hardware properly supports it.
 
I understand, just looking for some real-world feedback before we jump in all the way on a large order.
 
On large servers ( SPARC ) we have commands for turning off/on power feed to individual drive slot or individual PCIE-slot before removing / inserting the HOTSWAP Storage/HBA-PCIe unit. If Power is ON in the slot when you try to insert the device , risk of failure increase. Much less risk , if power is off to device while HOTswapping and power can be applied to the slot when the exchange is completed.
While Hotswap storage cabinette can be purchased for X86 solutions, Is there any that arrives with the ability to manipulate Power ON/OFF to the slots ?
Hotswapping a PCIE device with Power-on does not seem risk free even with your Anti-static wristwrap ground lead attached .
 
Just want to be clear in my ask here - I know this is supported in other OSes and in theory in FreeBSD as well, and I'm guessing it likely works. Just kind of trying to find some real-world users/data though to help with the decision-making process and get a feel for if this is still "early days" or if there's people like Netflix out there just running NVMe exclusively across their entire fleet...
 
So the x86 servers I've had in the last 5 years or so just work with FreeBSD and PCIe/NVMe U.2 hotplug. Arm has been a mixed bag, since there's a firmware interaction and the firmware there is less mature. I've not had time to chase down the cause since I don't hotplug that often...
 
You must log in or register to reply here.
Back
Top