Other NVMe hot swap working?

spork

Active Member

Reaction score: 10
Messages: 128

I'm about to buy a few more servers and it seems like NVMe is the way things are going, and the pricing is decent enough. The "U.2" standard seems here to stay and there are plenty of servers with NVMe "hot swap" bays for "U.2" format drives.

I've done a fair amount of googling and looking through mailing list archives and I'm not really seeing anything definitive on whether or not FreeBSD supports hot swap on NVMe drives or not. I imagine it's a tricky mess - it's not like pulling a drive off the SATA or SAS bus, you're basically removing a PCIe device without powering the host off.

We'd probably be looking at FreeBSD 12.1 as the OS by the time we have everything in place...

Anyone with practical experience who can weigh in on whether hot swap exists and more importantly, how well it works?
 

tingo

Daemon

Reaction score: 368
Messages: 1,945

NVMe is new, very new. So, with regards to FreeBSD you are asking a question where the answer might be in the future some time.
Questions more suited for today would be things like:
- how good is NVMe support in FreeBSD today?
- how stable is it? (Production, can be tested, or do not put data on it that you can't afford to lose)
- are all NVMe controllers / chips supported?
and so on. No, I do not now the answers to those questions myself, but this is the feeling I get from reading some of FreeBSD's mailing lists on the subject.
 
OP
OP
spork

spork

Active Member

Reaction score: 10
Messages: 128

I came here hoping for those answers. :)

Based on how little there is on the forums and mailing lists, I'm kind of leaning towards avoiding NVMe until one of the big FreeBSD sponsors starts pouring money/hardware into improving support.
 

olli@

Well-Known Member
Developer

Reaction score: 269
Messages: 315

NVMe is new, very new. So, with regards to FreeBSD you are asking a question where the answer might be in the future some time.
Questions more suited for today would be things like:
  • how good is NVMe support in FreeBSD today?
  • how stable is it? (Production, can be tested, or do not put data on it that you can't afford to lose)
  • are all NVMe controllers / chips supported?
and so on. No, I do not now the answers to those questions myself, but this is the feeling I get from reading some of FreeBSD's mailing lists on the subject.
No, NVMe isn't that new. FreeBSD supports it for more than 5 years. NVMe support in FreeBSD is excellent, it's rock stable, and I have never come across a controller that was not supported.

I've got a 1TB Samsung 970 PRO (not the cheaper EVO variant) in a high-end Asus mainboard (M.2 slot), and it's by far the fastest single disk I have ever used to date. You can create a RAID of several of those babies to make it even faster.

However, I can confirm that hot-swap is not yet supported for NVMe devices. According to a recent message on one of the mailing lists, work is under way to implement it, but it's probably much too early to hold one's breath.
 

bsdimp

New Member

Reaction score: 21
Messages: 11

However, I can confirm that hot-swap is not yet supported for NVMe devices. According to a recent message on one of the mailing lists, work is under way to implement it, but it's probably much too early to hold one's breath.
Netflix has been using NVMe to serve video for 3 years now in our OCA platform. It's quite solid and able to deliver full PCIe bandwidth if the NVMe drive supports that. I've been running a NUC as a personal desktop machine for a while now too now that graphics is supported on it.

Hot swap hasn't been too important to Netflix, however, so I've not spent much time with it. Parts of it work, parts not so much. A lot depends on the hot plug controller that's being used (which means you need a custom kernel with options PCI_HP at least). The nvme driver has a detach method, so if that's called manually, the hot plug will work (assuming you've cleaned up all references to it up the stack). Surprise unplug likely doesn't work too well. I have no hot-plug gear, so I can't easily test this stuff.

Warner
 
OP
OP
spork

spork

Active Member

Reaction score: 10
Messages: 128

Yeah, I think for server use, not having hot-swap kind of pulls it from consideration. Supermicro (and I assume others) make backplanes that have SAS/SATA/NVMe connections, and the U.2 form factor slips right in like any other drive. The idea that we'd have to power stuff down to swap, or in the case of M.2 drives, de-rack and disassemble to replace is a show-stopper.

Any realistic thoughts on when FreeBSD gets up to speed on hot-swap? I also wonder if perhaps the server market is going the Apple Way - big disposable box with few moving parts other than some fans and making the U.2 format and hot-swap sort of dead on arrival?

(I know, I know, we're just supposed to move everything to the Cloud) :)
 

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,215
Messages: 4,066

You don't have to power down. Hot-plug (meaning plugging in) works. It's basically just hot-unplug (detaching) that's not automatic.

You unmount the filesystem (UFS) or offline the drive (ZFS).

You use camcontrol(8) or nvmecontrol(8) to detach the device from the bus manually.

You physically remove the drive.

You physically plug in the new drive, whereby the controller detects it and adds it normally.

You start using the device.
 
OP
OP
spork

spork

Active Member

Reaction score: 10
Messages: 128

phoenix - are you running any U.2 drives in production?
 

Phishfry

Son of Beastie

Reaction score: 1,180
Messages: 3,417

I wanted to chime in here. I have some U.2 Samsung PM983 drives for testing and it seems hot swapping does not work.

To take down a drive there is no mechanism in nvmecontrol(8) so I went to the next subsystem and its tool, devctl.
It performs like it takes down the device but there are lingering fragments.
Here is what I used after unmounting:
devctl detach -f nvme0
nvme0 detached

So that seemed to work but when I checked /dev I still see the physical device nvd0.
The namespace is detached as is parent device nvme0.

I try and eject the drive and then reinsert. Nothing comes across the console like direct attached devices would.
Then I try devctl rescan nvme0 and it won't work because nvme0 is not present.
I also try rescaning by bus address since NVMe are PCIe bus devices. No luck.
devctl rescan pci0:2:0:0

There is a mechanism in nvmecontrol to reset but with no NVMe showing that don't work.
Rebooting the machine does reattach.
I feel like you could probably use devctl to also down the physical device too, but the real issue is reattaching.
 

Phishfry

Son of Beastie

Reaction score: 1,180
Messages: 3,417

I bought a second hand LSI Tri-Mode card- 9400-8i to checkout to see how it handles detached NVMe drives.

I could see a Supermicro 24bay chassis with 4 NVMe bays and a couple of LSI 9400 tri mode cards.
 
Top