PCIe 3.0 M.2 Card Suggestions for NVME

[ClintBSD]

Hi All,

I am interested to buy a PCIe 3.0 M.2 card for using an NVME drive. I wanted to use this drive as a cache drive for transcoding and possibly other tasks that create a lot of write cycles. I may store my jails on there as well. I am just looking for some suggestions since maybe someone has a similar setup and has success with a certain make and model. The goal is to not have a lot of write cycles on the host OS boot drive.

Hardware Details:
Dell R720 server

CPU 1

Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz

Model 62 Stepping 4

2100 MHz

Presence Detected

6

CPU 2

Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz

Model 62 Stepping 4

2100 MHz

Presence Detected

6

	Connector Name	Type	Size	State	Rank	Speed
	DIMM A1	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz
	DIMM A2	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz
	DIMM A3	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz
	DIMM A4	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz
	DIMM B1	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz
	DIMM B2	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz
	DIMM B3	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz
	DIMM B4	DDR-3	8.00 GB	Presence Detected	Dual Rank	1600 MHz

6 x 6TB SAS drives in a raidz1-0

I found this adapter on Amazon:
Dual M.2 PCIE Adapter

Any suggests would be much appreciated.

 

[cracauer@]

I buy the WD sn850x when I don't have Optane or enterprise drives.

 

[ClintBSD]

I buy the WD sn850x when I don't have Optane or enterprise drives.

Which PCIE adapter card are you using?

 

[sko]

I've been using WD blue as low-power NVMes where low TBW is sufficient (e.g. boot drives). These are also our 'go-to' NVMe drives in clients - out of ~40 WD blue M.2 drives I haven't had a single failure.
For more write-intensive workloads (e.g. my poudriere build pool) I've tried the WD red series, but IMHO they run way too hot and aren't as performant as micron 7300/7400/7450... but if budget is tight they are quite OK if cooling is sufficient. However, TBW-ratings vary wildly across different models/generations of those WD red drives (after all, 'WD red' always were glorified prosumer drives, not enterprise...), so pick one that suits your use-case. If the budget allows it I'd go for micron.

As PCIe adapters I'm using the SuperMicro AOC-SLG3-2M2 pretty much everywhere. I've tried various x16 / 4xM.2 cards but they weren't as reliable as the SM dual carriers. Proper/sufficient power supply to 4 drives seems to be an issue, as I often had drives not showing up on boot or disappearing under load with those quad adapters. Cooling is also an issue on them, especially if airflow is passing lateral over the drives, so the last one gets significantly hotter than the first one.

In desktops I've also been using those cheap single-drive carrier cards from aliexpress and they never gave me any issues - after all those are nothing but an adapter card and power draw of a single drive is also easy to handle.

 

[cracauer@]

Which PCIE adapter card are you using?

Probably this one:

https://www.amazon.com/dp/B07Q1BYPP8

 

[diizzy]

A few things, you need PCI-bifurcation for multiple devices working in a single slot unless there's a PCIe bridge on the card itself.
I would highly recommend getting a branded board, such as https://www.ebay.de/itm/364642021822 or the SuperMicro card mentioned above or you need a card with a PCIe bridge such as https://www.delock.de/produkt/90409/merkmale.html?setLanguage=en . I have no idea what bridge chipset is the best one so you have to do your own research on that one.

In general you want to avoid DRAM-less NVME drives and looking at https://github.com/openzfs/zfs/discussions/14793 there might be some issues with at least some models from WD. I've personally seen various drives from the WD Blue line die so I'd look elsewhere. I'd probably look at Crucial T500 / T700 / T705 / T710 although T7**-series are likely overkill for your use-case. Another alternative would be Seagate Firecuda 530R but I would also be cautious about Samsung due to their various firmware quirks. I'm personally very happy with the P5 Plus (replaced by T500) series in various applications.

 

[ClintBSD]

A few things, you need PCI-bifurcation for multiple devices working in a single slot unless there's a PCIe bridge on the card itself.
I would highly recommend getting a branded board, such as https://www.ebay.de/itm/364642021822 or the SuperMicro card mentioned above or you need a card with a PCIe bridge such as https://www.delock.de/produkt/90409/merkmale.html?setLanguage=en . I have no idea what bridge chipset is the best one so you have to do your own research on that one.

In general you want to avoid DRAM-less NVME drives and looking at https://github.com/openzfs/zfs/discussions/14793 there might be some issues with at least some models from WD. I've personally seen various drives from the WD Blue line die so I'd look elsewhere. I'd probably look at Crucial T500 / T700 / T705 / T710 although T7**-series are likely overkill for your use-case. Another alternative would be Seagate Firecuda 530R but I would also be cautious about Samsung due to their various firmware quirks. I'm personally very happy with the P5 Plus (replaced by T500) series in various applications.

I read the R720 doesn't support bifurcation. So I guess I can just stick to a single slot NVME device to keep it simple or go with the model you suggest that has an internal PCIE bridge. Thanks very much for the information.