ZFS Buying a NVMe drive

Tracker

Tracker

Planning to buy an PCIe NVMe drive for the laptop - are there any recommendations or things to keep in mind?
- compatibility with FreeBSD (13.1)
- DRAM drives better?
- connector for thunderbolt helps?
- Any suggestions what to look (out) for?

Mostly intending to buy TLC technology - laptop specification mentions (I'm not sure what this implies) :
- PCIe 3.0 x 4
-PCIe 3.0 x 2 (2nd storage)
- Installed M.2 ssd is PCIe 3.0x4 but run at PCIe 3.0x2 due to M.2 SSD adapter limitation
- Intel Optane memory PCIe NVMe PCIe 3.0x2
 
Well, Samsung 870, 980 and 990 are on the shit list.

I would get a WD SN850X at this point. Disclaimer: I didn't get one yet but they seem to be the most drama free M.2 NVMe drives right now.

Do you plan to make this an external drive or why are you mentioning rhunderbolt?
 
- There's no such thing as "FreeBSD compatibility" either it's NVME or not
- You preferably want drives with a RAM cache (not DRAM less)
- Depends on use case
- I'd look for Crucial's P5 Plus series and/or WD Black series, there are cheaper alternatives but I'd advise you to do some research carefully

If you're going to for a laptop heat is going to be an issue and you likely wont make use of a PCIe 4.0 device anyway so you can look for mid-range instead.

WD SN770 or P3 Plus might be what you're looking for if you're going to use it in a laptop
 
cracauer@ said:
Well, Samsung 870, 980 and 990 are on the shit list.

I would get a WD SN850X at this point.
Click to expand...
That's surprising to me - those are the names of models I found the most when looking on the internet for which ones are good.
Where does one keep track of which drives are decent? Or cost/TB and such metrics?
cracauer@ said:
Do you plan to make this an external drive or why are you mentioning rhunderbolt?
Click to expand...
I'm planning to use it as internal - but might have to use it intermittently as externally - so just wondering about connectivity issues that could come up.
diizzy said:
- There's no such thing as "FreeBSD compatibility" either it's NVME or not
- You preferably want drives with a RAM cache (not DRAM less)
- Depends on use case
Click to expand...
- Got it.
- Ah yes - I saw a few videos mention to go for DRAM ones - since they take care of wear levelling, given that most cells eventually lose capacity with usage - so need an even wear. Is DRAM soldered to the drive and visible to the untrained eye? Can you test such metrics/properties by running some kind of software? (If I opt for a refurbished/used drive).
- It might be subjected to some heavy usage - might even run a btc node just for the heck of it - so going to be constant writes/reads.
diizzy said:
If you're going to for a laptop heat is going to be an issue and you likely wont make use of a PCIe 4.0 device
Click to expand...
I don't quite follow - given the description of the laptop manual it says PCIe 3.0 x(4/2) - I haven't used an NVMe before and unaware of the distinctions/terminology. Am I supposed to go for PCIe 4.0 and it will be backward and future proof?
cracauer@ said:
I would get a WD SN850X at this point.
Click to expand...
diizzy said:
WD SN770 or P3 Plus might be what you're looking for if you're going to use it in a laptop
Click to expand...
Thanks for the recommendations - let me know if my (possible) use case affects the choice keeping in mind durability/laptop heat/usage.
 
Under "normal" use cases you're very unlikely to see a difference between PCIe 3.0 and 4.0 devices except in synthetic benchmarks.
 
Samsung 990 had the very much increased wear rating, it was shooting up like crazy. A firmware fix has been released, but it doesn't give the wear rating back.

Samsung 980 had a big dustup about breaking, for which another firmware fix has been released.

Samsung 870 (not 970) seems to have had bad batches.

25 pages of Samsung details:
www.techpowerup.com

Samsung 870 EVO - Beware, certain batches prone to failure!

Certain 870 EVO 4TB and 2TB drives are affected by early failures where they develop uncorrectable errors and some data just cannot be read from them anymore. This seems to primarily affect drives produced in January/February 2021. For example, i have three 870 EVO 4TB, only one is affected (so...
www.techpowerup.com www.techpowerup.com

They just aren't as good as they were with e.g. the 860 Pro line anymore. This is very sloppy work. And bad communication.

WD and Crucial don't have such screwups on record.
 
I am an ardent user of Samsung PM983. Server drives are the way to go.
Problem is the M.2 drives are 110mm long. That is longer than the 80mm consumer drives.
I have no idea why Samsung is doing so poorly on the EVO's.

I am using an Kioxia 256GB baby Nvme drive in my RockPro64 on an adapter.
 
diizzy said:
Under "normal" use cases you're very unlikely to see a difference between PCIe 3.0 and 4.0 devices except in synthetic benchmarks.
Click to expand...
Are most NVMes PCIe 3.0 or 4.0 these days? If it's not too much of a difference in price then maybe it would make sense to buy a 4.0 for a shift to a more modern system later ?
cracauer@ said:
Samsung 990 had the very much increased wear rating, it was shooting up like crazy. A firmware fix has been released, but it doesn't give the wear rating back.

Samsung 980 had a big dustup about breaking, for which another firmware fix has been released.

Samsung 870 (not 970) seems to have had bad batches.
Click to expand...
Are the "pro" models better for these? I think I read somewhere that most of them had issues until the firmware fix was released.
cracauer@ said:
WD and Crucial don't have such screwups on record
Click to expand...
Ok thanks.
Phishfry said:
I am an ardent user of Samsung PM983. Server drives are the way to go.
Problem is the M.2 drives are 110mm long
Click to expand...
4 tb seems a bit too much for me, was considering 2 tb options. Also since I am planning to run a btc node - I read somewhere that an hdd would suffice for it and I should spare the "precious" NVMe from such loads - wonder if anyone has a take on such a perspective.
Phishfry said:
I am using an Kioxia 256GB baby Nvme drive in my RockPro64 on an adapter.
Click to expand...
Ah yes, my laptop has a slot for one of those baby NVMe drives - need to get that as well, but not so sure if I'll be able to find the right form.

I was also offered these kind of Micron 1 tb drives at a decent rate used Mircon drive - the shopkeeper claimed they were new - but I was wondering how can one verify if such drives are new or not? I don't trust the person selling it. Is there some software one could use?
 
Tracker said:
Ah yes, my laptop has a slot for one of those baby NVMe drives - need to get that as well, but not so sure if I'll be able to find the right form.

I was also offered these kind of Micron 1 tb drives at a decent rate used Mircon drive - the shopkeeper claimed they were new - but I was wondering how can one verify if such drives are new or not? I don't trust the person selling it. Is there some software one could use?
Click to expand...
Could always try running some data recovery software. If all he did was delete the data on the drive you may be able to pick something up with it. I used one years ago. I don't recall what one it was but here's a list of a bunch of them:
www.lifewire.com

The 21 Best Free Data Recovery Software Tools of 2023

Free data recovery software, aka free file recovery or undelete software, can help recover deleted files. Here are reviews of the best ones out there.
www.lifewire.com www.lifewire.com
Tracker said:
4 tb seems a bit too much for me, was considering 2 tb options. Also since I am planning to run a btc node - I read somewhere that an hdd would suffice for it and I should spare the "precious" NVMe from such loads - wonder if anyone has a take on such a perspective.
Click to expand...
If you're worried about it they do have decent hybrid drives out there. Never tried one with FreeBSD. Maybe someone has though.
 
Tracker said:
Are most NVMes PCIe 3.0 or 4.0 these days? If it's not too much of a difference in price then maybe it would make sense to buy a 4.0 for a shift to a more modern system later ?

Are the "pro" models better for these? I think I read somewhere that most of them had issues until the firmware fix was released.

Ok thanks.

4 tb seems a bit too much for me, was considering 2 tb options. Also since I am planning to run a btc node - I read somewhere that an hdd would suffice for it and I should spare the "precious" NVMe from such loads - wonder if anyone has a take on such a perspective.

Ah yes, my laptop has a slot for one of those baby NVMe drives - need to get that as well, but not so sure if I'll be able to find the right form.

I was also offered these kind of Micron 1 tb drives at a decent rate used Mircon drive - the shopkeeper claimed they were new - but I was wondering how can one verify if such drives are new or not? I don't trust the person selling it. Is there some software one could use?
Click to expand...

The "pro" label with Samsung lost all meaning starting with the 980 "pro". Previously they used better Flash.

I really don't think you will get much more performance out of PCIe 4.0 on today's drives in a future computer. On the other hand, power consumption might go up quite a bit.

Bitcoin nodes should be fine on HDDs, even external USB.

Smartmontools on the NVMe will give you past power-on hours.
 
Tracker said:
4 tb seems a bit too much for me, was considering 2 tb options. Also since I am planning to run a btc node - I read somewhere that an hdd would suffice for it and I should spare the "precious" NVMe from such loads - wonder if anyone has a take on such a perspective.
Click to expand...
Let me say the problem with Samsung Enterprise drives is availability.
That was why I posted that link. PM983 are not mainstream.
You should find a good OEM'er to buy them from. There is no factory warranty. They go back to OEM.
So that means your 5 year warranty is at the mercy of the OEM seller.
So find a quality OEM that will be around for 5 years.

If your not committed to M.2 than consider U.2 drives in the 2.5" format.
I have an 4 bay dock with U.2 drives. Nice but expensive.

I also have gmirror of pairs of PM983 M.2 on Supermicro NVMe paddle card. Dual 1TB and Dual 2TB.

For me I use SATA only when necessary. Samsung Enterprise PM983 drives have 1.3DWD.
That is 1.3 drives worth of storage written every day for 5 years.
It can take a beating with high IOPS.
 
+1 for avoiding Samsung. Their firmwares and overall quality have been a dumpster fire for several years now. With the last 3 series they cut their TBW ratings in half with each new model and even those marks seem to be unrealistic for some models if you look at their RMA-rates.
Also Samsung NVMes are complete space heaters and require cooling if you want to really make use of them, or they will constantly trigger thermal throttling.
In short: avoid Samsung, they have become nothing but an expensive, glorified name and there are far better alternatives out there.

For desktop use I can also recommend WD (the blue series are great for low power/temperature applications like laptops and are dirt-cheap) or Transcend (great endurance ratings!). Microns are also very reliable, but I've used them mainly in servers (i.e. the 7xxx series, not the 5xxx consumer-grade drives).

Regarding DRAM: If you are planning on hitting the drive with constant high-bandwidth writes, you might want to get an enterprise drive that can sustain those rates - i.e. doesn't only rely on a (small-ish) cache, but is heavily overprovisioned to get as much work done in parallel as possible. If the DRAM cache in a consumer drive is saturated, writes drop down to the bandwidth those few (1-2) memory chips can handle. That's another reason why you will usually find more cells on an enterprise drive than on a desktop/consumer drive, where the price point is much more important.
OTOH - with only an x2 interface you won't ever get even a consumer drive drive fully utilized, because you can't use all its queues. So in practice I doubt you will ever hit cache saturation and an enterprise-grade drive would definitely be a waste of money - except maybe for their high endurance.
 
Thanks for the suggestions - I ended up getting a 512 gb stick that goes as secondary storage for the machine for now.

It was a used stick - but seemed reasonable for $30ish - or maybe I overpaid, I have no idea.

Is there a way to check the health of the m.2 drive? Somehow camcontrol devlist won't show it, neither does gpart list - they only shows my normal drive (I haven't enabled anything in rc.conf if there is something to be done). A bootable ubuntu stick detected it though - so I thought might as well take it. Now I got to see how much this temporary drive has been bruised and battered.

Update : so apparently this detected it
Code: 
sudo nvmecontrol devlist
nvme0: HFM512GDHTNG-8310A
    nvme0ns1 (488386MB)
 
yuripv79 said:
Install sysutils/smartmontools port, run smartctl -a /dev/nvme0.
Click to expand...
Thanks will try that - meanwhile I tried nvme-cli utility and it gave this output
Code: 
sudo nvme smart-log /dev/nvme0
0xc0484e41: opc: 0x2 fuse: 0 cid 0 nsid:0xffffffff cmd2: 0 cmd3: 0
          : cdw10: 0x7f0002 cdw11: 0 cdw12: 0 cdw13: 0
          : cdw14: 0 cdw15: 0 len: 0x200 is_read: 0
<--- 0 cid: 0 status 0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning                    : 0
temperature                         : 38 C
available_spare                     : 100%
available_spare_threshold           : 10%
percentage_used                     : 4%
data_units_read                     : 31,412,011
data_units_written                  : 16,754,283
host_read_commands                  : 319,422,315
host_write_commands                 : 208,708,309
controller_busy_time                : 2,004
power_cycles                        : 1,373
power_on_hours                      : 6,364
unsafe_shutdowns                    : 24
media_errors                        : 0
num_err_log_entries                 : 0
Warning Temperature Time            : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1                : 38 C
Temperature Sensor 2                : 37 C
Thermal Management T1 Trans Count   : 0
Thermal Management T2 Trans Count   : 0
Thermal Management T1 Total Time    : 0
Thermal Management T2 Total Time    : 0

Update : output of smartmontools - not sure how good/bad it is ? :
Code: 
sudo smartctl -a /dev/nvme0
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       HFM512GDHTNG-8310A
Serial Number:                      CD9AN80441040AU03
Firmware Version:                   80060C00
PCI Vendor/Subsystem ID:            0x1c5c
IEEE OUI Identifier:                0xace42e
Controller ID:                      1
NVMe Version:                       1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Apr  5 03:35:31 2023 +04
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0016):     Wr_Unc DS_Mngmt Sav/Sel_Feat
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     81 Celsius
Critical Comp. Temp. Threshold:     82 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +   3.5000W       -        -    0  0  0  0        5       5
 1 +   2.4000W       -        -    1  1  1  1       30      30
 2 +   1.9000W       -        -    2  2  2  2      100     100
 3 -   0.0350W       -        -    3  3  3  3     1000    1000
 4 -   0.0035W       -        -    3  3  3  3     1000    5000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0
 1 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    4%
Data Units Read:                    31,412,011 [16.0 TB]
Data Units Written:                 16,754,283 [8.57 TB]
Host Read Commands:                 319,422,315
Host Write Commands:                208,708,309
Controller Busy Time:               2,004
Power Cycles:                       1,373
Power On Hours:                     6,364
Unsafe Shutdowns:                   24
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               38 Celsius
Temperature Sensor 2:               37 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged
 
The most important values for SSDs are "Percentage used", 4% is pretty good for a used one, and "Data Units Written", the SSD specification should mention TBW, it depends on the make and capacity, and meaning something like "you can write that many TBs before you'll need to replace SSD", and 8.57TB really looks like "unused".
 
yuripv79 said:
The most important values for SSDs are "Percentage used", 4% is pretty good for a used one, and "Data Units Written", the SSD specification should mention TBW, it depends on the make and capacity, and meaning something like "you can write that many TBs before you'll need to replace SSD", and 8.57TB really looks like "unused".
Click to expand...
Thanks. Yes that's exactly why I went ahead and took the risk of buying it since the nvme-cli mentioned 4% used.

Will try to get hold of the specification and see if I can understand more.

Btw is it possible that these values can be tinkered with easily?

If not, then might go to that shop again and test a few used drives ro pick up next time - what are the most important parameters when judging health of a disk/nvme?
 
I once read an article about overprovisioning by Seagate.
Made sense to me.

Maybe some may confirm, or tell it's a crock?
 
cracauer@ said:
Well, Samsung 870, 980 and 990 are on the shit list.
Click to expand...
You can add the 970 EVO Plus to that list.

Yesterday I ran some stress tests on my Samsung 970 EVO Plus 500GB SSDs.

I have a small Noctua PWM fan which blows air directly onto the SSDs (and which runs flat out at 5250 rpm when the SSDs heat up), and a BeQuiet MC1 Pro M.2 2280 heatsink .

The case is reasonably well ventilated and has some pretty good passive cooling features.

According to Samsung the operating operating temperature of the 970 EVO Plus is 0 - 70C.

Without a heatsink or fan, one of the 970 EVO Plus SSDs reached 100C in less than 10 minutes of running a simple "dd" benchmark, writing at maximum speed. At this temperature it was being seriously throttled.

With the heatsink (and an adjacent fan) it got to 80C after about 20 minutes, and was still rising slowly when the I stopped the test. I collected enough data to believe that it would probably settle at 85C, but there was no point in running the test further, since all I needed to know was that 100% duty cycle was not possible.

It may be possible to run the 970 EVO Plus at 100% duty cycle, at less than 70C, but it would certainly take industrial class cooling to achieve that.

My experience of the older Samsung SSDs has been positive (I still have an 840 PRO somewhere). But this latest M.2 generation is a huge disappointment. How can they sell a storage device that can't be run safely at 100% duty cycle for more than a few minutes?
 
You must log in or register to reply here.
Back
Top