4 drive zpool, two NVMe, two SSD

dvl@

dvl@

Developer
I have four 1TB storage devices and I'm planning to combine them into a 2TB ZFS array. I have scavenged these devices from hosts which are being decommissioned.

The devices, as seen on FreeBSD 13.2, are:

Code: 
nvd0: <Samsung SSD 980 PRO with Heatsink 1TB> NVMe namespace
nvd0: 953869MB (1953525168 512 byte sectors)
nvd1: <Samsung SSD 980 PRO with Heatsink 1TB> NVMe namespace
nvd1: 953869MB (1953525168 512 byte sectors)

da12 at mrsas0 bus 1 scbus1 target 12 lun 0
da12: <ATA Samsung SSD 860 2B6Q> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number [redacted]     
da12: 150.000MB/s transfers
da12: 953869MB (1953525168 512 byte sectors)
da13 at mrsas0 bus 1 scbus1 target 13 lun 0
da13: <ATA Samsung SSD 860 2B6Q> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number [redacted]     
da13: 150.000MB/s transfers
da13: 953869MB (1953525168 512 byte sectors)

The NVMe sticks are on two PCIe slots in a Dell R730. The two SSDs are in the drive bays at the front of the chassis.

Ignore 150.000MB/s transfers - that's an mrsas driver issue - it's false reporting - a hardcoded value.

My idea: mirror one of the NVMe with an SSD. Why? Diversity. Same size, different manufacturer.

Yes, they will have different latency. What's the downside? I'm sure it will still work well. This host is in my basement homelab. Any comments?

Thanks.
 
4 1TB devices into a 2TB vdev so stripe of mirrors? I agree with NVME and SSD as the mirror pairs, in theory writes will complete at the speed of the slowest (SSD) and reads at the speed of the fastest (NVME). Toss in the stripe aspect and I would expect reads to be overall very fast, writes at the speed of the SSDs.

Good experiment.

dvl@ I've been following this advice for a while, especially with mixed devices. (Not sure if it sounds familiar)


ZFS: do not give it all your HDD

Sometimes, holding out is the best decision.
www.freebsddiary.org
 
mer said:
dvl@ I've been following this advice for a while, especially with mixed devices. (Not sure if it sounds familiar)


ZFS: do not give it all your HDD

Sometimes, holding out is the best decision.
www.freebsddiary.org
Click to expand...
I do do that. Every. Single. Time.

Case in point, these drives are already partitioned but not the same size. Some information at Preparing a server for sale – Supermicro 846 – 10 x 5TB HDD - search for clear. I'll blog about what I do for that later.

I've just moved the array from r730-03 (mostly storage) to r730-01 (mostly development). I figured it would be more useful there. Blog post soon.
 
  • Like
Reactions: mer
dvl@ Dan, thanks for being a good sport on this. And yes, I also do that every single time. My systems are mostly just mirrors, but making a 3-way mirror to replace a failing drive, I do this. I also recommend it to people when they ask about "ZFS gpart or whole device".

And serious thanks for the blog over the years.
 
dvl@ said:
I've just moved the array from r730-03 (mostly storage) to r730-01 (mostly development). I figured it would be more useful there. Blog post soon.
Click to expand...

Here we go, the host is powered up. I'm about done for the day. History: these were pulled from working hosts, hence the existing partitions. Note how they are not identical. I have no doubt that ZFS could handle this mismatch, but I'll adjust them so all four match.

Code: 
[19:13 r730-01 dvl ~] % gpart show nvd0 nvd1 da12 da13
=>        40  1953525088  nvd0  GPT  (932G)
          40  1953484800     1  freebsd-zfs  (931G)
  1953484840       40288        - free -  (20M)

=>        40  1953525088  nvd1  GPT  (932G)
          40  1953484800     1  freebsd-zfs  (931G)
  1953484840       40288        - free -  (20M)

=>        40  1953525088  da12  GPT  (932G)
          40  1953525088     1  freebsd-zfs  (932G)

=>        40  1953525088  da13  GPT  (932G)
          40  1953525088     1  freebsd-zfs  (932G)

[19:13 r730-01 dvl ~] %
 
mer said:
dvl@

And serious thanks for the blog over the years.
Click to expand...

I enjoy blogging. I really do. It is quite satisfying. I refer to my blogs for my own use. That is what they started off as, way back when I was trying to get my host running with DHCP on ADSL. Notes to help myself, mostly when seeking help and explaining to others what I did.

I'm glad the blogs are useful to others. One of my goals is to provide enough detail that anyone, including myself, can go back and repeat the steps and have a high chance of success. I include the mistakes and errors messages because everyone gets those and needs to know how to remedy them. Me included.

I'll be back on this on Sunday, but for now, there's a birthday party to attend.
 
  • Like
Reactions: mer
I'm assuming you are going to mirror nvd0 da12 and nvd1 da13 and then stripe across those mirrors.
Sorry I don't have the exact zpool create syntax at hand.
I think it will be interesting to see what the "iostats" wind up at (read performance vs write performance)
As for mismatch I think ZFS goes to smallest, so 1953484800 instead of 1953525088.
 
mer said:
I'm assuming you are going to mirror nvd0 da12 and nvd1 da13 and then stripe across those mirrors.
Sorry I don't have the exact zpool create syntax at hand.
I think it will be interesting to see what the "iostats" wind up at (read performance vs write performance)
As for mismatch I think ZFS goes to smallest, so 1953484800 instead of 1953525088.
Click to expand...
Yes, that is the plan: nvd0 & da12 in one mirror, and nvd1 & da13 in the other mirror.

Yeah, so the mismatch, I'm going to adjust so they match.
 
Have you looked at the NVMe with smartmontools?
You can query the namespace and it gives alot of information about the drives.
Some Samsungs have 512 or 4K sector size which is configurable.
Since these were scraped you might want to check to see how the two drives compare.
smartctl -a /dev/nvme0

Thank You for all your various publications both new and old.
 
Phishfry said:
Have you looked at the NVMe with smartmontools?
You can query the namespace and it give alot of information about the drives.
Some Samsungs have 512 or 4K sector size which is configurable.
Since these were scraped you might want to check to see how the two drives compare.
smartctl -a /dev/nvme0
Click to expand...

FYI, they weren't so much scrapped as sitting in one of my own servers which had been powered off for months as I moved to new servers.

Here's one:

Code: 
[20:31 r730-01 dvl ~] % sudo smartctl -a /dev/nvme0
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.2-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 980 PRO with Heatsink 1TB
Serial Number:                      [redacted]
Firmware Version:                   4B2QGXA7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      6
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization:            920,447,864,832 [920 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 b22140998d
Local Time is:                      Sun Oct 22 20:31:28 2023 UTC
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0057):     Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     82 Celsius
Critical Comp. Temp. Threshold:     85 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.49W       -        -    0  0  0  0        0       0
 1 +     4.48W       -        -    1  1  1  1        0     200
 2 +     3.18W       -        -    2  2  2  2        0    1000
 3 -   0.0400W       -        -    3  3  3  3     2000    1200
 4 -   0.0050W       -        -    4  4  4  4      500    9500

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        37 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    10,154,825 [5.19 TB]
Data Units Written:                 7,953,536 [4.07 TB]
Host Read Commands:                 65,823,607
Host Write Commands:                154,501,870
Controller Busy Time:               156
Power Cycles:                       9
Power On Hours:                     3,206
Unsafe Shutdowns:                   2
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               37 Celsius
Temperature Sensor 2:               41 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
No Self-tests Logged
 
Yup 512 LBA only. On Samsung PM983 drives I see 4048 as well as 512.
The plus sign + means the current setting.
Lots of power states.

dvl@ said:
Power On Hours: 3,206
Click to expand...
Barely broke in. I like my samsung drives. No failures yet knock on wood.

I am glad to hear the two drive Dell NVMe M.2 paddle card works with FreeBSD.
I have seen them on fleabay and wondered if they were proprietary.
 
BTW, the PCIe cards are not Dell gear. They are https://www.amazon.com/gp/product/B07JJTVGZM

This is one of the SSDs: 42044 hours.

Code: 
[20:52 r730-01 dvl ~] % sudo smartctl -a /dev/da13
smartctl 7.4 2023-08-01 r5530 [FreeBSD 13.2-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 860 EVO 1TB
Serial Number:    [redacted]
LU WWN Device Id: 5 002538 e4094967b
Firmware Version: RVT02B6Q
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Oct 22 20:52:43 2023 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (    0) seconds.
Offline data collection
capabilities:              (0x53) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (  85) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       42044
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       66
177 Wear_Leveling_Count     0x0013   001   001   000    Pre-fail  Always       -       2074
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   078   047   000    Old_age   Always       -       22
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   099   099   000    Old_age   Always       -       2
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       17
241 Total_LBAs_Written      0x0032   097   097   000    Old_age   Always       -       7136126656590

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      1721         -
# 2  Short offline       Completed without error       00%      1553         -
# 3  Extended offline    Completed without error       00%      1400         -
# 4  Short offline       Completed without error       00%      1385         -
# 5  Short offline       Completed without error       00%      1217         -
# 6  Extended offline    Completed without error       00%      1076         -
# 7  Short offline       Completed without error       00%      1049         -
# 8  Short offline       Completed without error       00%       881         -
# 9  Short offline       Completed without error       00%       713         -
#10  Extended offline    Completed without error       00%       656         -
#11  Extended offline    Completed without error       00%         1         -
#12  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The above only provides legacy SMART information - try 'smartctl -x' for more
 
Right single slot M.2 adapters. Boxes of them. I took some to mill and miniaturized for RockPro64.
Real low profile.

This is the Dell one I am refering to;
www.ebay.com

Dell DPWC300 SSD PCIE Adapter Card 2 Slot M.2 0NTRCY 023PX6 No m.2 NVME SSD | eBay

These are pulled from working systems. They are in great condition and are ready for reuse. So any other details would have to be answered by the manufacturer based on the serial number and or part number we are providing.
www.ebay.com
www.ebay.com

Dell SSD NVMe M.2 PCI-e 2x Solid State Storage Adapter Card 23PX6 NTRCY | eBay

If you intend to buy it for use in any other system out of Dell's compatibility matrix, it will void our warranty and support on this product. For Dell Precision T5810 T7810 T7910 T5820 T7820 T7920.
www.ebay.com

I used SuperMicro 2 slot NVMe adapter with success. Needs bifurcation.
I wonder if Dell card works in SuperMicro and vice versa.
Is the bifurcation scheme a standard or proprietary with these cards.
 
I considered cards which required bifurcation. I decided to avoid them and go for the simple solution. I'm not short of slots yet.
 
Phishfry said:
Using two NVMe per slot could be seen as a weakening of the mirror. You can't just yank the card out. Extra step.
Click to expand...
Yes, it would. However, I am creating a mirror of an NVMe and an SSD.

Additionally, if either of the NVMe needs to be replaced, it requires a power off. The SSDs are in hot-swappable front-panel drive slots.

I also have some SSDs in PCIe slots which are hot-swappable. Very nice solution.
 
Because of this smart status.

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 17
Click to expand...
A count of the number of sudden power off cases. If there is a sudden power off, the firmware must recover all of the mapping and user data during the next power on. This is a count of the number of times this has happened.
Click to expand...
 
You must log in or register to reply here.
Back
Top