ZFS Adding NVMe with 4kb logical/256kb physical sector size as cache to ashift 12 zpool

S

shorton

I'm trying to add NVMe cache to several zpool that are set with ashift 12. When trying to add the NVMes (Micron_7450_MTFDKCC800TFS), the zpool status shows the following:

status: One or more devices are configured to use a non-native block size.
Expect reduced performance.

cache
nvme3n1p1 ONLINE 0 0 0 block size: 4096B configured, 262144B native


Will these NVMe drives not work as cache devices on these zpools?

parted /dev/nvme3n1 print
Model: Micron_7450_MTFDKCC800TFS (nvme)
Disk /dev/nvme3n1: 800GB
Sector size (logical/physical): 4096B/262144B


Thanks any help provided with this.
 
My understanding is that using a nonnative blocksize with any drive smaller than its native requires reading a larger native block, updating the part for your write, and writing the modified larger block which all takes place instead of just writing your intended block. SSDs have normally used larger pages behind the scenes despite reporting as 512B/4K sector sizes and behind the scenes management means you don't control it or use it natively even if you wanted to. Impacts of this would be if 'rewriting' so writing an empty block should keep up performance.

ZFS currently goes up to ashift=16 (64kB) maximum so your native size (ashift=18) isn't an option. Don't know if >12 has received much more testing but thought I recall uberblock(?) count drops as ashift goes up so a degree of redundancy is lost as it increases and fewer blocks receive those writes accordingly (shouldn't matter unless discussing a drive without wear leveling like an old/cheap USB stick, SD card, etc.). Increasing ashift decreases pool space based on things like file count (no smaller blocks than ashift used for data that doesn't fit exactly), pool layout (RAID parity and padding). Wonder if ZFS requires much effort to rework to future larger ashift values.

I thought I recalled FreeBSD fits writes and reads into 128K as its largest size but it seems that is a tunable as kern.maxphys. Discussion of making it tunable say ZFS didn't follow that and used larger writes but I haven't seen it happening; maybe it is a limitation of `systat -v` output.

https://www.micron.com/content/dam/...keting-brief/7450-nvme-ssd-tech-prod-spec.pdf only mentions 512 and 4k as sector choices with no discussion of 262144 so you are probably fine as it sounds like micron designed the drive with 4k alignment/sizing as an intended use.

https://github.com/openzfs/zfs/pull/13798 sounded like an interesting read implying similar so some of what I said above and ends saying there is a firmware update. I do not know if there is a value gained or lost in having a drive go back to keeping secrets if it is truly 256k but magnetic drives sure made a mess of things when they lied saying they are 512 instead of 4096.
 
Thanks for the detailed response! It turned out to be a firmware issue with the Micron NVMes. I ended following the last thread in the github issues page to update the firmware. I'm now seeing the 4k physical sector size and I'm able to add as cache to the zpool now.


parted /dev/nvme0n1 print
Model: Micron_7450_MTFDKCC800TFS (nvme)
Disk /dev/nvme0n1: 800GB
Sector size (logical/physical): 512B/4096B


Thanks,
 
I have several Micron 7450s running in various servers with ZFS. It's not a firmware issue, just reformat the namespace to 4k (LBA #01).
Most SSDs come with 512b sectors pre-formated because a certain inferior OS often has problems with sectors >512b. Just use nvmecontrol format to switch to the LBA format #01 (=4096b) prior to adding the namespace to a ZFS pool.
 
Meant to say github issue 13917 instead of 13798 but glad you found it. https://www.mouser.com/PCN/Micron_Technology_PCN_34996_Micron_7450_E2MU200_Change_List_External.pdf (available at a few places elsewhere on the web) sounds like there are other good fixes to the firmware update. I always wonder what really changes on these things as companies are normally giving a very brief summary for firmware and BIOS/UEFI changes. Would be interesting to know how things showed before and after for all nvmecontrol reported details.
 
I just checked some of those 7450 drives - it appears they are all running the E2MU200 firmware version - so ignore my previous post, sorry.
 
You must log in or register to reply here.
Back
Top