ZFS Force trim SSD before heavy IO

zeRusski

zeRusski

Hi. I have a ZFS pool of NVMe SSDs which is used as a "scratchpad" in a long running heavy IO multi-process task. vfs.zfs.trim.enabled=1 however I wonder given the peculiar nature of the task that basically writes all over non stop for hours if shouldn't force TRIM of each drive in a pool after these tasks finish, before starting a new batch. Should I? If so, then with what command. I thought I'd seen zpool trim mentioned on the interwebs but it isn't in man. fsck_ffs -E but on zfs???

Although prefixed ZFS I'd like to know the answer for UFS, too since I've been experimenting with filesystems and UFS is still my preferred option. So far ZFS perf for task at hand hasn't impressed, sadly.

*EDIT* When I say "scratchpad" I do actually mean that. It is perfectly ok to have those drives completely wiped before every batch if that helps perf.

Thank you
 
I can't give you an immediate answer, but I doubt this would accomplish anything. When I delete stuff in a VM with trim enabled, I almost immediately see the host's sparse zvol providing the storage shrink.

Maybe it would be better to have a thread title refering to the actual problem, not to a presumed solution.
 
One way to figure if "there is a problem" would be to have the ability to trigger TRIM on demand e.g. with a command. Then I could run my test load without forced trim before and with and see if it improves perf any. Also, when one does read about trim its all a bit magical in that your fs manager essentially notifies SSD firmware to reclaim stuff ... crude and wrong wording but I don't want to reproduce better explanations available on the web. Another reason to trim IIRC is to save SSD wear and tear somewhat - now sadly I can't reasonably measure that other than run multiple SSDs until their entire reported TBW is gone etc etc. I think it is also not unreasonable to expect current "trim" trigging algos to be optimized for typical e.g. desktop use, whereas my case is extremely different with heavy IO and load known upfront - whole disk utilized etc etc.

Now, I don't know if what I ask for is entirely stupid and it is highly likely that I don't undrestand TRIMming at all. I'll be happy for someone with the knowledge to explain the need away. But right now I see a reasonable way to just run a test and find out.
 
A sane implementation of TRIM will always notify the drive as soon as a block is no longer occupied, so I'm pretty sure doing a re-trim (which will tell the drive again about ALL currently unused blocks) won't change anything for you. You need such a command for example when you had TRIM previously disabled.

Yes, TRIM was introduced for SSDs cause they need it. SSD firmware maps blocks internally to make sure writes are equally distributed, and to do this efficiently, it must know which blocks are currently unused.

Still, TRIM is also very useful for virtual storage as the host can reuse space.
 
The ZFS version of trim is ... complicated (*). And a relatively new addition. I don't know which FreeBSD version supports it.

The reason it's complicated is that ZFS is a log-based or log-structured file system. It doesn't overwrite things in place, and it doesn't free and overwrite deleted space immediately, instead writing new data at the end of a log-like data structure. Deletes in ZFS are mostly asynchronous (mostly because sometimes you can have a deliberate priority inversion and "foreground asynchronous" when space runs short).

I don't know whether in normal production usage, ZFS's log cleaner falls behind significantly. The answer clearly depends on the pattern of deletes. How many files are there? Are they all deleted at once? Are for example small files in that pool left behind, making it hard for the log cleaner to coalesce free space? Or are only small files deleted, making coalescing unprofitable?

As to zeRusski's claim that "expect current "trim" trigging algos to be optimized for typical e.g. desktop use": Probably not, and depends heavily on what type of disk this is. If you look at worldwide disk production, about half are consumer disks (typ. low RPM), half are enterprise. Of the enterprise disks, over 90% are sold to a handful of customers (the FAANG), and used in very particular fashions. For SSDs, the differentiation between consumer and enterprise usage is even more stark, as there is a lot more firmware and virtualization in an SSD, allowing the SSD manufacturers to optimize them for desktop versus enterprise workload patterns. So if you are buying enterprise SSDs, they are very much not optimized for desktop usage.
 
well, the question has been rendered moot by ZFS exhibiting a substantial slowdown compared to just targeting all those drives as separate UFS without soft updates. I expect and hope that FreeBSD performs TRIM on those out of the box without my tweaking anything, right? :)

Obviously, this says very little about ZFS. Like I mentioned this a very unusual type of workload I'm doing here.
Another interesting experiment I guess would be to put those UFS drives in software RAID0
 
gstripe RAID0 performs substantially better than ZFS but still not as good as jbod. Since I only have two pairs of NVMe drives of different size I ended up with two RAID0 arrays 2 drives each. Dunno, maybe with more drives in RAID0 I could gain some, but it isn't worth it.

Well. Much time wasted that could've been used to just .. perform said task. At least I learnt something. No regrets, I guess.
Thank you everyone for playing.
 
Mjölnir said:
zeRusski (Vladilen?), are you that guy from the mailing list who I recommended to try ZFS anyway? Just curious.
Click to expand...
that would be me. I didn't know forums existed and I'm still confused what the appropriate (better) venue for FreeBSD discourse is :) One can gauge Forum activity at a glance, not so easily done with mailing lists. If you think these belong in the mailing list, I'm happy to move there
 
No no, the forum is the "modern" clickybunty way to discuss technical questions. More developers & wizzards are on the mailing-lists, though. That doesn't mean you can't get a good answer here, too.
 
zeRusski said:
… vfs.zfs.trim.enabled=1 …
Click to expand...

Which version of FreeBSD?

vfs.zfs.trim.enabled is unknown with FreeBSD 14.0-CURRENT (below); seeking vfs.zfs.trim in commit logs finds nothing to suggest that it's specific to main/head, so I assume that OpenZFS 2.⋯ is, somehow, the significant difference.

Related: FreeBSD 12.2-RELEASE-p9 trim: open failed: /dev/ada0: Operation not permitted with reference to FreeBSD 10 Trim for ZFS | The FreeBSD Forums

<https://cgit.freebsd.org/src/log/?qt=grep&q=vfs.zfs.trim>

<https://cgit.freebsd.org/src/log/?h=stable/13&qt=grep&q=TRIM> – various ZFS-related commits found by seeking TRIM.

The example below is just an experiment in response to the freebsd-questions post; I don't intend to actually trim anything here.

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # gpart show
=>        40  1953525088  ada0  GPT  (932G)
          40      532480     1  efi  (260M)
      532520        2008        - free -  (1.0M)
      534528    33554432     2  freebsd-swap  (16G)
    34088960  1919434752     3  freebsd-zfs  (915G)
  1953523712        1416        - free -  (708K)

=>     40  3913648  da1  GPT  (1.9G)
       40     2008       - free -  (1.0M)
     2048  3911640    1  !8c8f8eff-ac95-4770-814a-21994f2dbc8f  (1.9G)

=>       34  976773101  da6  GPT  (466G)
         34       2014       - free -  (1.0M)
       2048  976771087    1  freebsd-zfs  (466G)

=>      34  30310333  da7  GPT  (14G)
        34      2014       - free -  (1.0M)
      2048  30308319    1  freebsd-zfs  (14G)

=>      34  60437425  da8  GPT  (29G)
        34  60437425    1  freebsd-zfs  (29G)

root@mowa219-gjp4-8570p-freebsd:~ # /usr/home/grahamperrin/dev/lsblk/lsblk da0
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
da0              0:183 1.9G freebsd-swap                                      - SWAP
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep /dev/da
root@mowa219-gjp4-8570p-freebsd:~ # trim -v /dev/da0
trim /dev/da0 offset 0 length 1993342976
dry run: add -f to actually perform the operation
root@mowa219-gjp4-8570p-freebsd:~ # trim -fv /dev/da0
trim /dev/da0 offset 0 length 1993342976
trim: /dev/da0: TRIM/UNMAP not supported by driver
root@mowa219-gjp4-8570p-freebsd:~ # trim -fv /dev/da1
trim /dev/da1 offset 0 length 2003828736
trim: /dev/da1: TRIM/UNMAP not supported by driver
root@mowa219-gjp4-8570p-freebsd:~ # trim -fv /dev/da7
trim /dev/da7 offset 0 length 15518924800
trim: open failed: /dev/da7: Operation not permitted
root@mowa219-gjp4-8570p-freebsd:~ # gpart show -l da7 da8
=>      34  30310333  da7  GPT  (14G)
        34      2014       - free -  (1.0M)
      2048  30308319    1  cache-transcend  (14G)

=>      34  60437425  da8  GPT  (29G)
        34  60437425    1  cache-august  (29G)

root@mowa219-gjp4-8570p-freebsd:~ # zpool offline Transcend gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # trim -fv /dev/da7
trim /dev/da7 offset 0 length 15518924800
trim: /dev/da7: TRIM/UNMAP not supported by driver
root@mowa219-gjp4-8570p-freebsd:~ # zpool online Transcend gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # trim -fv /dev/da7
trim /dev/da7 offset 0 length 15518924800
trim: open failed: /dev/da7: Operation not permitted
root@mowa219-gjp4-8570p-freebsd:~ # sysctl vfs.zfs.trim.enabled
sysctl: unknown oid 'vfs.zfs.trim.enabled'
root@mowa219-gjp4-8570p-freebsd:~ # sysctl vfs.zfs.trim.enabled=1
sysctl: unknown oid 'vfs.zfs.trim.enabled'
root@mowa219-gjp4-8570p-freebsd:~ # date ; uname -aKU
Mon Aug 30 05:45:47 BST 2021
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #105 main-n248685-c9f833abf1d: Fri Aug 13 20:24:43 BST 2021     root@mowa219-gjp4-zbook-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG  amd64 1400030 1400032
root@mowa219-gjp4-8570p-freebsd:~ #

<https://www.freebsd.org/cgi/man.cgi...tion=8&manpath=FreeBSD+12.2-RELEASE+and+Ports> there's the manual page for zpool-trim(8) only with ports for FreeBSD 12.2-RELEASE.

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim august
cannot trim: no devices in pool support trim operations
root@mowa219-gjp4-8570p-freebsd:~ # zpool trim Transcend
cannot trim: no devices in pool support trim operations
root@mowa219-gjp4-8570p-freebsd:~ #
 
You must log in or register to reply here.
Back
Top