This leaves me wondering how this should ever relate to caching? After all, trim is just about letting the "layer below" (whatever it is) know about blocks that aren't in use any more. I use that with one VM that needs its own ZFS although it's hosted on a zvol already, that way I can at least make sure that sparse zvol shrinks whenever possible.I wouldn't suggest autotrim depending on the scenario because several reasons, some of them I can be wrong (I don't know how the zfs caches works/interacts with trim, to start)
My guess (no official statement!) about the reason: There are lots of scenarios where trim wouldn't do any good. In a nutshell whenever the backing storage is a) fixed size and b) not an SSD. So it should be opt-in.Why is it diabled by default?
No practical usage in this doc.
On the other hand there is https://github.com/freebsd/freebsd-src/commit/5e7eaf8fbda348db9878ce013ecb4ab5e9b5cf5a. (Maybe that doesn't work with pool upgrades? I don't quite remember what it is my issue with that option.)Autotrim used to enabled by default, but then ZoL-rebase happened and somehow, suddenly, it wasn't. The best part, this is not actually documented anywhere.
I don't know what occurs if, for example, the trim happens when a huge R/O occurs?Could you maybe elaborate on the other reasons? Maybe there's anything I'm not aware of?
No practical usage in this doc.
[sherman.129] # zpool status -t zroot
pool: zroot
state: ONLINE
scan: scrub repaired 0B in 00:04:39 with 0 errors on Thu Jul 28 03:43:26 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/236009L240AGN:p3 ONLINE 0 0 0 (untrimmed)
gpt/410008H400VGN:p3 ONLINE 0 0 0 (untrimmed)
errors: No known data errors
[sherman.130] # zpool trim zroot
[sherman.131] # zpool status -t zroot
pool: zroot
state: ONLINE
scan: scrub repaired 0B in 00:04:39 with 0 errors on Thu Jul 28 03:43:26 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/236009L240AGN:p3 ONLINE 0 0 0 (4% trimmed, started at Mon Aug 8 07:37:50 2022)
gpt/410008H400VGN:p3 ONLINE 0 0 0 (4% trimmed, started at Mon Aug 8 07:37:50 2022)
errors: No known data errors
[sherman.133] # zpool status -t zroot
pool: zroot
state: ONLINE
scan: scrub repaired 0B in 00:04:39 with 0 errors on Thu Jul 28 03:43:26 2022
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/236009L240AGN:p3 ONLINE 0 0 0 (100% trimmed, completed at Mon Aug 8 07:38:21 2022)
gpt/410008H400VGN:p3 ONLINE 0 0 0 (100% trimmed, completed at Mon Aug 8 07:38:18 2022)
errors: No known data errors
You can turn on autotrim at zpool-create(8) or zpool-import(8) time, or at any time with zpool-set(8).
zpool set autotrim=on zroot
works, and it also works after rebooting.I had one server with 28% ZFS fragmentation. The server has NVME disks in RAID-1. I manually runIf your over-provisioning is sufficient, there won't ever be any shortage of pages to pre-zero, and you won't need to trim.
zpool trim zroot
and after it finish it shows 16% fragmentation. The ZFS pool uses 21% capacity. Does manually doing TRIM helps in my case?[sherman.135] # crontab -l | grep trim
5 3 * * * /sbin/zpool trim zroot
After TRIM:NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 3.47T 1.68T 1.79T - - 40% 48% 1.00x ONLINE -
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 3.47T 1.68T 1.79T - - 15% 48% 1.00x ONLINE -
File systems generally TRIM'd when initialised. Without any further TRIMing, free capacity will be known to the SSD controller for as long as it has never been used by the kernel. However, once used by the kernel, they have to be TRIM'd for the SSD controller to ever use the again. So, the situation depends on how active your file system is, and if it ever fills up.So the free capacity doesn't play any role even if the capacity (usage) is low because after a lot of writes & deletes I have to use TRIM to make the SSD controller "know" the pages free list (where it can immediately write and not to move first the data and then write).
You will always have the over-provisioning provided by the manufacturer. It's generally meant to be sufficient for the intended market of the product (i.e. should maintain write speed). Generally "better" SSDs have more over-provisioning. The definition of sufficient depends on your duty cycle. Wear leveling is complex, and I don't think I want to speculate on the specifics.If the SSD controller doesn't have many pages in free list, then the writes will be slower and the disk wear will be higher, right? Are these the only disadvantages if I don't TRIM?
Your empirical observations are quite convincing. Fragmentation of free space can be viewed from the perspective of the kernel, or the perspective of the SSD controller. The SSD controllers have a mountain of code, implementing algorithms that I don't understand in detail. So, once again, I don't know enough to be sure of my ground.I believe the "ZFS Fragmentation" is related to TRIM: