ZFS SSD Tuning for a Desktop Use

freezr

freezr

Dear All,

looking for some info to tuning and extend the life of my SSD disks I couldn't find any valuable information.

The documentation reports:

vfs.zfs.vdev.trim_on_init - Control whether new devices added to the pool have the TRIM command run on them. This ensures the best performance and longevity for SSDs, but takes extra time. If the device has already been secure erased, disabling this setting will make the addition of the new device faster. Adjust this value at any time with sysctl(8).
Click to expand...
vfs.zfs.l2arc_write_max - Limit the amount of data written to the L2ARC per second. This tunable extends the longevity of SSDs by limiting the amount of data written to the device. Adjust this value at any time with sysctl(8).
Click to expand...
vfs.zfs.l2arc_write_boost - Adds the value of this tunable to vfs.zfs.l2arc_write_max and increases the write speed to the SSD until evicting the first block from the L2ARC. This "Turbo Warmup Phase" reduces the performance loss from an empty L2ARC after a reboot. Adjust this value at any time with sysctl(8).
Click to expand...
L2ARC is the second level of the ZFS caching system. RAM stores the primary ARC. Since the amount of available RAM is often limited, ZFS can also use cache vdevs. Solid State Disks (SSDs) are often used as these cache devices due to their higher speed and lower latency compared to traditional spinning disks. L2ARC is entirely optional, but having one will increase read speeds for cached files on the SSD instead of having to read from the regular disks. L2ARC can also speed up deduplication because a deduplication table (DDT) that does not fit in RAM but does fit in the L2ARC will be much faster than a DDT that must read from disk. Limits on the data rate added to the cache devices prevents prematurely wearing out SSDs with extra writes. Until the cache is full (the first block evicted to make room), writes to the L2ARC limit to the sum of the write limit and the boost limit, and afterwards limit to the write limit. A pair of sysctl(8) values control these rate limits. vfs.zfs.l2arc_write_max controls the number of bytes written to the cache per second, while vfs.zfs.l2arc_write_boost adds to this limit during the "Turbo Warmup Phase" (Write Boost).
Click to expand...

None of these make sense for me.

My knowledge of FreeBSD is still very limited and my knowledge of ZFS is even narrower.
On Linux I can enable fstrim through a cron task or systemd, I can also reduce swappiness and enable some other options on the fstab file, and this would be enough for a desktop use (in Linux).

So far I don't have my computer with me, but if my description is enough, maybe someone can already give me some advices.

I installed FreeBSD with ZFS-ON-ROOT over two 250 GB SSD in stripe mode, I already noticed that there is anything inside my fstab file hence the disks are handled somewhere else by ZFS. Maybe I heard from Allan Jude in a episode of BSDNow that ZFS is smart enough to handle SSD maintenance by itself, but I am not very sure.

I heard and read that you can tune ZFS for particular cases but mine user case is mainly a desktop use: web browsing, emailimg, compiling from time to time, graphic design, sometimes Blender and video editing.

Thanks in advance, I can share more info about my disks later in the evening.

tgl
 
ZFS have his own method of trim
Code: 
zpool trim pool
if you use fstrim, I guess you're doing two trims.
I usually have my swap on rotational disks, since swap isn't something that will be used often, but YMMV.
 
ZFS has a lot of knobs. Perhaps these will get you started:
  1. ZFS 101—Understanding ZFS storage and performance
  2. ZFS for Newbies by Dan Langille
  3. Today's ZFS Michael W Lucas
Important when you start defining your pool layout: click. Depending on the chosen layout configuration, changes and disk additions are limited.

For further in depth and reference the two readable books about using ZFS: FreeBSD Development: Books, Papers, Slides.

At various places on this forum, there are other links but, as mentioned above: your use case matters!

Very short answer on some topics: don't use deduplication. If you have an SSD, or better NVME SSD, or even spinning rust platters; and a decent amount of ram (say > 8GB) for your system and the ZFS ARC then, most likely you don't need a L2ARC; more:
___
Edit: videos #2 & #3 are both for beginning ZFS users, both cover quite a lot (ZFS has a lot to offer). I mainly gave these two so you have a choice; I think that either one will give you a worthwhile introduction of the basics of ZFS.
 
Last edited:
Menelkir said:
ZFS have his own method of trim
Code: 
zpool trim pool
if you use fstrim, I guess you're doing two trims.
I usually have my swap on rotational disks, since swap isn't something that will be used often, but YMMV.
Click to expand...
Is this something to run weekly? I use a swapfile, is it a good recommendation reduce swappiness since I have 16GB or ram?

Thanks!
 
I'd like to clarify that I installed ZFS-ON-ROOT through the automatic wizard provided by the installer, the only configuration I made was to stripe the two disks to have 500GB of storage.
 
tgl said:
I'd like to clarify that I installed ZFS-ON-ROOT through the automatic wizard provided by the installer, the only configuration I made was to stripe the two disks to have 500GB of storage.
Click to expand...

I missed that, if you indeed have a stripe over the two 250GB disks and not have them in a mirror then, I strongly advise you to reconsider now: you have no redundancy. If there is an error on one disk you lose all your data!

Look into the layout configuration options and their redundancy. It is like a traditional RAID0 layout and with that one time referring to traditional RAID, I immediately say that, when thinking about ZFS, you have to let go of thinking in traditional RAID terminology. If you want a quick start-over with two 250GB SSDs: make a mirror (and have a look at the article or the ZFS youtube videos :))
 
  • Like
Reactions: mer
I'll second what Erichans says here. If we're understanding how you have it configured (the output of zpool status will tell us exactly) any problem in either device will result in a non bootable system. Mirrors are good and yes you only have "disk size" space.

As for your very original post, the defaults work very nicely for a desktop system. Don't worry about L2ARC (it's a secondary cache and can make a difference on a server with specific workloads), trim the defaults are fine.
 
tgl said:
Is this something to run weekly? I use a swapfile, is it a good recommendation reduce swappiness since I have 16GB or ram?

Thanks!
Click to expand...
Don't use a swapfile at all. Use a swap partition, even if it's a 500Mb partition, it will be better than a swapfile.
I have 16Gb of ram too. I have two HDD (that's a pool called zdata, my zroot is on single SSD), and two swap partitions of 2G on each HDD.
 
Just adding: A swap file on ZFS can even be dangerous (cause deadlocks) because ZFS needs to dynamically allocate memory itself. So, under heavy memory pressure, writing this one inactive page to swap that's so desperately needed to free a page of physical RAM could fail because ZFS can't get the RAM it needs to do this...

Yes, always use swap partitions.
 
Code: 
 $ zfs list
NAME                   USED  AVAIL     REFER  MOUNTPOINT
zroot                  199G   241G       96K  /zroot
zroot/ROOT            4.78G   241G       96K  none
zroot/ROOT/default    4.78G   241G     4.16G  /
zroot/ROOT/pre220107     8K   241G     3.87G  /
zroot/tmp              172K   241G      172K  /tmp
zroot/usr              194G   241G       96K  /usr
zroot/usr/home         193G   241G      193G  /usr/home
zroot/usr/ports        786M   241G      786M  /usr/ports
zroot/usr/src          702M   241G      702M  /usr/src
zroot/var             1.10M   241G       96K  /var
zroot/var/audit         96K   241G       96K  /var/audit
zroot/var/crash         96K   241G       96K  /var/crash
zroot/var/log          580K   241G      580K  /var/log
zroot/var/mail         160K   241G      160K  /var/mail
zroot/var/tmp           96K   241G       96K  /var/tmp

Code: 
$geom disk list
Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 240057409536 (224G)
   Sectorsize: 512
   Mode: r2w2e4
   descr: KINGSTON SA400S37240G
   lunid: 50026b7682359175
   ident: 50026B7682359175
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: Samsung SSD 860 EVO 250GB
   lunid: 5002538e40f0f259
   ident: S3YHNX0M415404N
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

I am not concerned using striped disks, I have been always using it without issues (lvm/ext4, btrfs) so far, I understand the risks but for me is fine.

I heard that Allan Jude suggested to use a physical partition for ZFS, however I allocated 4GB for the swap file and I hope this shouldn't represent a risk if I also reduce the access to swap memory. I may try to reduce the Linux partition to recover 4GB if really having a swap file is considered a danger. 🤷‍♂️

By the way the latter topic isn't specified anywhere in the handbook, I personally consider the swap partition something old and space wasted, however I found a suggestion this very forum that could help me out:

Code: 
# Prefer losing cache over swapping
vm.defer_swapspace_pageouts=1

forums.FreeBSD.org

ZFS - SWAP partition or swap file?

Greetings all, I am planning new server installation. The operating system will reside on SSDs (mirrored) and the data on mechanical hard drives. I am wondering whether to use SWAP partition or swap file. As best as I understand, a SWAP partition needs to be mirrored and only the allocated...
forums.FreeBSD.org forums.FreeBSD.org
 
mefizto said:
… What would be the best practice for a stable system? …
Click to expand...

lkateley said:
… Maybe someone who knows more about what might be swapping/paging on a freebsdFreeBSD system will have more insight, …
Click to expand...

tgl said:
… I heard that Allan Jude suggested to use a physical partition for ZFS, however I allocated 4GB for the swap file and I hope this shouldn't represent a risk if I also reduce the access to swap memory. …
Click to expand...

mark_j said:
… this bug.

In reality, swap on ZFS is no more problematic than mounting anything else. …
Click to expand...

grahamperrin said:
github.com

Swap deadlock in 0.7.9 · Issue #7734 · openzfs/zfs

System information Type Version/Name Distribution Name Debian Distribution Version stretch Linux Kernel 4.16.0-0.bpo.2-amd64 Architecture amd64 ZFS Version 0.7.9-3~bpo9+1 SPL Version 0.7.9-3~bpo9+1...
github.com github.com
Click to expand...

2021-10-10:

… I found that the issue is still not solved.
Click to expand...
 
You can leave swap management for pkgs like swapd and swapmon in freshports.
You could alternatively use zvol for swap and adjust its size over time according to your need.
 
If you plan to do some tasks which require a lot of memory and data read from disc, like for example playing in some AAA modern games (from my experience, yes it is possible on FreeBSD), I would recommend to set maximum memory size for ZFS cache, ARC. I was a few times in a situation, when a game crashed due to lack of RAM, caused by the default setting of ZFS to use 5/8 of all RAM. Most guides recommend setting it 1 GB RAM per 1 TB of disk. In my case, setting it to double of this value not only prevented crashes mentioned above, but I also noticed that the whole system started works a bit faster. Mostly because it stopped trying to load to ARC all these games' data and more often hit the cache. But, as they say, YMMV.
 
Lamia said:
You can leave swap management for pkgs like swapd and swapmon in freshports.
You could alternatively use zvol for swap and adjust its size over time according to your need.
Click to expand...
Beware where that zvol is located. It will be completely duplicated for each snapshot you do. So anything that does snapshots regulary might become a disk space hog.
 
Lamia said:
alternatively use zvol for swap
Click to expand...

Ignoring the risk?

I mean, it might be deadlock-free for many people most of the time, however: with users (of OpenZFS) still reporting deadlocks, I shouldn't recommend it (as an alternative) to someone who's finding their feet.

From <https://openzfs.github.io/openzfs-docs/man/8/zfs-create.8.html#ZFS_Volumes_as_Swap>:

… ZFS volumes may be used as swap devices. …
Click to expand...

– however I don't expect manual pages such as this to list bugs such as OpenZFS issue 7734.
 
All of these depend on the workload, the default swap size "2gb" on fresh installation most times suffice. For a desktop with spinning rust, get an SSD into the SSD cache SATA slot on the motherboard (some boards have it) and use it for layer 2 ARC, you will notice a performance boost.

The need for cache (and log) comes at a time when the machine has limited resources (RAM mostly) yet gets heavy workload thrown at it.
 
The documentation about making a swap on a zvol is really scarce for FreeBSD:

FAQ — OpenZFS documentation

openzfs.github.io openzfs.github.io

I think I had better to shrink the Linux partition and recover 4 or at least 2GB... 🤔

By the way this has been deprecated in FreeBSD 11:

Code: 
# Prefer losing cache over swapping
vm.defer_swapspace_pageouts=1
 
tgl said:
deprecated
Click to expand...

Thanks, I removed the quote from my earlier comment.

Confirming that there's no mention in FreeBSD documentation:

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # gh repo sync grahamperrin/freebsd-doc && cd /usr/doc && gh repo sync && git -C /usr/ports pull --ff-only && git -C /usr/src pull --ff-only && cd
✓ Synced the "grahamperrin:main" branch from "freebsd:main"
✓ Synced the "main" branch from grahamperrin/freebsd-doc to local repository
Already up to date.
Already up to date.
root@mowa219-gjp4-8570p-freebsd:~ # logout
% time grep -r vm.defer_swapspace_pageouts /usr/doc/documentation/
0.137u 0.420s 0:10.66 5.1%      26+180k 4406+0io 0pf+0w
%
 
There are two ZFS tunables I know that may be useful for FreeBSD on the Desktop.



1. Limit the number of TRIM instructions sent on burst to device.

By default 64 TRIM requests are sent to SSD device. I once had SSD device that took 500ms (half a second) for each TRIM operation) so on the default 64 TRIM instructions the system almost frozen when I deleted large files. I set that to 1 and system was smooth as usual. The only downside is that TRIM instructions will be issued later, but that does not change anything really.

/etc/sysctl.conf

vfs.zfs.vdev.trim_max_active=1



2. Really force small ARC.

... and by really I mean REALLY. The vfs.zfs.arc.min and vfs.zfs.arc.max are not always respected ... but there is also vfs.zfs.arc_free_target which can be used to REALLY force ARC to be small. The tricky part is that you need to find your value.

For 32 GB of RAM its 1024 x 1024 x 7.75 which means 8126464 value.

For 16 GB RAM its 1024 x 1024 x 3.85 which means 4037017 value.

Example setting in /etc/sysctl.conf for 16 GB RAM:

vfs.zfs.arc_free_target=4037017

Keep in mind that these are 'maximum' values which mean 'smallest' ARC size.



Other things.

Disable prefetch.

Disable ATIME with atime=off for entire ZFS pool.



Hope that helps.
 
You must log in or register to reply here.
Back
Top