ZFS Zfs Tuning

mayers

mayers

I'm trying to follow this guide to do zfs optimization.

docs.freebsd.org

Chapter 22. The Z File System (ZFS)

ZFS is an advanced file system designed to solve major problems found in previous storage subsystem software
docs.freebsd.org docs.freebsd.org

However, these parameters are found to be non-existent when checked with sysctl

vfs.zfs.arc_meta_limit

vfs.zfs.prefetch_disable

Furthermore, it is not clear to me what these 2 parameters set

vm.kmem_size="330M"
vm.kmem_size_max="330M"

Thanks
 

261212 – Update the ZFS chapter of the FreeBSD Handbook, and other OpenZFS-related pages

bugs.freebsd.org bugs.freebsd.org

Assume that much of the chapter is outdated. Coverage of the current edition of the FreeBSD Handbook includes FreeBSD 12.4, which preceded OpenZFS in FreeBSD base.

These commands should help to set you in the right direction:

sysctl vfs.zfs.arc

sysctl vfs.zfs.prefetch

sysctl vfs.zfs

In addition to <https://wiki.freebsd.org/Myths#ZFS_will_use_too_much_memory>, <https://www.freebsd.org/releases/14.0R/relnotes/#storage-zfs> notes:
  • fully adaptive ARC, a unified ARC that minimizes the need for manual tuning
Click to expand...
 
I subscribe to this point of view.
My /etc/sysctl.conf has as few lines as possible.
Code: 
vfs.zfs.arc_max= 10000000000              
vfs.zfs.arc_min=  4000000000             
vfs.zfs.min_auto_ashift=12               
vfs.zfs.txg.timeout=5
 
SirDice has very sage advice. By design ZFS may use a lot of free memory, but note that it "may". A lot of read operations of different files (think something at NetFlix serving streaming video to a lot of clients) would probably try to use everything it can because "RAM is faster than disk".
The two vfs.zfs.arc things Alain De Vos shows are about all you need to limit memory usage.
The min_auto_ashift relates to blocksize stuff (12 I think means 4K blocks), I think the 12 is the current default, it used to be 9.
txg.timeout relates to how often writes are done, I think 5 is the current default. txg is "transaction group" and represents write operations.

People using a machine as a typical desktop will usually set the arc_max to less than the default. That can help if you open and close a lot of different applications.
 
mer said:
People using a machine as a typical desktop will usually set the arc_max to less than the default. That can help if you open and close a lot of different applications.
Click to expand...

Desktop environment use since around 2015. I never needed the setting. What's the supposed benefit?
 
grahamperrin said:
Desktop environment use since around 2015. I never needed the setting. What's the supposed benefit?
Click to expand...
Going by memory, arc_max defaults to "all of ram minus a couple GB for the kernel".
If the use pattern causes ARC to max out (lots of reads of different files because ARC is effectively read-cache) and then the user start firefox, you have a situtation where firefox wants memory, but there isn't enough available. The good thing about ZFS ARC is ZFS gives it up. Problem is it takes time. That means user starts an application, not enough memory, system reclaims, ZFS gives it up, but the start of the application is slowed down and a bad user experience.

It's tightly coupled to specific use patterns, you may never hit it. The package zfs-stats is very useful, tells you how much memory you are actually using, cache hit ratios.
I have a desktop with 16GB total, I limit the ARC Max to 4GB, zfs-stats shows my total ARC usage is around 50M, so my limit is never even close to being used.
 
In the past I have limited ARC, too. My current laptop has 8GB RAM. Weeks ago I compiled world and kernel without limiting ARC. I have simply forgotten to do so. But everything went smooth and well. The few times I have checked the process even swap has not been touched.

Actually I run X with awesome, firefox and one terminal. According to top ARC uses 440MB, but why not? It is available. But I am not a "power user" or so with a huge desktop environment, a browser with lots of tabs open and additional fancy stuff running in the background.

At least for this laptop there is no need to limit anything. I am not sure it is due to the amount RAM I have here compared to older systems. The step to OpenZFS could have caused some progress, too.
 
chrbr Thanks, perfect example.
First step of tuning anything is "don't tune, collect data"
Second step is "what is this data telling me"

Tuning anything means you have to define the workload you are tuning for.
Massive SQL database with millions of accountants updating client data is vastly different from lots of users streaming videos from a server.

I would guess that most (90%) of people that "use" a computer daily have a similar workload to you. Browsers, text editors open, email, periodic online meetings, maybe light photo/video editing. Default values of the system are probably good enough for them. If you start to think about need to tune, you probably know enough about what to do and what to not do.

The "average system" of today is a lot different than the "average system" of 10, 15 yrs ago. By default more RAM, bigger storage devices, faster/more powerful CPUs so rules for tuning from 10 yrs ago are likely obsolete for today.

My opinions only.
 
on Discord, I have seen claims that ZFS can be used in systems with as little as 2 GB RAM (like the rpi)... But... I did have a laptop with 8 GB of RAM and ZFS, and it ran KDE. It actually compiled stuff OK, but compilations did take a long time and graphics sometimes crashed - hard. Now I have a laptop with 40 GB RAM and a Ryzen 7 5825U - and it's MUCH better now.
 
astyle said:
on Discord, I have seen claims that ZFS can be used in systems with as little as 2 GB RAM (like the rpi)... But... I did have a laptop with 8 GB of RAM and ZFS, and it ran KDE. It actually compiled stuff OK, but compilations did take a long time and graphics sometimes crashed - hard. Now I have a laptop with 40 GB RAM and a Ryzen 7 5825U - and it's MUCH better now.
Click to expand...
I used ZFS on a 768 MB i386 laptop. It still has FreeBSD 13-CURRENT or 14-CURRENT installed on it. It worked but I had to seriously tune it. You can see some of my notes at wiki.freebsd.org. Though, many of the tuning recommendations have changed since then, IIRC my 768 MB laptop tuning recommendations are still there.

Though, I would recommend people put as much RAM into the machine as possible. The ARC gets pretty hungry and with adequately large RAM, depending on your access patterns, you can get hit ratios of greater than 97%. This will speed up the machine significantly.

With ZFS the more RAM the better. Even without ZFS, don't cheap out on RAM.

Personally, I'd get a slower CPU and more RAM. You will get more bang for your buck from RAM than you will money spent on CPU.
 
The pfSense stuff default install is ZFS; the appliances sold by Netgate range from 1GB to 16GB. As these are firewall devices, the use pattern of the storage is mostly write, I don't think there is much if any tuning for ZFS.

cy@ said:
Personally, I'd get a slower CPU and more RAM. You will get more bang for your buck from RAM than you will money spent on CPU.
Click to expand...
Completely agree with this. It's always been my philosophy on building a system and it often winds up costing a little bit less.
 
mer said:
The pfSense stuff default install is ZFS; the appliances sold by Netgate range from 1GB to 16GB. As these are firewall devices, the use pattern of the storage is mostly write, I don't think there is much if any tuning for ZFS.


Completely agree with this. It's always been my philosophy on building a system and it often winds up costing a little bit less.
Click to expand...
... and performing much better.
 
  • Like
Reactions: mer
Based on what I've read. First of all I bought some new material. So 2 128Gb SSD disks, one for the operating system and the other for the L2ARC raid cache. 3 3TB disks with which to raid ZFS. The RAM is currently 4GB but I intend to upgrade to at least 8GB.

In your opinion, since I have 2 SSDs it is better to use them both as an SSD cache and install the operating system on another disk, for example a 1TB SATA disk that I have available. Thanks
 
I'd install the OS on the two mirrored SSDs. You might not benefit from L2ARC at all but that's going to depend on your workload.
 
mayers said:
Based on what I've read. First of all I bought some new material. So 2 128Gb SSD disks, one for the operating system and the other for the L2ARC raid cache. 3 3TB disks with which to raid ZFS. The RAM is currently 4GB but I intend to upgrade to at least 8GB.

In your opinion, since I have 2 SSDs it is better to use them both as an SSD cache and install the operating system on another disk, for example a 1TB SATA disk that I have available. Thanks
Click to expand...
Just to let you know, most ppl would use SSD for the OS, and spinners for data, not the other way around.
 
mayers perhaps I've missed it, but what is the intended use for the system? Is it a general purpose user desktop/workstation?
Based on what you said, I agree with SirDice and astyle Mirror the two SSDs for the OS, the spinning device use for data or user home directories. I've got a system where I've had /usr/home on a mirror pair of spinning disks, OS on a mirror of SSDs. It lets me upgrade the OS without losing any user data, heck I've even gone and bought new SSDs for the OS and done fresh installs without losing any of my user data. The user data has actually migrated from FreeBSD 9.x to now, mirror size expanded, all kinds of good stuff.
 
The machine must become a data storage [nas] server. They are synced from other data sources to it via rsync. So I thought about making 16GB of RAM, 1 SSD operating system, 1 SSD L2ARC cache and 3 ZFS raid disks with active deduplication. I also add a 1TB disk on which I store snapshots of the raid.
 
astyle said:
on Discord, I have seen claims that ZFS can be used in systems with as little as 2 GB RAM …
Click to expand...

Here, two years ago:

grahamperrin said:
Firefox ESR and LibreOffice with x11-wm/twm in a different machine with ZFS and just 1 GB memory:

buba and kiki.png


Playback of <https://www.ted.com/talks/james_geary_metaphorically_speaking?language=nl>: faultless.
Click to expand...
 
astyle said:
😲 Hey, mer, cy@, take a look! 972 MB of RAM!
Click to expand...
Yep, the thing about ZFS is RAM use is tied to usage patterns. ARC is basically read cache, so if the workload trends towards reads (netflix serving movies) ARC will likely increase towards the limits.
If workload is mostly writes (firewall, standard home user desktop), ARC is not going to increase very much.

Never said ZFS can't be run on memory limited systems, keep saying some workloads wants more RAM.
 
SirDice said:
I'd install the OS on the two mirrored SSDs. You might not benefit from L2ARC at all but that's going to depend on your workload.
Click to expand...

If that is becoming a NAS system, i.e. random access, L2ARC may not be useful. I would also install the OS mirrored (I don't trust those SSD beasts), but only use half of the devices, or so (should suffice). The remainder can then always be added as l2arc (and removed again if detrimental).
 
You must log in or register to reply here.
Back
Top