Solved vfs.zfs.arc_free_target - affecting performance?

G

garrett

I can't find much about how to approach arc_free_target but I did end up here https://wiki.freebsd.org/Myths#ZFS_will_use_too_much_memory

I am still learning a lot about zfs and ARC. I tuned all of our databases and restructured our datasets so there was no db cache in the arc cache (metadata caching for those).

I am still troubleshooting ARC behaving badly - it bottlenecks everything and peculiar things are showing up.

For example:
- not running out of vnodes, that's fine.
- htop displays ARC: 6.00G Used: 4.90G MFU: 2.65G MRU: 1.25G ...

But ZFS stats is showing MFU as higher by 1G:
Code: 
# zfs-stats -AE

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Feb  5 10:57:59 2026
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                5.47    m
        Mutex Misses:                           1.73    k
        Evict Skips:                            3.81    k

ARC Size:                               81.28%  4.88    GiB
        Target Size: (Adaptive)         81.62%  4.90    GiB
        Min Size (Hard Limit):          8.31%   510.67  MiB
        Max Size (High Water):          12:1    6.00    GiB
        Compressed Data Size:                   2.68    GiB
        Decompressed Data Size:                 7.19    GiB
        Compression Factor:                     2.68

ARC Size Breakdown:
        Recently Used Cache Size:       25.44%  1.25    GiB
        Frequently Used Cache Size:     74.56%  3.65    GiB

ARC Hash Breakdown:
        Elements Max:                           409.53  k
        Elements Current:               100.00% 409.53  k
        Collisions:                             1.38    m
        Chain Max:                              5
        Chains:                                 34.95   k

------------------------------------------------------------------------

ARC Efficiency:                                 119.62  m
        Cache Hit Ratio:                94.14%  112.61  m
        Cache Miss Ratio:               5.86%   7.01    m
        Actual Hit Ratio:               94.12%  112.58  m

        Data Demand Efficiency:         93.88%  9.29    m
        Data Prefetch Efficiency:       99.43%  1.05    k

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           22.29%  25.10   m
          Most Frequently Used:         77.69%  87.48   m
          Most Recently Used Ghost:     0.09%   102.37  k
          Most Frequently Used Ghost:   0.55%   619.80  k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  7.74%   8.72    m
          Prefetch Data:                0.00%   1.04    k
          Demand Metadata:              90.05%  101.40  m
          Prefetch Metadata:            2.21%   2.49    m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  8.10%   567.95  k
          Prefetch Data:                0.00%   6
          Demand Metadata:              73.07%  5.12    m
          Prefetch Metadata:            18.82%  1.32    m

------------------------------------------------------------------------

So it seems in htop like it is keeping it below 5G. If I bump the ARC to 7G with

sysctl vfs.zfs.arc_max=7516192768

then immediately everything is functioning again on the server.

Things will be fine again until it isn't. I can't really figure out what is happening but from reading around it seems that for some reason the cache is no longer evicting? We have no L2ARC. We are combining a few things into one server that are not ideal, like a database, plus much of our primary web app activity deals with smaller image files and the creation/destruction of temp folders and file movements, etc. I realize more resources often helps, but...

-> Is the vfs.zfs.arc_free_target also needing to be changed?
-> Why is the MFU not being displayed properly in htop?
-> Why is the cache total not nearing the ARC max, and why does it fix it when I bump up the ARC max? What is actually happening that makes it better? Why wouldn't the Target Size: (Adaptive) also increase?

Thank you for any ideas and help.
 
Addtional, from top after bumping ARC max to 7G:
Code: 
167 processes: 1 running, 166 sleeping
CPU:  6.7% user,  0.0% nice,  0.4% system,  0.0% interrupt, 92.9% idle
Mem: 1496M Active, 6315M Inact, 4708K Laundry, 7560M Wired, 1162M Buf, 572M Free
ARC: 4718M Total, 2671M MFU, 973M MRU, 2655K Anon, 91M Header, 974M Other
     2446M Compressed, 7019M Uncompressed, 2.87:1 Ratio
Swap: 8192M Total, 8192M Free
 
cracauer@ said:
ZFS ARC is not an exact science. At least to me.
Click to expand...
Me neither! In the spirit of scientific discovery in inexact sciences, I will note here what has resolved the issue for me.

First, more cache was needed. I set sysctl vfs.zfs.arc.max to 8GB. This permitted things like nightly reports to actually complete, though it would take 4 hours. Not ideal, but I identified this as the high end. More cache delayed the ultimate result a bit longer, and as expected at some point things locked up again. While viewing zfs-stats it became clear that the target size (Adaptive) was just too small. I watched it get smaller and smaller, when the cache need more, not less.

Secondly, I set sysctl vfs.zfs.arc.min to 5GB. This was the real game changer for me. I was now seeing evictions of much larger amounts of cache. I performed testing of running find over and over, and I could not get it to lock up. Nightly reports are done in 10 minutes.

Current non-default settings on system with 16GB RAM:
Code: 
vfs.zfs.arc.max=8589934592 (8GB)
vfs.zfs.arc.min=5368709120 (5GB)

I still don't understand exactly why it helps, because I believe the memory space was earmarked for the ARC cache anyway... but, for whatever reason it would not use it when it mattered based on some aspect of the default algorithm. The overall RAM usage is not higher than it was before. htop ARC stats now shows more or less what it did, but the minimum is set to 5G now:

ARC: 8.00G Used: 5.00G MFU: 2.14G MRU: 1.49G

Also, excellent resource that helped me a lot to understand what ARC is trying to achieve, in case other get here and want to read/watch some details for understanding: https://papers.freebsd.org/2019/fosdem/jude-eli5_zfs_caching/
 
You must log in or register to reply here.
Back
Top