ZFS vfs.zfs.arc_free_target - affecting performance?

G

garrett

I can't find much about how to approach arc_free_target but I did end up here https://wiki.freebsd.org/Myths#ZFS_will_use_too_much_memory

I am still learning a lot about zfs and ARC. I tuned all of our databases and restructured our datasets so there was no db cache in the arc cache (metadata caching for those).

I am still troubleshooting ARC behaving badly - it bottlenecks everything and peculiar things are showing up.

For example:
- not running out of vnodes, that's fine.
- htop displays ARC: 6.00G Used: 4.90G MFU: 2.65G MRU: 1.25G ...

But ZFS stats is showing MFU as higher by 1G:
Code: 
# zfs-stats -AE

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Feb  5 10:57:59 2026
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                5.47    m
        Mutex Misses:                           1.73    k
        Evict Skips:                            3.81    k

ARC Size:                               81.28%  4.88    GiB
        Target Size: (Adaptive)         81.62%  4.90    GiB
        Min Size (Hard Limit):          8.31%   510.67  MiB
        Max Size (High Water):          12:1    6.00    GiB
        Compressed Data Size:                   2.68    GiB
        Decompressed Data Size:                 7.19    GiB
        Compression Factor:                     2.68

ARC Size Breakdown:
        Recently Used Cache Size:       25.44%  1.25    GiB
        Frequently Used Cache Size:     74.56%  3.65    GiB

ARC Hash Breakdown:
        Elements Max:                           409.53  k
        Elements Current:               100.00% 409.53  k
        Collisions:                             1.38    m
        Chain Max:                              5
        Chains:                                 34.95   k

------------------------------------------------------------------------

ARC Efficiency:                                 119.62  m
        Cache Hit Ratio:                94.14%  112.61  m
        Cache Miss Ratio:               5.86%   7.01    m
        Actual Hit Ratio:               94.12%  112.58  m

        Data Demand Efficiency:         93.88%  9.29    m
        Data Prefetch Efficiency:       99.43%  1.05    k

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           22.29%  25.10   m
          Most Frequently Used:         77.69%  87.48   m
          Most Recently Used Ghost:     0.09%   102.37  k
          Most Frequently Used Ghost:   0.55%   619.80  k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  7.74%   8.72    m
          Prefetch Data:                0.00%   1.04    k
          Demand Metadata:              90.05%  101.40  m
          Prefetch Metadata:            2.21%   2.49    m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  8.10%   567.95  k
          Prefetch Data:                0.00%   6
          Demand Metadata:              73.07%  5.12    m
          Prefetch Metadata:            18.82%  1.32    m

------------------------------------------------------------------------

So it seems in htop like it is keeping it below 5G. If I bump the ARC to 7G with

sysctl vfs.zfs.arc_max=7516192768

then immediately everything is functioning again on the server.

Things will be fine again until it isn't. I can't really figure out what is happening but from reading around it seems that for some reason the cache is no longer evicting? We have no L2ARC. We are combining a few things into one server that are not ideal, like a database, plus much of our primary web app activity deals with smaller image files and the creation/destruction of temp folders and file movements, etc. I realize more resources often helps, but...

-> Is the vfs.zfs.arc_free_target also needing to be changed?
-> Why is the MFU not being displayed properly in htop?
-> Why is the cache total not nearing the ARC max, and why does it fix it when I bump up the ARC max? What is actually happening that makes it better? Why wouldn't the Target Size: (Adaptive) also increase?

Thank you for any ideas and help.
 
Addtional, from top after bumping ARC max to 7G:
Code: 
167 processes: 1 running, 166 sleeping
CPU:  6.7% user,  0.0% nice,  0.4% system,  0.0% interrupt, 92.9% idle
Mem: 1496M Active, 6315M Inact, 4708K Laundry, 7560M Wired, 1162M Buf, 572M Free
ARC: 4718M Total, 2671M MFU, 973M MRU, 2655K Anon, 91M Header, 974M Other
     2446M Compressed, 7019M Uncompressed, 2.87:1 Ratio
Swap: 8192M Total, 8192M Free
 
You must log in or register to reply here.
Back
Top