HOWTO: Tuning L2ARC in ZFS

General suggestions

1. Design the file system intelligently. Group frequently used files together.

2. Get enough cache capacity so that the working set fit into the cache.

3. Set secondarycache=all only for frequently used data. For example:

# zfs set secondarycache=all data/frequent-used-data data

Sysctl tunables

The following are all the sysctl tunables involving L2ARC and their defaults.

Code:
sysctl -a | grep l2arc

vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_norw: 1

I will focus on the following 3 tunables.

1. vfz.zfs.l2arc_write_max

This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives.

2. vfs.zfs.l2arc_write_boost

This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system.

3. vfs.zfs.l2arc_noprefetch

This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%.

The following is what I had added to /etc/sysctl.conf.

Code:
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_write_max=26214400
vfs.zfs.l2arc_write_boost=52428800

The following is the iostat from my system when I write this howto:

Code:
iostat -xz -w 10 -c 10
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.5   2.7   585.1    73.2    0  18.1   3
da1        9.5   2.7   583.7    73.3    0  16.9   3
da3        9.4   2.7   584.0    73.3    0  17.3   3
da4        9.5   2.8   583.1    73.2    0  16.7   3
ada0      19.9   1.5  2375.9   157.5    0   1.5   1
ada1      19.4   1.6  2360.4   172.9    0   2.1   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.7   0.0   142.4     0.0    0  14.5   1
da1        2.3   0.0   104.6     0.0    0   5.1   0
da3        3.3   0.0   181.1     0.0    0  16.1   1
da4        2.4   0.0   129.2     0.0    0  10.7   1
ada0      28.9   0.2  3539.3    25.6    0   1.3   1
ada1      30.2   0.3  3506.9    38.4    0   1.5   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.8   0.0   115.2     0.0    0   8.2   1
da1        2.2   0.0   140.8     0.0    0  10.6   0
da3        1.8   0.0   115.2     0.0    0  11.5   0
da4        1.0   0.0    64.0     0.0    0   7.8   0
ada0      15.0   0.1  1841.4    12.8    0   1.4   1
ada1      16.7   1.2  2003.0   153.6    0   2.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.1   0.0    64.6     0.0    0   8.9   0
da1        1.4   0.0    89.6     0.0    0   9.8   1
da3        1.2   0.0    76.8     0.0    0  13.7   1
da4        1.8   0.0   115.2     0.0    0  14.4   1
ada0      14.3   0.1  1763.0    12.8    0   0.9   1
ada1      12.0   0.4  1311.9    51.2    0   0.9   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.6   5.5   169.8    29.8    0  10.2   2
da1        2.8   3.9   105.7    11.6    0   7.0   1
da3        2.6   3.9   154.2    11.6    0   5.5   1
da4        1.7   5.5    78.3    29.8    0   4.9   1
ada0      12.2   0.2  1368.7    14.4    0   1.0   0
ada1       9.2   0.9   874.0    65.6    0   0.9   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        5.2   0.0   297.2     0.0    0  13.7   2
da1        4.1   0.0   219.8     0.0    0  10.3   1
da3        4.0   0.0   243.8     0.0    0  23.5   2
da4        5.9   0.0   347.1     0.0    0  15.7   2
ada0      10.8   0.5  1279.1    52.8    0   1.0   0
ada1      13.0   0.1  1384.3     1.6    0   1.3   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.0   0.0    79.4     0.0    0  17.8   1
da1        2.3   0.0   104.5     0.0    0  20.0   1
da3        3.4   0.0   193.2     0.0    0  11.3   1
da4        3.5   0.0   205.7     0.0    0  18.8   2
ada0      18.2   0.1  2237.4    12.8    0   1.2   1
ada1      16.7   0.2  1801.8    25.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        4.8   5.9   259.5    27.8    0   6.5   2
da1        5.3   4.1   283.8    12.3    0  13.7   3
da3        3.8   4.1   243.2    12.3    0   7.7   1
da4        5.5   5.9   310.5    27.8    0   8.6   2
ada0      16.1   0.3  1912.6    27.2    0   1.3   1
ada1      14.0   0.4  1457.5    17.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.4   5.0   540.5    16.0    0  10.9   3
da1        8.6   3.5   513.9     8.7    0  13.1   3
da3        8.8   3.5   526.4     8.7    0  16.0   4
da4        9.9   5.0   579.0    16.0    0  14.1   3
ada0      30.1   1.0  3703.2   116.8    0   1.6   1
ada1      33.0   1.2  3921.3   108.8    0   1.6   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.1   0.0   155.8     0.0    0   7.1   1
da1        3.5   0.0   205.7     0.0    0  13.5   1
da3        2.9   0.0   179.5     0.0    0  10.0   1
da4        3.5   0.0   193.5     0.0    0   8.3   1
ada0      12.8   0.7  1547.4    89.6    0   1.1   1
ada1      11.8   0.2  1301.5     3.2    0   1.4   1

da0, da1, da3 and da4 formed a ZFS pool, while ada0, ada1 are SSDs.

Reference

ZFS L2ARC (Brendan Gregg)
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
 
According to the wiki, you have these set incorrectly; /etc/sysctl.conf is not honored by ZFS:
Note that this value is a run time sysctl that is read at pool import time, which means the default mechanism of setting sysctls in FreeBSD will not work. (Values set in /etc/sysctl.conf are set after ZFS pools are imported)

I believe these tunables should be moved to /boot/loader.conf
 
Update information on sysctl tunables and add some examples

The following are the sysctl tunables and their defaults for L2ARC. I have add a short description at the end of each tunables.

Code:
vfs.zfs.l2arc_write_max: 8388608    # Maximum number of bytes written to l2arc per feed
vfs.zfs.l2arc_write_boost: 8388608  # Mostly only relevant at the first few hours after boot
vfs.zfs.l2arc_headroom: 2           # Not sure
vfs.zfs.l2arc_feed_secs: 1          # l2arc feeding period
vfs.zfs.l2arc_feed_min_ms: 200      # minimum l2arc feeding period
vfs.zfs.l2arc_noprefetch: 1         # control whether streaming data is cached or not
vfs.zfs.l2arc_feed_again: 1         # control whether feed_min_ms is used or not
vfs.zfs.l2arc_norw: 1               # no read and write at the same time

As mention in the above post, changes to vfs.zfs.l2arc_write_max will not affect the behaviour of a running pool. For best compatibility, it should be set in /boot/loader.conf.

The other interesting tunables, vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again do not have such restrictions. You can set them in the command line, and it will immediately take effect.

If vfs.zfs.l2arc_feed_again=1, then when the data that are going to written onto L2ARC is larger than vfs.zfs.l2arc_write_max, the feeding period of L2ARC is reduced, up to vfs.zfs.l2arc_feed_min_ms. To illustrate how it works, here are three ways to set maximum write speed on L2ARC to 80MB/s

Code:
vfs.zfs.l2arc_write_max=83886080
vfs.zfs.l2arc_feed_again=0
# feed 80MB per second

or

Code:
vfs.zfs.l2arc_feed_min_ms=100
# feed 8MB per 100 ms

or

Code:
vfs.zfs.l2arc_write_max=16777216
# feed 16MG per 200 ms

When the L2ARC consists only one drive, the difference is not much, but if the L2ARC consists 2 drives, cache1 and cache2, then in the above example, when L2ARC is writing at its maximum speed, we will see

Code:
80MB written to cache1 in the first second
80MB written to cache2 in the next second

or

Code:
8MB written to cache1 in the first 100ms,
8MB written to cache2 in the next 100ms, etc.

or

Code:
16MB written to cache1 in the first 200ms,
16MB written to cache2 in the next 200ms, etc.
 
I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.

That would only be needed if they can not be adjusted without a reboot.
 
Actually, you may adjust vfs.zfs.l2arc_write_max on the command line also. But it will not take effect until you export and then import back your pool. If the data pool is not separately from the root, this have to set on /boot/loader.conf instead of /etc/sysctl.conf. And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.

However, you can change vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again on the command line and it will immediately change the behviour of the pool without a need for export/import.

vfs.zfs.l2arc_feed_secs and vfs.zfs.l2arc_norw seem to have reasonable defaults.
 
I believe these tunables should be moved to /boot/loader.conf
I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.
And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.
I have placed those modified vfs.zfs.l2arc_write_max and vfs.zfs.l2arc_write_boost variables to /boot/loader.conf, but they have default values after system reboot. What's wrong?
My system version is 10.2.
 
I have placed those modified vfs.zfs.l2arc_write_max and vfs.zfs.l2arc_write_boost variables to /boot/loader.conf, but they have default values after system reboot. What's wrong?
My system version is 10.2.
I have similar problem.

Also this setting do not work.
sysctl vfs.zfs.l2arc_noprefetch="0"
 
If you can change the values in the command line, they should be to put in /etc/sysctl.conf.
 
… this setting do not work.

sysctl vfs.zfs.l2arc_noprefetch="0"

I set it for myself around an hour ago (without using quotation marks),

sysctl vfs.zfs.l2arc.noprefetch=0

Also set in /etc/sysctl.conf.

How can I tell whether the setting is effective?

Partly related:

 
Thanks, no observable downside yet. I'll observe things over the next few days or weeks.

1637646099027.png 1637646190874.png 1637646325000.png

– second and third shots taken after using the File menu of Recoll deskutils/recoll to update its index. (I also run the daemon for real-time indexing of changed files.)

Cache devices for ada0 are currently at da0 and da1.

Code:
% zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Tue Nov 23 05:42:34 2021
------------------------------------------------------------------------

L2 ARC Summary: (DEGRADED)
        Low Memory Aborts:                      1.97    k
        Free on Write:                          14.70   k
        R/W Clashes:                            20
        Bad Checksums:                          185.12  k
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         57.91   GiB
        Decompressed Data Size:                 137.19  GiB
        Compression Factor:                     2.37
        Header Size:                    0.13%   184.88  MiB

L2 ARC Evicts:
        Lock Retries:                           8
        Upon Reading:                           0

L2 ARC Breakdown:                               3.82    m
        Hit Ratio:                      65.29%  2.50    m
        Miss Ratio:                     34.71%  1.33    m
        Feeds:                                  85.53   k

L2 ARC Writes:
        Writes Sent:                    100.00% 17.05   k

------------------------------------------------------------------------
 
The change in sysctl should take effect immediately.

Since each case is different, the best way to determine whether a setting is good or bad is to try it out.
 
From zfs.4 — OpenZFS documentation:



l2arc_noprefetch=1|0 (int)

Do not write buffers to L2ARC if they were prefetched but not used by applications. In case there are prefetched buffers in L2ARC and this option is later set, we do not read the prefetched buffers from L2ARC. Unsetting this option is useful for caching sequential reads from the disks to L2ARC and serve those reads from L2ARC later on. This may be beneficial in case the L2ARC device is significantly faster in sequential reads than the disks of the pool.

Use 1 to disable and 0 to enable caching/reading prefetches to/from L2ARC.

Postscript

After four weeks of caching/reading prefetches to/from L2ARC:
– so I have no plans to revert from 0 to (the default) 1.
 
Last edited:
Back
Top