• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

HOWTO: Tuning L2ARC in ZFS

t1066

Active Member

Thanks: 82
Messages: 218

#1
General suggestions

1. Design the file system intelligently. Group frequently used files together.

2. Get enough cache capacity so that the working set fit into the cache.

3. Set secondarycache=all only for frequently used data. For example:

# zfs set secondarycache=all data/frequent-used-data data

Sysctl tunables

The following are all the sysctl tunables involving L2ARC and their defaults.

Code:
sysctl -a | grep l2arc

vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_norw: 1
I will focus on the following 3 tunables.

1. vfz.zfs.l2arc_write_max

This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives.

2. vfs.zfs.l2arc_write_boost

This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system.

3. vfs.zfs.l2arc_noprefetch

This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%.

The following is what I had added to /etc/sysctl.conf.

Code:
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_write_max=26214400
vfs.zfs.l2arc_write_boost=52428800
The following is the iostat from my system when I write this howto:

Code:
iostat -xz -w 10 -c 10
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.5   2.7   585.1    73.2    0  18.1   3
da1        9.5   2.7   583.7    73.3    0  16.9   3
da3        9.4   2.7   584.0    73.3    0  17.3   3
da4        9.5   2.8   583.1    73.2    0  16.7   3
ada0      19.9   1.5  2375.9   157.5    0   1.5   1
ada1      19.4   1.6  2360.4   172.9    0   2.1   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.7   0.0   142.4     0.0    0  14.5   1
da1        2.3   0.0   104.6     0.0    0   5.1   0
da3        3.3   0.0   181.1     0.0    0  16.1   1
da4        2.4   0.0   129.2     0.0    0  10.7   1
ada0      28.9   0.2  3539.3    25.6    0   1.3   1
ada1      30.2   0.3  3506.9    38.4    0   1.5   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.8   0.0   115.2     0.0    0   8.2   1
da1        2.2   0.0   140.8     0.0    0  10.6   0
da3        1.8   0.0   115.2     0.0    0  11.5   0
da4        1.0   0.0    64.0     0.0    0   7.8   0
ada0      15.0   0.1  1841.4    12.8    0   1.4   1
ada1      16.7   1.2  2003.0   153.6    0   2.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.1   0.0    64.6     0.0    0   8.9   0
da1        1.4   0.0    89.6     0.0    0   9.8   1
da3        1.2   0.0    76.8     0.0    0  13.7   1
da4        1.8   0.0   115.2     0.0    0  14.4   1
ada0      14.3   0.1  1763.0    12.8    0   0.9   1
ada1      12.0   0.4  1311.9    51.2    0   0.9   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.6   5.5   169.8    29.8    0  10.2   2
da1        2.8   3.9   105.7    11.6    0   7.0   1
da3        2.6   3.9   154.2    11.6    0   5.5   1
da4        1.7   5.5    78.3    29.8    0   4.9   1
ada0      12.2   0.2  1368.7    14.4    0   1.0   0
ada1       9.2   0.9   874.0    65.6    0   0.9   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        5.2   0.0   297.2     0.0    0  13.7   2
da1        4.1   0.0   219.8     0.0    0  10.3   1
da3        4.0   0.0   243.8     0.0    0  23.5   2
da4        5.9   0.0   347.1     0.0    0  15.7   2
ada0      10.8   0.5  1279.1    52.8    0   1.0   0
ada1      13.0   0.1  1384.3     1.6    0   1.3   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.0   0.0    79.4     0.0    0  17.8   1
da1        2.3   0.0   104.5     0.0    0  20.0   1
da3        3.4   0.0   193.2     0.0    0  11.3   1
da4        3.5   0.0   205.7     0.0    0  18.8   2
ada0      18.2   0.1  2237.4    12.8    0   1.2   1
ada1      16.7   0.2  1801.8    25.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        4.8   5.9   259.5    27.8    0   6.5   2
da1        5.3   4.1   283.8    12.3    0  13.7   3
da3        3.8   4.1   243.2    12.3    0   7.7   1
da4        5.5   5.9   310.5    27.8    0   8.6   2
ada0      16.1   0.3  1912.6    27.2    0   1.3   1
ada1      14.0   0.4  1457.5    17.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.4   5.0   540.5    16.0    0  10.9   3
da1        8.6   3.5   513.9     8.7    0  13.1   3
da3        8.8   3.5   526.4     8.7    0  16.0   4
da4        9.9   5.0   579.0    16.0    0  14.1   3
ada0      30.1   1.0  3703.2   116.8    0   1.6   1
ada1      33.0   1.2  3921.3   108.8    0   1.6   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.1   0.0   155.8     0.0    0   7.1   1
da1        3.5   0.0   205.7     0.0    0  13.5   1
da3        2.9   0.0   179.5     0.0    0  10.0   1
da4        3.5   0.0   193.5     0.0    0   8.3   1
ada0      12.8   0.7  1547.4    89.6    0   1.1   1
ada1      11.8   0.2  1301.5     3.2    0   1.4   1
da0, da1, da3 and da4 formed a ZFS pool, while ada0, ada1 are SSDs.

Reference

ZFS L2ARC (Brendan Gregg)
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
 

Weaseal

Member

Thanks: 9
Messages: 57

#2
According to the wiki, you have these set incorrectly; /etc/sysctl.conf is not honored by ZFS:
Note that this value is a run time sysctl that is read at pool import time, which means the default mechanism of setting sysctls in FreeBSD will not work. (Values set in /etc/sysctl.conf are set after ZFS pools are imported)
I believe these tunables should be moved to /boot/loader.conf
 

t1066

Active Member

Thanks: 82
Messages: 218

#3
Update information on sysctl tunables and add some examples

The following are the sysctl tunables and their defaults for L2ARC. I have add a short description at the end of each tunables.

Code:
vfs.zfs.l2arc_write_max: 8388608    # Maximum number of bytes written to l2arc per feed
vfs.zfs.l2arc_write_boost: 8388608  # Mostly only relevant at the first few hours after boot
vfs.zfs.l2arc_headroom: 2           # Not sure
vfs.zfs.l2arc_feed_secs: 1          # l2arc feeding period
vfs.zfs.l2arc_feed_min_ms: 200      # minimum l2arc feeding period
vfs.zfs.l2arc_noprefetch: 1         # control whether streaming data is cached or not
vfs.zfs.l2arc_feed_again: 1         # control whether feed_min_ms is used or not
vfs.zfs.l2arc_norw: 1               # no read and write at the same time
As mention in the above post, changes to vfs.zfs.l2arc_write_max will not affect the behaviour of a running pool. For best compatibility, it should be set in /boot/loader.conf.

The other interesting tunables, vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again do not have such restrictions. You can set them in the command line, and it will immediately take effect.

If vfs.zfs.l2arc_feed_again=1, then when the data that are going to written onto L2ARC is larger than vfs.zfs.l2arc_write_max, the feeding period of L2ARC is reduced, up to vfs.zfs.l2arc_feed_min_ms. To illustrate how it works, here are three ways to set maximum write speed on L2ARC to 80MB/s

Code:
vfs.zfs.l2arc_write_max=83886080
vfs.zfs.l2arc_feed_again=0
# feed 80MB per second
or

Code:
vfs.zfs.l2arc_feed_min_ms=100
# feed 8MB per 100 ms
or

Code:
vfs.zfs.l2arc_write_max=16777216
# feed 16MG per 200 ms
When the L2ARC consists only one drive, the difference is not much, but if the L2ARC consists 2 drives, cache1 and cache2, then in the above example, when L2ARC is writing at its maximum speed, we will see

Code:
80MB written to cache1 in the first second
80MB written to cache2 in the next second
or

Code:
8MB written to cache1 in the first 100ms,
8MB written to cache2 in the next 100ms, etc.
or

Code:
16MB written to cache1 in the first 200ms,
16MB written to cache2 in the next 200ms, etc.
 

gkontos

Daemon

Thanks: 437
Messages: 2,069

#4
I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.

That would only be needed if they can not be adjusted without a reboot.
 

t1066

Active Member

Thanks: 82
Messages: 218

#5
Actually, you may adjust vfs.zfs.l2arc_write_max on the command line also. But it will not take effect until you export and then import back your pool. If the data pool is not separately from the root, this have to set on /boot/loader.conf instead of /etc/sysctl.conf. And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.

However, you can change vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again on the command line and it will immediately change the behviour of the pool without a need for export/import.

vfs.zfs.l2arc_feed_secs and vfs.zfs.l2arc_norw seem to have reasonable defaults.
 

arabesc

Member

Thanks: 3
Messages: 35

#6
I believe these tunables should be moved to /boot/loader.conf
I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.
And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.
I have placed those modified vfs.zfs.l2arc_write_max and vfs.zfs.l2arc_write_boost variables to /boot/loader.conf, but they have default values after system reboot. What's wrong?
My system version is 10.2.
 

borov

New Member

Thanks: 2
Messages: 7

#7
I have placed those modified vfs.zfs.l2arc_write_max and vfs.zfs.l2arc_write_boost variables to /boot/loader.conf, but they have default values after system reboot. What's wrong?
My system version is 10.2.
I have similar problem.

Also this setting do not work.
sysctl vfs.zfs.l2arc_noprefetch="0"
 

t1066

Active Member

Thanks: 82
Messages: 218

#8
If you can change the values in the command line, they should be to put in /etc/sysctl.conf.