HOWTO: Tuning L2ARC in ZFS

Would you like to share some of your solutions for certain problems? Tips or tricks? Post here. All new topics are automatically moderated.

HOWTO: Tuning L2ARC in ZFS

Postby t1066 » 18 Feb 2012, 07:14

General suggestions

1. Design the file system intelligently. Group frequently used files together.

2. Get enough cache capacity so that the working set fit into the cache.

3. Set secondarycache=all only for frequently used data. For example:

[CMD="#"]zfs set secondarycache=all data/frequent-used-data data[/CMD]

Sysctl tunables

The following are all the sysctl tunables involving L2ARC and their defaults.

Code: Select all
sysctl -a | grep l2arc

vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_norw: 1


I will focus on the following 3 tunables.

1. [FILE]vfz.zfs.l2arc_write_max[/FILE]

This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives.

2. [FILE]vfs.zfs.l2arc_write_boost[/FILE]

This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system.

3. [FILE]vfs.zfs.l2arc_noprefetch[/FILE]

This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%.

The following is what I had added to [FILE]/etc/sysctl.conf[/FILE].

Code: Select all
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_write_max=26214400
vfs.zfs.l2arc_write_boost=52428800


The following is the [FILE]iostat[/FILE] from my system when I write this howto:

Code: Select all
iostat -xz -w 10 -c 10
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.5   2.7   585.1    73.2    0  18.1   3
da1        9.5   2.7   583.7    73.3    0  16.9   3
da3        9.4   2.7   584.0    73.3    0  17.3   3
da4        9.5   2.8   583.1    73.2    0  16.7   3
ada0      19.9   1.5  2375.9   157.5    0   1.5   1
ada1      19.4   1.6  2360.4   172.9    0   2.1   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.7   0.0   142.4     0.0    0  14.5   1
da1        2.3   0.0   104.6     0.0    0   5.1   0
da3        3.3   0.0   181.1     0.0    0  16.1   1
da4        2.4   0.0   129.2     0.0    0  10.7   1
ada0      28.9   0.2  3539.3    25.6    0   1.3   1
ada1      30.2   0.3  3506.9    38.4    0   1.5   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.8   0.0   115.2     0.0    0   8.2   1
da1        2.2   0.0   140.8     0.0    0  10.6   0
da3        1.8   0.0   115.2     0.0    0  11.5   0
da4        1.0   0.0    64.0     0.0    0   7.8   0
ada0      15.0   0.1  1841.4    12.8    0   1.4   1
ada1      16.7   1.2  2003.0   153.6    0   2.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.1   0.0    64.6     0.0    0   8.9   0
da1        1.4   0.0    89.6     0.0    0   9.8   1
da3        1.2   0.0    76.8     0.0    0  13.7   1
da4        1.8   0.0   115.2     0.0    0  14.4   1
ada0      14.3   0.1  1763.0    12.8    0   0.9   1
ada1      12.0   0.4  1311.9    51.2    0   0.9   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.6   5.5   169.8    29.8    0  10.2   2
da1        2.8   3.9   105.7    11.6    0   7.0   1
da3        2.6   3.9   154.2    11.6    0   5.5   1
da4        1.7   5.5    78.3    29.8    0   4.9   1
ada0      12.2   0.2  1368.7    14.4    0   1.0   0
ada1       9.2   0.9   874.0    65.6    0   0.9   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        5.2   0.0   297.2     0.0    0  13.7   2
da1        4.1   0.0   219.8     0.0    0  10.3   1
da3        4.0   0.0   243.8     0.0    0  23.5   2
da4        5.9   0.0   347.1     0.0    0  15.7   2
ada0      10.8   0.5  1279.1    52.8    0   1.0   0
ada1      13.0   0.1  1384.3     1.6    0   1.3   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.0   0.0    79.4     0.0    0  17.8   1
da1        2.3   0.0   104.5     0.0    0  20.0   1
da3        3.4   0.0   193.2     0.0    0  11.3   1
da4        3.5   0.0   205.7     0.0    0  18.8   2
ada0      18.2   0.1  2237.4    12.8    0   1.2   1
ada1      16.7   0.2  1801.8    25.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        4.8   5.9   259.5    27.8    0   6.5   2
da1        5.3   4.1   283.8    12.3    0  13.7   3
da3        3.8   4.1   243.2    12.3    0   7.7   1
da4        5.5   5.9   310.5    27.8    0   8.6   2
ada0      16.1   0.3  1912.6    27.2    0   1.3   1
ada1      14.0   0.4  1457.5    17.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.4   5.0   540.5    16.0    0  10.9   3
da1        8.6   3.5   513.9     8.7    0  13.1   3
da3        8.8   3.5   526.4     8.7    0  16.0   4
da4        9.9   5.0   579.0    16.0    0  14.1   3
ada0      30.1   1.0  3703.2   116.8    0   1.6   1
ada1      33.0   1.2  3921.3   108.8    0   1.6   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.1   0.0   155.8     0.0    0   7.1   1
da1        3.5   0.0   205.7     0.0    0  13.5   1
da3        2.9   0.0   179.5     0.0    0  10.0   1
da4        3.5   0.0   193.5     0.0    0   8.3   1
ada0      12.8   0.7  1547.4    89.6    0   1.1   1
ada1      11.8   0.2  1301.5     3.2    0   1.4   1


[FILE]da0[/FILE], [FILE]da1[/FILE], [FILE]da3[/FILE] and [FILE]da4[/FILE] formed a ZFS pool, while [FILE]ada0[/FILE], [FILE]ada1[/FILE] are SSDs.

Reference

ZFS L2ARC (Brendan Gregg)
[FILE]/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c[/FILE]
t1066
Member
 
Posts: 154
Joined: 07 Jun 2010, 16:49

Postby Weaseal » 13 Sep 2012, 23:00

According to the wiki, you have these set incorrectly; /etc/sysctl.conf is not honored by ZFS:
Note that this value is a run time sysctl that is read at pool import time, which means the default mechanism of setting sysctls in FreeBSD will not work. (Values set in /etc/sysctl.conf are set after ZFS pools are imported)


I believe these tunables should be moved to /boot/loader.conf
Systems Administrator and Software Engineer
Weaseal
Junior Member
 
Posts: 57
Joined: 16 Nov 2008, 16:34

Update information on sysctl tunables and add some examples

Postby t1066 » 17 Sep 2012, 15:51

The following are the [FILE]sysctl[/FILE] tunables and their defaults for L2ARC. I have add a short description at the end of each tunables.

Code: Select all
vfs.zfs.l2arc_write_max: 8388608    # Maximum number of bytes written to l2arc per feed
vfs.zfs.l2arc_write_boost: 8388608  # Mostly only relevant at the first few hours after boot
vfs.zfs.l2arc_headroom: 2           # Not sure
vfs.zfs.l2arc_feed_secs: 1          # l2arc feeding period
vfs.zfs.l2arc_feed_min_ms: 200      # minimum l2arc feeding period
vfs.zfs.l2arc_noprefetch: 1         # control whether streaming data is cached or not
vfs.zfs.l2arc_feed_again: 1         # control whether feed_min_ms is used or not
vfs.zfs.l2arc_norw: 1               # no read and write at the same time


As mention in the above post, changes to [FILE]vfs.zfs.l2arc_write_max[/FILE] will not affect the behaviour of a running pool. For best compatibility, it should be set in [FILE]/boot/loader.conf[/FILE].

The other interesting tunables, [FILE]vfs.zfs.l2arc_feed_min_ms[/FILE], [FILE]vfs.zfs.l2arc_noprefetch[/FILE], [FILE]vfs.zfs.l2arc_feed_again[/FILE] do not have such restrictions. You can set them in the command line, and it will immediately take effect.

If [FILE]vfs.zfs.l2arc_feed_again=1[/FILE], then when the data that are going to written onto L2ARC is larger than [FILE]vfs.zfs.l2arc_write_max[/FILE], the feeding period of L2ARC is reduced, up to [FILE]vfs.zfs.l2arc_feed_min_ms[/FILE]. To illustrate how it works, here are three ways to set maximum write speed on L2ARC to 80MB/s

Code: Select all
vfs.zfs.l2arc_write_max=83886080
vfs.zfs.l2arc_feed_again=0
# feed 80MB per second


or

Code: Select all
vfs.zfs.l2arc_feed_min_ms=100
# feed 8MB per 100 ms


or

Code: Select all
vfs.zfs.l2arc_write_max=16777216
# feed 16MG per 200 ms


When the L2ARC consists only one drive, the difference is not much, but if the L2ARC consists 2 drives, [FILE]cache1[/FILE] and [FILE]cache2[/FILE], then in the above example, when L2ARC is writing at its maximum speed, we will see

Code: Select all
80MB written to cache1 in the first second
80MB written to cache2 in the next second


or

Code: Select all
8MB written to cache1 in the first 100ms,
8MB written to cache2 in the next 100ms, etc.


or

Code: Select all
16MB written to cache1 in the first 200ms,
16MB written to cache2 in the next 200ms, etc.
t1066
Member
 
Posts: 154
Joined: 07 Jun 2010, 16:49

Postby gkontos » 17 Sep 2012, 16:58

I think he means that those variables should go to [FILE]/boot/loader.conf [/FILE]instead of [FILE]/etc/sysctl.conf[/FILE].

That would only be needed if they can not be adjusted without a reboot.
Powered by BareBSD
User avatar
gkontos
Senior Member
 
Posts: 1370
Joined: 09 Dec 2009, 08:36
Location: Polidendri, GR

Postby t1066 » 18 Sep 2012, 00:38

Actually, you may adjust [FILE]vfs.zfs.l2arc_write_max[/FILE] on the command line also. But it will not take effect until you export and then import back your pool. If the data pool is not separately from the root, this have to set on [FILE]/boot/loader.conf[/FILE] instead of [FILE]/etc/sysctl.conf[/FILE]. And it is also better to put [FILE]vfs.zfs.l2arc_write_boost[/FILE] on [FILE]/boot/loader.conf[/FILE].

However, you can change [FILE]vfs.zfs.l2arc_feed_min_ms[/FILE], [FILE]vfs.zfs.l2arc_noprefetch[/FILE], [FILE]vfs.zfs.l2arc_feed_again[/FILE] on the command line and it will immediately change the behviour of the pool without a need for export/import.

[FILE]vfs.zfs.l2arc_feed_secs[/FILE] and [FILE]vfs.zfs.l2arc_norw[/FILE] seem to have reasonable defaults.
t1066
Member
 
Posts: 154
Joined: 07 Jun 2010, 16:49


Return to Howtos & FAQs (Moderated)

Who is online

Users browsing this forum: No registered users and 1 guest