1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HOWTO: Tuning L2ARC in ZFS

Discussion in 'Howtos and FAQs (Moderated)' started by t1066, Feb 18, 2012.

  1. t1066

    t1066 Member

    Messages:
    173
    Thanks Received:
    33
    General suggestions

    1. Design the file system intelligently. Group frequently used files together.

    2. Get enough cache capacity so that the working set fit into the cache.

    3. Set secondarycache=all only for frequently used data. For example:

    # zfs set secondarycache=all data/frequent-used-data data

    Sysctl tunables

    The following are all the sysctl tunables involving L2ARC and their defaults.

    Code:
    sysctl -a | grep l2arc
    
    vfs.zfs.l2arc_write_max: 8388608
    vfs.zfs.l2arc_write_boost: 8388608
    vfs.zfs.l2arc_headroom: 2
    vfs.zfs.l2arc_feed_secs: 1
    vfs.zfs.l2arc_feed_min_ms: 200
    vfs.zfs.l2arc_noprefetch: 1
    vfs.zfs.l2arc_feed_again: 1
    vfs.zfs.l2arc_norw: 1


    I will focus on the following 3 tunables.

    1. vfz.zfs.l2arc_write_max

    This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives.

    2. vfs.zfs.l2arc_write_boost

    This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system.

    3. vfs.zfs.l2arc_noprefetch

    This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%.

    The following is what I had added to /etc/sysctl.conf.

    Code:
    vfs.zfs.l2arc_noprefetch=0
    vfs.zfs.l2arc_write_max=26214400
    vfs.zfs.l2arc_write_boost=52428800


    The following is the iostat from my system when I write this howto:

    Code:
    iostat -xz -w 10 -c 10
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        9.5   2.7   585.1    73.2    0  18.1   3
    da1        9.5   2.7   583.7    73.3    0  16.9   3
    da3        9.4   2.7   584.0    73.3    0  17.3   3
    da4        9.5   2.8   583.1    73.2    0  16.7   3
    ada0      19.9   1.5  2375.9   157.5    0   1.5   1
    ada1      19.4   1.6  2360.4   172.9    0   2.1   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        2.7   0.0   142.4     0.0    0  14.5   1
    da1        2.3   0.0   104.6     0.0    0   5.1   0
    da3        3.3   0.0   181.1     0.0    0  16.1   1
    da4        2.4   0.0   129.2     0.0    0  10.7   1
    ada0      28.9   0.2  3539.3    25.6    0   1.3   1
    ada1      30.2   0.3  3506.9    38.4    0   1.5   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        1.8   0.0   115.2     0.0    0   8.2   1
    da1        2.2   0.0   140.8     0.0    0  10.6   0
    da3        1.8   0.0   115.2     0.0    0  11.5   0
    da4        1.0   0.0    64.0     0.0    0   7.8   0
    ada0      15.0   0.1  1841.4    12.8    0   1.4   1
    ada1      16.7   1.2  2003.0   153.6    0   2.0   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        1.1   0.0    64.6     0.0    0   8.9   0
    da1        1.4   0.0    89.6     0.0    0   9.8   1
    da3        1.2   0.0    76.8     0.0    0  13.7   1
    da4        1.8   0.0   115.2     0.0    0  14.4   1
    ada0      14.3   0.1  1763.0    12.8    0   0.9   1
    ada1      12.0   0.4  1311.9    51.2    0   0.9   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        3.6   5.5   169.8    29.8    0  10.2   2
    da1        2.8   3.9   105.7    11.6    0   7.0   1
    da3        2.6   3.9   154.2    11.6    0   5.5   1
    da4        1.7   5.5    78.3    29.8    0   4.9   1
    ada0      12.2   0.2  1368.7    14.4    0   1.0   0
    ada1       9.2   0.9   874.0    65.6    0   0.9   0
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        5.2   0.0   297.2     0.0    0  13.7   2
    da1        4.1   0.0   219.8     0.0    0  10.3   1
    da3        4.0   0.0   243.8     0.0    0  23.5   2
    da4        5.9   0.0   347.1     0.0    0  15.7   2
    ada0      10.8   0.5  1279.1    52.8    0   1.0   0
    ada1      13.0   0.1  1384.3     1.6    0   1.3   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        2.0   0.0    79.4     0.0    0  17.8   1
    da1        2.3   0.0   104.5     0.0    0  20.0   1
    da3        3.4   0.0   193.2     0.0    0  11.3   1
    da4        3.5   0.0   205.7     0.0    0  18.8   2
    ada0      18.2   0.1  2237.4    12.8    0   1.2   1
    ada1      16.7   0.2  1801.8    25.6    0   1.0   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        4.8   5.9   259.5    27.8    0   6.5   2
    da1        5.3   4.1   283.8    12.3    0  13.7   3
    da3        3.8   4.1   243.2    12.3    0   7.7   1
    da4        5.5   5.9   310.5    27.8    0   8.6   2
    ada0      16.1   0.3  1912.6    27.2    0   1.3   1
    ada1      14.0   0.4  1457.5    17.6    0   1.0   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        9.4   5.0   540.5    16.0    0  10.9   3
    da1        8.6   3.5   513.9     8.7    0  13.1   3
    da3        8.8   3.5   526.4     8.7    0  16.0   4
    da4        9.9   5.0   579.0    16.0    0  14.1   3
    ada0      30.1   1.0  3703.2   116.8    0   1.6   1
    ada1      33.0   1.2  3921.3   108.8    0   1.6   1
                            extended device statistics
    device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
    da0        3.1   0.0   155.8     0.0    0   7.1   1
    da1        3.5   0.0   205.7     0.0    0  13.5   1
    da3        2.9   0.0   179.5     0.0    0  10.0   1
    da4        3.5   0.0   193.5     0.0    0   8.3   1
    ada0      12.8   0.7  1547.4    89.6    0   1.1   1
    ada1      11.8   0.2  1301.5     3.2    0   1.4   1
    


    da0, da1, da3 and da4 formed a ZFS pool, while ada0, ada1 are SSDs.

    Reference

    ZFS L2ARC (Brendan Gregg)
    /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
     
  2. Weaseal

    Weaseal New Member

    Messages:
    57
    Thanks Received:
    7
    According to the wiki, you have these set incorrectly; /etc/sysctl.conf is not honored by ZFS:
    I believe these tunables should be moved to /boot/loader.conf
     
  3. t1066

    t1066 Member

    Messages:
    173
    Thanks Received:
    33
    Update information on sysctl tunables and add some examples

    The following are the sysctl tunables and their defaults for L2ARC. I have add a short description at the end of each tunables.

    Code:
    vfs.zfs.l2arc_write_max: 8388608    # Maximum number of bytes written to l2arc per feed
    vfs.zfs.l2arc_write_boost: 8388608  # Mostly only relevant at the first few hours after boot
    vfs.zfs.l2arc_headroom: 2           # Not sure
    vfs.zfs.l2arc_feed_secs: 1          # l2arc feeding period
    vfs.zfs.l2arc_feed_min_ms: 200      # minimum l2arc feeding period
    vfs.zfs.l2arc_noprefetch: 1         # control whether streaming data is cached or not
    vfs.zfs.l2arc_feed_again: 1         # control whether feed_min_ms is used or not
    vfs.zfs.l2arc_norw: 1               # no read and write at the same time


    As mention in the above post, changes to vfs.zfs.l2arc_write_max will not affect the behaviour of a running pool. For best compatibility, it should be set in /boot/loader.conf.

    The other interesting tunables, vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again do not have such restrictions. You can set them in the command line, and it will immediately take effect.

    If vfs.zfs.l2arc_feed_again=1, then when the data that are going to written onto L2ARC is larger than vfs.zfs.l2arc_write_max, the feeding period of L2ARC is reduced, up to vfs.zfs.l2arc_feed_min_ms. To illustrate how it works, here are three ways to set maximum write speed on L2ARC to 80MB/s

    Code:
    vfs.zfs.l2arc_write_max=83886080
    vfs.zfs.l2arc_feed_again=0
    # feed 80MB per second


    or

    Code:
    vfs.zfs.l2arc_feed_min_ms=100
    # feed 8MB per 100 ms


    or

    Code:
    vfs.zfs.l2arc_write_max=16777216
    # feed 16MG per 200 ms


    When the L2ARC consists only one drive, the difference is not much, but if the L2ARC consists 2 drives, cache1 and cache2, then in the above example, when L2ARC is writing at its maximum speed, we will see

    Code:
    80MB written to cache1 in the first second
    80MB written to cache2 in the next second


    or

    Code:
    8MB written to cache1 in the first 100ms,
    8MB written to cache2 in the next 100ms, etc.


    or

    Code:
    16MB written to cache1 in the first 200ms,
    16MB written to cache2 in the next 200ms, etc.
     
  4. gkontos

    gkontos Active Member

    Messages:
    1,390
    Thanks Received:
    245
    I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.

    That would only be needed if they can not be adjusted without a reboot.
     
  5. t1066

    t1066 Member

    Messages:
    173
    Thanks Received:
    33
    Actually, you may adjust vfs.zfs.l2arc_write_max on the command line also. But it will not take effect until you export and then import back your pool. If the data pool is not separately from the root, this have to set on /boot/loader.conf instead of /etc/sysctl.conf. And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.

    However, you can change vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again on the command line and it will immediately change the behviour of the pool without a need for export/import.

    vfs.zfs.l2arc_feed_secs and vfs.zfs.l2arc_norw seem to have reasonable defaults.