General suggestions
1. Design the file system intelligently. Group frequently used files together.
2. Get enough cache capacity so that the working set fit into the cache.
3. Set secondarycache=all only for frequently used data. For example:
Sysctl tunables
The following are all the sysctl tunables involving L2ARC and their defaults.
I will focus on the following 3 tunables.
1. vfz.zfs.l2arc_write_max
This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives.
2. vfs.zfs.l2arc_write_boost
This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system.
3. vfs.zfs.l2arc_noprefetch
This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%.
The following is what I had added to /etc/sysctl.conf.
The following is the iostat from my system when I write this howto:
da0, da1, da3 and da4 formed a ZFS pool, while ada0, ada1 are SSDs.
Reference
ZFS L2ARC (Brendan Gregg)
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
1. Design the file system intelligently. Group frequently used files together.
2. Get enough cache capacity so that the working set fit into the cache.
3. Set secondarycache=all only for frequently used data. For example:
# zfs set secondarycache=all data/frequent-used-data data
Sysctl tunables
The following are all the sysctl tunables involving L2ARC and their defaults.
Code:
sysctl -a | grep l2arc
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_norw: 1
I will focus on the following 3 tunables.
1. vfz.zfs.l2arc_write_max
This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives.
2. vfs.zfs.l2arc_write_boost
This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system.
3. vfs.zfs.l2arc_noprefetch
This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%.
The following is what I had added to /etc/sysctl.conf.
Code:
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_write_max=26214400
vfs.zfs.l2arc_write_boost=52428800
The following is the iostat from my system when I write this howto:
Code:
iostat -xz -w 10 -c 10
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 9.5 2.7 585.1 73.2 0 18.1 3
da1 9.5 2.7 583.7 73.3 0 16.9 3
da3 9.4 2.7 584.0 73.3 0 17.3 3
da4 9.5 2.8 583.1 73.2 0 16.7 3
ada0 19.9 1.5 2375.9 157.5 0 1.5 1
ada1 19.4 1.6 2360.4 172.9 0 2.1 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 2.7 0.0 142.4 0.0 0 14.5 1
da1 2.3 0.0 104.6 0.0 0 5.1 0
da3 3.3 0.0 181.1 0.0 0 16.1 1
da4 2.4 0.0 129.2 0.0 0 10.7 1
ada0 28.9 0.2 3539.3 25.6 0 1.3 1
ada1 30.2 0.3 3506.9 38.4 0 1.5 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 1.8 0.0 115.2 0.0 0 8.2 1
da1 2.2 0.0 140.8 0.0 0 10.6 0
da3 1.8 0.0 115.2 0.0 0 11.5 0
da4 1.0 0.0 64.0 0.0 0 7.8 0
ada0 15.0 0.1 1841.4 12.8 0 1.4 1
ada1 16.7 1.2 2003.0 153.6 0 2.0 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 1.1 0.0 64.6 0.0 0 8.9 0
da1 1.4 0.0 89.6 0.0 0 9.8 1
da3 1.2 0.0 76.8 0.0 0 13.7 1
da4 1.8 0.0 115.2 0.0 0 14.4 1
ada0 14.3 0.1 1763.0 12.8 0 0.9 1
ada1 12.0 0.4 1311.9 51.2 0 0.9 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 3.6 5.5 169.8 29.8 0 10.2 2
da1 2.8 3.9 105.7 11.6 0 7.0 1
da3 2.6 3.9 154.2 11.6 0 5.5 1
da4 1.7 5.5 78.3 29.8 0 4.9 1
ada0 12.2 0.2 1368.7 14.4 0 1.0 0
ada1 9.2 0.9 874.0 65.6 0 0.9 0
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 5.2 0.0 297.2 0.0 0 13.7 2
da1 4.1 0.0 219.8 0.0 0 10.3 1
da3 4.0 0.0 243.8 0.0 0 23.5 2
da4 5.9 0.0 347.1 0.0 0 15.7 2
ada0 10.8 0.5 1279.1 52.8 0 1.0 0
ada1 13.0 0.1 1384.3 1.6 0 1.3 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 2.0 0.0 79.4 0.0 0 17.8 1
da1 2.3 0.0 104.5 0.0 0 20.0 1
da3 3.4 0.0 193.2 0.0 0 11.3 1
da4 3.5 0.0 205.7 0.0 0 18.8 2
ada0 18.2 0.1 2237.4 12.8 0 1.2 1
ada1 16.7 0.2 1801.8 25.6 0 1.0 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 4.8 5.9 259.5 27.8 0 6.5 2
da1 5.3 4.1 283.8 12.3 0 13.7 3
da3 3.8 4.1 243.2 12.3 0 7.7 1
da4 5.5 5.9 310.5 27.8 0 8.6 2
ada0 16.1 0.3 1912.6 27.2 0 1.3 1
ada1 14.0 0.4 1457.5 17.6 0 1.0 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 9.4 5.0 540.5 16.0 0 10.9 3
da1 8.6 3.5 513.9 8.7 0 13.1 3
da3 8.8 3.5 526.4 8.7 0 16.0 4
da4 9.9 5.0 579.0 16.0 0 14.1 3
ada0 30.1 1.0 3703.2 116.8 0 1.6 1
ada1 33.0 1.2 3921.3 108.8 0 1.6 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 3.1 0.0 155.8 0.0 0 7.1 1
da1 3.5 0.0 205.7 0.0 0 13.5 1
da3 2.9 0.0 179.5 0.0 0 10.0 1
da4 3.5 0.0 193.5 0.0 0 8.3 1
ada0 12.8 0.7 1547.4 89.6 0 1.1 1
ada1 11.8 0.2 1301.5 3.2 0 1.4 1
da0, da1, da3 and da4 formed a ZFS pool, while ada0, ada1 are SSDs.
Reference
ZFS L2ARC (Brendan Gregg)
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c