|
|
|
|
|
|||||||
| Howtos & FAQs (Moderated) Would you like to share some of your solutions for certain problems? Tips or tricks? Post here. All new topics are automatically moderated. |
 |
|
|
Thread Tools | Display Modes |
|
#1
February 18th, 2012, 08:14
|
|||
|
|||
HOWTO: Tuning L2ARC in ZFS
General suggestions
1. Design the file system intelligently. Group frequently used files together. 2. Get enough cache capacity so that the working set fit into the cache. 3. Set secondarycache=all only for frequently used data. For example: # zfs set secondarycache=all data/frequent-used-data dataSysctl tunables The following are all the sysctl tunables involving L2ARC and their defaults. Code:
sysctl -a | grep l2arc vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_norw: 1 1. vfz.zfs.l2arc_write_max This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives. 2. vfs.zfs.l2arc_write_boost This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system. 3. vfs.zfs.l2arc_noprefetch This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%. The following is what I had added to /etc/sysctl.conf. Code:
vfs.zfs.l2arc_noprefetch=0 vfs.zfs.l2arc_write_max=26214400 vfs.zfs.l2arc_write_boost=52428800 Code:
iostat -xz -w 10 -c 10
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 9.5 2.7 585.1 73.2 0 18.1 3
da1 9.5 2.7 583.7 73.3 0 16.9 3
da3 9.4 2.7 584.0 73.3 0 17.3 3
da4 9.5 2.8 583.1 73.2 0 16.7 3
ada0 19.9 1.5 2375.9 157.5 0 1.5 1
ada1 19.4 1.6 2360.4 172.9 0 2.1 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 2.7 0.0 142.4 0.0 0 14.5 1
da1 2.3 0.0 104.6 0.0 0 5.1 0
da3 3.3 0.0 181.1 0.0 0 16.1 1
da4 2.4 0.0 129.2 0.0 0 10.7 1
ada0 28.9 0.2 3539.3 25.6 0 1.3 1
ada1 30.2 0.3 3506.9 38.4 0 1.5 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 1.8 0.0 115.2 0.0 0 8.2 1
da1 2.2 0.0 140.8 0.0 0 10.6 0
da3 1.8 0.0 115.2 0.0 0 11.5 0
da4 1.0 0.0 64.0 0.0 0 7.8 0
ada0 15.0 0.1 1841.4 12.8 0 1.4 1
ada1 16.7 1.2 2003.0 153.6 0 2.0 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 1.1 0.0 64.6 0.0 0 8.9 0
da1 1.4 0.0 89.6 0.0 0 9.8 1
da3 1.2 0.0 76.8 0.0 0 13.7 1
da4 1.8 0.0 115.2 0.0 0 14.4 1
ada0 14.3 0.1 1763.0 12.8 0 0.9 1
ada1 12.0 0.4 1311.9 51.2 0 0.9 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 3.6 5.5 169.8 29.8 0 10.2 2
da1 2.8 3.9 105.7 11.6 0 7.0 1
da3 2.6 3.9 154.2 11.6 0 5.5 1
da4 1.7 5.5 78.3 29.8 0 4.9 1
ada0 12.2 0.2 1368.7 14.4 0 1.0 0
ada1 9.2 0.9 874.0 65.6 0 0.9 0
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 5.2 0.0 297.2 0.0 0 13.7 2
da1 4.1 0.0 219.8 0.0 0 10.3 1
da3 4.0 0.0 243.8 0.0 0 23.5 2
da4 5.9 0.0 347.1 0.0 0 15.7 2
ada0 10.8 0.5 1279.1 52.8 0 1.0 0
ada1 13.0 0.1 1384.3 1.6 0 1.3 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 2.0 0.0 79.4 0.0 0 17.8 1
da1 2.3 0.0 104.5 0.0 0 20.0 1
da3 3.4 0.0 193.2 0.0 0 11.3 1
da4 3.5 0.0 205.7 0.0 0 18.8 2
ada0 18.2 0.1 2237.4 12.8 0 1.2 1
ada1 16.7 0.2 1801.8 25.6 0 1.0 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 4.8 5.9 259.5 27.8 0 6.5 2
da1 5.3 4.1 283.8 12.3 0 13.7 3
da3 3.8 4.1 243.2 12.3 0 7.7 1
da4 5.5 5.9 310.5 27.8 0 8.6 2
ada0 16.1 0.3 1912.6 27.2 0 1.3 1
ada1 14.0 0.4 1457.5 17.6 0 1.0 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 9.4 5.0 540.5 16.0 0 10.9 3
da1 8.6 3.5 513.9 8.7 0 13.1 3
da3 8.8 3.5 526.4 8.7 0 16.0 4
da4 9.9 5.0 579.0 16.0 0 14.1 3
ada0 30.1 1.0 3703.2 116.8 0 1.6 1
ada1 33.0 1.2 3921.3 108.8 0 1.6 1
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
da0 3.1 0.0 155.8 0.0 0 7.1 1
da1 3.5 0.0 205.7 0.0 0 13.5 1
da3 2.9 0.0 179.5 0.0 0 10.0 1
da4 3.5 0.0 193.5 0.0 0 8.3 1
ada0 12.8 0.7 1547.4 89.6 0 1.1 1
ada1 11.8 0.2 1301.5 3.2 0 1.4 1
Reference ZFS L2ARC (Brendan Gregg) /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Last edited by DutchDaemon; February 19th, 2012 at 02:56. 
|
|
#2
September 14th, 2012, 00:00
|
|||
|
|||
|
According to the wiki, you have these set incorrectly; /etc/sysctl.conf is not honored by ZFS:
Quote:
__________________
Systems Administrator and Software Engineer
|
|
#3
September 17th, 2012, 16:51
|
|||
|
|||
|
Update information on sysctl tunables and add some examples
The following are the sysctl tunables and their defaults for L2ARC. I have add a short description at the end of each tunables.
Code:
vfs.zfs.l2arc_write_max: 8388608 # Maximum number of bytes written to l2arc per feed vfs.zfs.l2arc_write_boost: 8388608 # Mostly only relevant at the first few hours after boot vfs.zfs.l2arc_headroom: 2 # Not sure vfs.zfs.l2arc_feed_secs: 1 # l2arc feeding period vfs.zfs.l2arc_feed_min_ms: 200 # minimum l2arc feeding period vfs.zfs.l2arc_noprefetch: 1 # control whether streaming data is cached or not vfs.zfs.l2arc_feed_again: 1 # control whether feed_min_ms is used or not vfs.zfs.l2arc_norw: 1 # no read and write at the same time The other interesting tunables, vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again do not have such restrictions. You can set them in the command line, and it will immediately take effect. If vfs.zfs.l2arc_feed_again=1, then when the data that are going to written onto L2ARC is larger than vfs.zfs.l2arc_write_max, the feeding period of L2ARC is reduced, up to vfs.zfs.l2arc_feed_min_ms. To illustrate how it works, here are three ways to set maximum write speed on L2ARC to 80MB/s Code:
vfs.zfs.l2arc_write_max=83886080 vfs.zfs.l2arc_feed_again=0 # feed 80MB per second Code:
vfs.zfs.l2arc_feed_min_ms=100 # feed 8MB per 100 ms Code:
vfs.zfs.l2arc_write_max=16777216 # feed 16MG per 200 ms Code:
80MB written to cache1 in the first second 80MB written to cache2 in the next second Code:
8MB written to cache1 in the first 100ms, 8MB written to cache2 in the next 100ms, etc. Code:
16MB written to cache1 in the first 200ms, 16MB written to cache2 in the next 200ms, etc.
|
|
#4
September 17th, 2012, 17:58
|
||||
|
||||
|
I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.
That would only be needed if they can not be adjusted without a reboot.
__________________
Powered by BareBSD
|
|
#5
September 18th, 2012, 01:38
|
|||
|
|||
|
Actually, you may adjust vfs.zfs.l2arc_write_max on the command line also. But it will not take effect until you export and then import back your pool. If the data pool is not separately from the root, this have to set on /boot/loader.conf instead of /etc/sysctl.conf. And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.
However, you can change vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again on the command line and it will immediately change the behviour of the pool without a need for export/import. vfs.zfs.l2arc_feed_secs and vfs.zfs.l2arc_norw seem to have reasonable defaults.
|
|
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| ZFS zil and l2arc on other controller | Sylhouette | System Hardware | 1 | November 2nd, 2011 11:18 |
| 8.2 zfs on i386 tuning help | spork | General | 4 | March 29th, 2011 00:04 |
| TRIM support on ZFS ZIL/L2ARC-devices | FLAGEL | General | 6 | October 9th, 2010 19:10 |
| [Solved] ZFS: How To Calulate for Tuning? | dave | General | 12 | October 3rd, 2010 00:01 |
| ZFS Performance and tuning | Ungaro | General | 16 | July 3rd, 2010 14:48 |
Linear Mode
