vfs.zfs.write_limit_override is missing in FreeBSD 9.3?

B

belon_cfy

My server is running FreeBSD 9.3-BETA for testing and evaluation and would like to tune vfs.zfs.write_limit_override to the best value which is working best on all of my FreeBSD server, however no such parameter exists in FreeBSD 9.3 and the behavior seems to be like the default value which is not that good for my I/O pattern.

Is there any parameter to replace vfs.zfs.write_limit_override? Or has it been integrated into part of ZFS and can be tuned by zpool set or zfs set?
 
t1066 said:
Maybe you could try the following tunables.

Code: 
vfs.zfs.vdev.aggregation_limit
vfs.zfs.vdev.write_gap_limit
Click to expand...

I have tried to increase the vfs.zfs.vdev.aggregation_limit and vfs.zfs.vdev.write_gap_limit, however it doesn't make any different as what vfs.zfs.write_limit_override does.

Any reason why the vfs.zfs.write_limit_override has been taken out of FreeBSD 9.3?
 
Looks like the write throttling has been re-written by the ZFS vendor and is now controlled via the following:

Code: 
Modified Wed May 21 13:36:04 2014 UTC (7 weeks, 5 days ago) by smh 
File length: 33136 byte(s) 
Diff to previous 259813
Added sysctls / tunables for ZFS dirty data tuning

Added the following new sysctls / tunables:
* vfs.zfs.dirty_data_max
* vfs.zfs.dirty_data_max_max
* vfs.zfs.dirty_data_max_percent
* vfs.zfs.dirty_data_sync
* vfs.zfs.delay_min_dirty_percent
* vfs.zfs.delay_scale

http://svnweb.freebsd.org/base/head/sys ... c?view=log

There are some comments about what the various tunables do and the overall throttling logic in recent versions of that file.

Edit: Also see this video from ~35.00 for some interesting information about the new write logic:
https://www.youtube.com/watch?v=EjGqVdCOIhM
 
usdmatt said:
Looks like the write throttling has been re-written by the ZFS vendor and is now controlled via the following:

Code: 
Modified Wed May 21 13:36:04 2014 UTC (7 weeks, 5 days ago) by smh 
File length: 33136 byte(s) 
Diff to previous 259813
Added sysctls / tunables for ZFS dirty data tuning

Added the following new sysctls / tunables:
* vfs.zfs.dirty_data_max
* vfs.zfs.dirty_data_max_max
* vfs.zfs.dirty_data_max_percent
* vfs.zfs.dirty_data_sync
* vfs.zfs.delay_min_dirty_percent
* vfs.zfs.delay_scale

http://svnweb.freebsd.org/base/head/sys ... c?view=log

There are some comments about what the various tunables do and the overall throttling logic in recent versions of that file.

Edit: Also see this video from ~35.00 for some interesting information about the new write logic:
https://www.youtube.com/watch?v=EjGqVdCOIhM
Click to expand...

Hi

Thanks for your explanation, I also noticed that my virtual machine's write latency is better than others VMs running on FreeBSD 9.1 storage with ZFS v28 when storages were congesting on committing data from zil to the disk.

I can't find any vfs.zfs.dirty tunable value in FreeBSD 9.3. Do I need to specify in sysctl.conf in order to activate it?
 
Hmm, I don't have a 9.3 machine to check with, but seeing as the new sysctls were added in the same commit that removed the old ones, you should have one or the other. If you're typing sysctl vfs.zfs.dirty you won't find anything as dirty is part of the tunable name, not an independent tunable group, if you see what I mean.

What do you get listed if you just run sysctl vfs.zfs?
 
usdmatt said:
Hmm, I don't have a 9.3 machine to check with, but seeing as the new sysctls were added in the same commit that removed the old ones, you should have one or the other. If you're typing sysctl vfs.zfs.dirty you won't find anything as dirty is part of the tunable name, not an independent tunable group, if you see what I mean.

What do you get listed if you just run sysctl vfs.zfs?
Click to expand...

Below are all of my vfs.zfs parameters in FreeBSD 9.3:
Code: 
vfs.zfs.l2c_only_size: 157159293952
vfs.zfs.mfu_ghost_data_lsize: 196150784
vfs.zfs.mfu_ghost_metadata_lsize: 110637056
vfs.zfs.mfu_ghost_size: 306787840
vfs.zfs.mfu_data_lsize: 2552218112
vfs.zfs.mfu_metadata_lsize: 0
vfs.zfs.mfu_size: 2588467712
vfs.zfs.mru_ghost_data_lsize: 11117056
vfs.zfs.mru_ghost_metadata_lsize: 3585509888
vfs.zfs.mru_ghost_size: 3596626944
vfs.zfs.mru_data_lsize: 374602240
vfs.zfs.mru_metadata_lsize: 1118208
vfs.zfs.mru_size: 437502464
vfs.zfs.anon_data_lsize: 0
vfs.zfs.anon_metadata_lsize: 0
vfs.zfs.anon_size: 11276288
vfs.zfs.l2arc_norw: 0
vfs.zfs.l2arc_feed_again: 0
vfs.zfs.l2arc_noprefetch: 0
vfs.zfs.l2arc_feed_min_ms: 1000
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.arc_meta_limit: 1073741824
vfs.zfs.arc_meta_used: 1081123648
vfs.zfs.arc_min: 536870912
vfs.zfs.arc_max: 4294967296
vfs.zfs.dedup.prefetch: 1
vfs.zfs.mdcomp_disable: 0
vfs.zfs.nopwrite_enabled: 1
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.block_cap: 256
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 0
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.resilver_min_time_ms: 3000
vfs.zfs.free_min_time_ms: 1000
vfs.zfs.scan_min_time_ms: 1000
vfs.zfs.scan_idle: 50
vfs.zfs.scrub_delay: 10
vfs.zfs.resilver_delay: 10
vfs.zfs.top_maxinflight: 32
vfs.zfs.write_to_degraded: 0
vfs.zfs.mg_noalloc_threshold: 0
vfs.zfs.condense_pct: 200
vfs.zfs.metaslab.weight_factor_enable: 0
vfs.zfs.metaslab.preload_enabled: 1
vfs.zfs.metaslab.preload_limit: 3
vfs.zfs.metaslab.unload_delay: 8
vfs.zfs.metaslab.load_pct: 50
vfs.zfs.metaslab.min_alloc_size: 10485760
vfs.zfs.metaslab.df_free_pct: 4
vfs.zfs.metaslab.df_alloc_threshold: 131072
vfs.zfs.metaslab.debug_unload: 0
vfs.zfs.metaslab.debug_load: 0
vfs.zfs.metaslab.gang_bang: 131073
vfs.zfs.check_hostid: 1
vfs.zfs.spa_asize_inflation: 24
vfs.zfs.deadman_enabled: 1
vfs.zfs.deadman_checktime_ms: 5000
vfs.zfs.deadman_synctime_ms: 1000000
vfs.zfs.recover: 0
vfs.zfs.txg.timeout: 12
vfs.zfs.max_auto_ashift: 13
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 0
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.trim_on_init: 1
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.scrub_max_active: 2
vfs.zfs.vdev.scrub_min_active: 1
vfs.zfs.vdev.async_write_max_active: 10
vfs.zfs.vdev.async_write_min_active: 1
vfs.zfs.vdev.async_read_max_active: 3
vfs.zfs.vdev.async_read_min_active: 1
vfs.zfs.vdev.sync_write_max_active: 10
vfs.zfs.vdev.sync_write_min_active: 10
vfs.zfs.vdev.sync_read_max_active: 10
vfs.zfs.vdev.sync_read_min_active: 10
vfs.zfs.vdev.max_active: 1000
vfs.zfs.vdev.bio_delete_disable: 0
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.vdev.trim_max_pending: 64
vfs.zfs.vdev.trim_max_bytes: 2147483648
vfs.zfs.cache_flush_disable: 1
vfs.zfs.zil_replay_disable: 0
vfs.zfs.sync_pass_rewrite: 2
vfs.zfs.sync_pass_dont_compress: 5
vfs.zfs.sync_pass_deferred_free: 2
vfs.zfs.zio.use_uma: 0
vfs.zfs.snapshot_list_prefetch: 0
vfs.zfs.version.ioctl: 3
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 5000
vfs.zfs.version.acl: 1
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
vfs.zfs.trim.enabled: 1
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
 
My bad. I was looking at the HEAD source incorrectly assuming that these changes would then be merged back in full. Looking in release/9.3.0 it appears they have merged the new throttling logic without any of the tunables, and just #if 0'd the old sysctls that are no longer used. You'd probably have to ask on the freebsd-fs mailing list why they did it this way.

It looks like the ability to tune this part of the system is unavailable on 9.3 and 10.0. It's also still missing from stable/9 and stable/10 so I don't know when these sysctls will be available in anything other than current.
 
Depending on your need, you may also try the following tunables.

Code: 
vfs.zfs.vdev.sync_write_max_active: Maximum number of I/O requests of type sync_write active for each device
vfs.zfs.vdev.async_write_max_active: Maximum number of I/O requests of type async_write active for each device

Increasing these values should increase throughput, but also increase latency.
 
t1066 said:
Depending on your need, you may also try the following tunables.

Code: 
vfs.zfs.vdev.sync_write_max_active: Maximum number of I/O requests of type sync_write active for each device
vfs.zfs.vdev.async_write_max_active: Maximum number of I/O requests of type async_write active for each device

Increasing these values should increase throughput, but also increase latency.
Click to expand...
Hi
Thanks for your advice, sacrificing latency is not what I'm willing to see.
 
belon_cfy said:
t1066 said:
Depending on your need, you may also try the following tunables.

Code: 
vfs.zfs.vdev.sync_write_max_active: Maximum number of I/O requests of type sync_write active for each device
vfs.zfs.vdev.async_write_max_active: Maximum number of I/O requests of type async_write active for each device

Increasing these values should increase throughput, but also increase latency.
Click to expand...
Hi
Thanks for your advice, sacrificing latency is not what I'm willing to see.
Click to expand...

You can change these tunables on the command line. There is no need to reboot. So it is possible to test the result of these changes immediately. If the subsequent behaviour is not desirable, just revert back to the old value.
 
You must log in or register to reply here.
Back
Top