ZFS 11.0p9 vs. 10.0p18 - Expected differences?

scorpion7

New Member


Messages: 8

All,

Notably compression may be handled differently between the two, however wanted to know if anyone has reference or opinion as to whether one would expect these two issues that I've noticed: 1) difference in "consumption" and 2) constant creeping into swap space.

* consumption *
Noticed this while replicating data from an existing ZFS server to it's replacement. Replicated data via cpio and running rsync multiple times to discern any differences and ensure that files were "copied" correctly - all via 11.x doing NFS mount from 10.x. For a given ZFS file system, the number of files and sizes appear to be "same", but storage consumption in most cases is different.

In the list below, the "consumption" is obtained via "zfs list", as one might expect "df" to have a different opinion, no? Here are some examples from the various zfs file systems.

Filesystem: 10.x: 11.x: Usage:
"A" : 255GB : 258GB : Most data is already compressed (zip/arj/etc).
"B" : 104GB : 104GB : Video
"C" : 253GB : 243GB : Data runs the gambit (home directories)
"D" : 497GB : 498GB : ISO type files
"E" : 347GB : 366GB : Image files
"F" : 116GB : 116GB : Image files
"G" : 1.36TB : 1.4TB : Video

Trying to determine why the delta - especially in cases where it's 10+GBs. If these types of deltas seem "sane" - OK. If someone suspects an issue, would like to know. Have tried to validate files, but can't seem to find any issue. Find it odd that in some cases it's more, others less and some same. Looking for a proverbial sanity check.

Config:
  • recordsize = 128K (both systems)
  • checksum = fletcher4 (both systems)
  • compression = lz4 (both systems)


* swap space *
Also noticed that no matter what is done (via /boot/loader.conf and /etc/sysctl.conf) that there is always swap consumption. Quick overview:
  • 128GB of RAM
  • E5-2630
  • 22 x Seagate ST4000NM0095 (data drives, 2 x RAIDZ3 in pool)
  • 2 x Samsung 850 Pro 512GB (gmirror for OS and apps)
  • 2 x Samsung 850 Pro 256GB (dedicated L2ARC)
  • 2 x Samsung 850 Pro 256GB (dedicated ZIL - oversized, but met price point)
Have tried backing down ARC through vfs.zfs.arc_max, however even moving down to 68GB still nets (currently) 208M in swap. Usually after 24-48 hours of uptime, start seeing some swap consumption. Even tried 64GB for arc_max, but still ran into swap space. To date, has been <1GB - but hoping to curb before it presents a problem (if it does). MRU and MFU appear to keep growing - perhaps source of RAM consumption? Hopeful target is tune system to where there are a couple GBs of "available" RAM for application use are available with zero ("0") swap consumption. Considering addition of another 128GB of RAM, but first want to make sure that things are properly limited and configured before adding more RAM - then adjust for additional memory.

Quick snippet from the output of top:
Code:
Mem: 517M Active, 79G Inact, 44G Wired, 1306M Buf, 1396M Free
ARC: 33G Total, 8293M MFU, 19G MRU, 16K Anon, 603M Header, 5260M Other
Swap: 128G Total, 208M Used, 128G Free
Have seen:
Code:
vfs.zfs.mru_ghost_data_lsize
vfs.zfs.mru_ghost_metadata_lsize
vfs.zfs.mru_ghost_size
vfs.zfs.mru_data_lsize
vfs.zfs.mru_metadata_lsize
vfs.zfs.mru_size

vfs.zfs.mfu_ghost_data_lsize
vfs.zfs.mfu_ghost_metadata_lsize
vfs.zfs.mfu_ghost_size
vfs.zfs.mfu_data_lsize
vfs.zfs.mfu_metadata_lsize
vfs.zfs.mfu_size
However, don't want to start "turning knobs" without understanding the implications and "rhyme/reason" behind how these should be set (if possible) to help properly limit the MFU and MRU. Thinking that MRU and MFU should be tuned, but to what values? Looking to determine a reasonable equation for:

(total memory) - (memory for applications) = (arc size) + (mru size) + (mfu size) + ???

Also hoping for any guidance on how related parameters should be adjusted (if needed) so that everything falls into place.

Tunings germane to ZFS:

/boot/loader.conf:
Code:
vfs.zfs.arc_max="68G"
vfs.zfs.l2arc_noprefetch="0"
vfs.zfs.l2arc_write_boost="524288000"
vfs.zfs.l2arc_write_max="268435456"
vfs.zfs.txg.synctime_ms="3000"
vfs.zfs.txg.timeout="3"
vfs.zfs.vdev.cache.size="64M"
vfs.zfs.vdev.cache.max="65536"
vm.kmem_size="192G"
vm.kmem_size_max="256GB"
/etc/sysctl.conf:
Code:
vfs.zfs.min_auto_ashift=12
vfs.zfs.no_scrub_prefetch=1
vfs.zfs.resilver_delay=1
vfs.zfs.resilver_min_time_ms=5000
vfs.zfs.scrub_delay=3
vfs.zfs.top_maxinflight=128
Thanks!
 
Top