All,
Notably compression may be handled differently between the two, however wanted to know if anyone has reference or opinion as to whether one would expect these two issues that I've noticed: 1) difference in "consumption" and 2) constant creeping into swap space.
* consumption *
Noticed this while replicating data from an existing ZFS server to it's replacement. Replicated data via cpio and running rsync multiple times to discern any differences and ensure that files were "copied" correctly - all via 11.x doing NFS mount from 10.x. For a given ZFS file system, the number of files and sizes appear to be "same", but storage consumption in most cases is different.
In the list below, the "consumption" is obtained via "zfs list", as one might expect "df" to have a different opinion, no? Here are some examples from the various zfs file systems.
Filesystem: 10.x: 11.x: Usage:
"A" : 255GB : 258GB : Most data is already compressed (zip/arj/etc).
"B" : 104GB : 104GB : Video
"C" : 253GB : 243GB : Data runs the gambit (home directories)
"D" : 497GB : 498GB : ISO type files
"E" : 347GB : 366GB : Image files
"F" : 116GB : 116GB : Image files
"G" : 1.36TB : 1.4TB : Video
Trying to determine why the delta - especially in cases where it's 10+GBs. If these types of deltas seem "sane" - OK. If someone suspects an issue, would like to know. Have tried to validate files, but can't seem to find any issue. Find it odd that in some cases it's more, others less and some same. Looking for a proverbial sanity check.
Config:
* swap space *
Also noticed that no matter what is done (via /boot/loader.conf and /etc/sysctl.conf) that there is always swap consumption. Quick overview:
Quick snippet from the output of top:
Have seen:
However, don't want to start "turning knobs" without understanding the implications and "rhyme/reason" behind how these should be set (if possible) to help properly limit the MFU and MRU. Thinking that MRU and MFU should be tuned, but to what values? Looking to determine a reasonable equation for:
(total memory) - (memory for applications) = (arc size) + (mru size) + (mfu size) + ???
Also hoping for any guidance on how related parameters should be adjusted (if needed) so that everything falls into place.
Tunings germane to ZFS:
/boot/loader.conf:
/etc/sysctl.conf:
Thanks!
Notably compression may be handled differently between the two, however wanted to know if anyone has reference or opinion as to whether one would expect these two issues that I've noticed: 1) difference in "consumption" and 2) constant creeping into swap space.
* consumption *
Noticed this while replicating data from an existing ZFS server to it's replacement. Replicated data via cpio and running rsync multiple times to discern any differences and ensure that files were "copied" correctly - all via 11.x doing NFS mount from 10.x. For a given ZFS file system, the number of files and sizes appear to be "same", but storage consumption in most cases is different.
In the list below, the "consumption" is obtained via "zfs list", as one might expect "df" to have a different opinion, no? Here are some examples from the various zfs file systems.
Filesystem: 10.x: 11.x: Usage:
"A" : 255GB : 258GB : Most data is already compressed (zip/arj/etc).
"B" : 104GB : 104GB : Video
"C" : 253GB : 243GB : Data runs the gambit (home directories)
"D" : 497GB : 498GB : ISO type files
"E" : 347GB : 366GB : Image files
"F" : 116GB : 116GB : Image files
"G" : 1.36TB : 1.4TB : Video
Trying to determine why the delta - especially in cases where it's 10+GBs. If these types of deltas seem "sane" - OK. If someone suspects an issue, would like to know. Have tried to validate files, but can't seem to find any issue. Find it odd that in some cases it's more, others less and some same. Looking for a proverbial sanity check.
Config:
- recordsize = 128K (both systems)
- checksum = fletcher4 (both systems)
- compression = lz4 (both systems)
* swap space *
Also noticed that no matter what is done (via /boot/loader.conf and /etc/sysctl.conf) that there is always swap consumption. Quick overview:
- 128GB of RAM
- E5-2630
- 22 x Seagate ST4000NM0095 (data drives, 2 x RAIDZ3 in pool)
- 2 x Samsung 850 Pro 512GB (gmirror for OS and apps)
- 2 x Samsung 850 Pro 256GB (dedicated L2ARC)
- 2 x Samsung 850 Pro 256GB (dedicated ZIL - oversized, but met price point)
Quick snippet from the output of top:
Code:
Mem: 517M Active, 79G Inact, 44G Wired, 1306M Buf, 1396M Free
ARC: 33G Total, 8293M MFU, 19G MRU, 16K Anon, 603M Header, 5260M Other
Swap: 128G Total, 208M Used, 128G Free
Code:
vfs.zfs.mru_ghost_data_lsize
vfs.zfs.mru_ghost_metadata_lsize
vfs.zfs.mru_ghost_size
vfs.zfs.mru_data_lsize
vfs.zfs.mru_metadata_lsize
vfs.zfs.mru_size
vfs.zfs.mfu_ghost_data_lsize
vfs.zfs.mfu_ghost_metadata_lsize
vfs.zfs.mfu_ghost_size
vfs.zfs.mfu_data_lsize
vfs.zfs.mfu_metadata_lsize
vfs.zfs.mfu_size(total memory) - (memory for applications) = (arc size) + (mru size) + (mfu size) + ???
Also hoping for any guidance on how related parameters should be adjusted (if needed) so that everything falls into place.
Tunings germane to ZFS:
/boot/loader.conf:
Code:
vfs.zfs.arc_max="68G"
vfs.zfs.l2arc_noprefetch="0"
vfs.zfs.l2arc_write_boost="524288000"
vfs.zfs.l2arc_write_max="268435456"
vfs.zfs.txg.synctime_ms="3000"
vfs.zfs.txg.timeout="3"
vfs.zfs.vdev.cache.size="64M"
vfs.zfs.vdev.cache.max="65536"
vm.kmem_size="192G"
vm.kmem_size_max="256GB"/etc/sysctl.conf:
Code:
vfs.zfs.min_auto_ashift=12
vfs.zfs.no_scrub_prefetch=1
vfs.zfs.resilver_delay=1
vfs.zfs.resilver_min_time_ms=5000
vfs.zfs.scrub_delay=3
vfs.zfs.top_maxinflight=128