Need assistance with sluggish ZFS host

Alright, after some more diagnostic:

- Running with 256G ARC is smooth after adding
Code: 
vfs.zfs.arc_shrink_shift: 11
Before adding it I was unable to complete rsync of 800G if ARC was over 64G.
Thinking about setting 12 and rising ARC to 512G

- Re-enabling trim killed it again on untaring. I disabled it and I am retesting it now to confirm.

Is there any manual for these tunes?
Code: 
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 0
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1

I believe trim might be tuned back to normal as these SAS3 drives don't lock on trim as SATAs do.
 
  • Thanks
Reactions: Oko
After a few more repeated testings my conclusion is:
- with ARC up to 64G it does not die
- next value tested is 128G and I can kill it every time
- shift fixed issue on reading, outbound rsync works even up to 256G ARC (never tested further)
- same rsync inbound and untar of exactly the same InnoDB files to second location kills zfs
- at the end everything waits for arc_reclaim which does not reclaim, also ARC is slightly over its limit
- there is interesting side effect - rsync from remote machine and untar from local file both runs at half speed until ARC hits its limit (no matter what this limit is)
- in case 128G ARC there is around 5k evict skips since reboot, in case 64G ARC after completed untar/rsync there is around 13k skips left

I have found linux case (https://github.com/zfsonlinux/zfs/issues/4726) which seems half relevant - saying it stops evicting at some point even if that is more related to size of metadata cache. However, I can't find any similar tunes to what they used to overcome issue on FreeBSD.
I use a few FreeNAS setups, none has that issue, however, all are 64G of memory. However, I can rsync from them many times with limited number of skips, etc. They use custom kernel.

What would you recommend as next step? Open a bug? Some kernel debugging?
Does anyone use large ARC (256G and over)?
 
You must log in or register to reply here.
Back
Top