vfs.zfs.arc_max has been set to 4G from /boot/loader.conf:
Yet memory usage is way above that and keeps growing.
Any reason for that? Some info that might be useful:
Please see how arc_meta_max is way above arc_meta_limit:
There are around 15 million files served (it's an image caching server), but other servers don't exhibit this problem, their ARC usage stays within limits.
Code:
vfs.zfs.arc_max="4G"
Code:
ARC: 25G Total, 8461M MFU, 391M MRU, 24M Anon, 132M Header, 16G Other
1823M Compressed, 7054M Uncompressed, 3.87:1 Ratio
Please see how arc_meta_max is way above arc_meta_limit:
There are around 15 million files served (it's an image caching server), but other servers don't exhibit this problem, their ARC usage stays within limits.
Code:
$ freebsd-version -uk
11.3-RELEASE-p3
11.3-RELEASE-p4
Code:
$ sysctl -a|fgrep -i arc
device arcmsr
kern.supported_archs: amd64 i386
vfs.zfs.arc_min_prescient_prefetch_ms: 6
vfs.zfs.arc_min_prefetch_ms: 1
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
[B]vfs.zfs.arc_meta_limit: 1073741824[/B]
vfs.zfs.arc_free_target: 339086
vfs.zfs.arc_kmem_cache_reap_retry_ms: 1000
vfs.zfs.compressed_arc_enabled: 1
vfs.zfs.arc_grow_retry: 60
vfs.zfs.arc_shrink_shift: 7
vfs.zfs.arc_average_blocksize: 8192
vfs.zfs.arc_no_grow_shift: 5
vfs.zfs.arc_min: 4294967296
vfs.zfs.arc_max: 4294967296
vfs.ffs.maxclustersearch: 10
debug.adaptive_machine_arch: 1
hw.machine_arch: amd64
kstat.zfs.misc.arcstats.demand_hit_prescient_prefetch: 0
kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 52697
kstat.zfs.misc.arcstats.async_upgrade_sync: 212
kstat.zfs.misc.arcstats.arc_meta_min: 2147483648
[B]kstat.zfs.misc.arcstats.arc_meta_max: 27683785944
kstat.zfs.misc.arcstats.arc_meta_limit: 1073741824
kstat.zfs.misc.arcstats.arc_meta_used: 26997506960[/B]
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0
kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0
kstat.zfs.misc.arcstats.l2_write_pios: 0
kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0
kstat.zfs.misc.arcstats.l2_write_full: 0
kstat.zfs.misc.arcstats.l2_write_not_cacheable: 50295
kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0
kstat.zfs.misc.arcstats.l2_write_in_l2: 0
kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0
kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0
kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.l2_asize: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_evict_l1cached: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_write_bytes: 0
kstat.zfs.misc.arcstats.l2_read_bytes: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 284554752
kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 4057600
kstat.zfs.misc.arcstats.mfu_ghost_size: 288612352
kstat.zfs.misc.arcstats.mfu_evictable_metadata: 0
kstat.zfs.misc.arcstats.mfu_evictable_data: 0
kstat.zfs.misc.arcstats.mfu_size: 8860957184
kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 3858164224
kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 12436992
kstat.zfs.misc.arcstats.mru_ghost_size: 3870601216
kstat.zfs.misc.arcstats.mru_evictable_metadata: 0
kstat.zfs.misc.arcstats.mru_evictable_data: 0
kstat.zfs.misc.arcstats.mru_size: 414954496
kstat.zfs.misc.arcstats.anon_evictable_metadata: 0
kstat.zfs.misc.arcstats.anon_evictable_data: 0
kstat.zfs.misc.arcstats.anon_size: 4942336
kstat.zfs.misc.arcstats.other_size: 17661001392
kstat.zfs.misc.arcstats.metadata_size: 9197769728
kstat.zfs.misc.arcstats.data_size: 83084288
kstat.zfs.misc.arcstats.hdr_size: 138735840
kstat.zfs.misc.arcstats.overhead_size: 7385242112
kstat.zfs.misc.arcstats.uncompressed_size: 7385242112
kstat.zfs.misc.arcstats.compressed_size: 1895620608
kstat.zfs.misc.arcstats.size: 27080591248
kstat.zfs.misc.arcstats.c_max: 4294967296
kstat.zfs.misc.arcstats.c_min: 4294967296
kstat.zfs.misc.arcstats.c: 4294967296
kstat.zfs.misc.arcstats.p: 3648431616
kstat.zfs.misc.arcstats.hash_chain_max: 3
kstat.zfs.misc.arcstats.hash_chains: 3643
kstat.zfs.misc.arcstats.hash_collisions: 579973
kstat.zfs.misc.arcstats.hash_elements_max: 519990
kstat.zfs.misc.arcstats.hash_elements: 493266
kstat.zfs.misc.arcstats.evict_l2_skip: 0
kstat.zfs.misc.arcstats.evict_l2_ineligible: 858988544
kstat.zfs.misc.arcstats.evict_l2_eligible: 1462018226176
kstat.zfs.misc.arcstats.evict_l2_cached: 0
kstat.zfs.misc.arcstats.evict_not_enough: 21387547
kstat.zfs.misc.arcstats.evict_skip: 1215126917
kstat.zfs.misc.arcstats.access_skip: 729118541
kstat.zfs.misc.arcstats.mutex_miss: 2682824
kstat.zfs.misc.arcstats.deleted: 17533423
kstat.zfs.misc.arcstats.allocated: 312640665
kstat.zfs.misc.arcstats.mfu_ghost_hits: 1102265
kstat.zfs.misc.arcstats.mfu_hits: 1043470003
kstat.zfs.misc.arcstats.mru_ghost_hits: 11899899
kstat.zfs.misc.arcstats.mru_hits: 43401992
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 558724
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 233008
kstat.zfs.misc.arcstats.prefetch_data_misses: 11803
kstat.zfs.misc.arcstats.prefetch_data_hits: 12
kstat.zfs.misc.arcstats.demand_metadata_misses: 31499035
kstat.zfs.misc.arcstats.demand_metadata_hits: 1084166851
kstat.zfs.misc.arcstats.demand_data_misses: 865029
kstat.zfs.misc.arcstats.demand_data_hits: 2581257
kstat.zfs.misc.arcstats.misses: 32934591
kstat.zfs.misc.arcstats.hits: 1086981128