ZFS ARC compressed ratio too good to be true?

Bobi B.

Well-Known Member

Reaction score: 155
Messages: 371

This is from a server running FreeBSD 12.1
Code:
last pid:  7433;  load averages:  1.86,  1.77,  1.79    up 8+17:19:56  08:11:03
111 processes: 2 running, 109 sleeping
CPU:  0.9% user, 13.8% nice,  3.1% system,  0.0% interrupt, 82.1% idle
Mem: 147M Active, 111M Inact, 312M Laundry, 21G Wired, 28M Buf, 14G Free
ARC: 19G Total, 1232M MFU, 18G MRU, 24M Anon, 108M Header, 249M Other
     18G Compressed, 34G Uncompressed, 1.91:1 Ratio
Swap:
with 36 GB of RAM, with two pools
Code:
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
vol1   556G   347G   209G        -         -    36%    62%  1.00x  ONLINE  -
vol2   278G  84.5G   194G        -         -     1%    30%  1.00x  ONLINE  -
storing mostly MPEG-TS files and working as a video server. Due to the fact that video and audio in MPEG-TS files are already compressed, this 1.91:1 ratio seems quite off. A quick compressibility test such as
Bash:
#!/bin/sh

[ $# -gt 0 ] || {
    echo "Usage: $0 <dir>"
    exit 1
}

find "$1" -type f | \
    while read path; do
    sz=$(stat -f%z "${path}") || continue
    csz=$(gzip -1 -c "${path}" | wc -c) || continue
    f=$((${csz}\*100/${sz}))
    printf "%3d\t%s\n" ${f} "${path}"
    done
yielded factors in 90-95% range. Is there a way to dig deeper in the ARC cache to see how can ZFS shrink this data nearly two-fold?

Or someone is lying and those 14G Free are also used for ARC cache: 18G + 14G = 32G, roughly 95%, but not according to top(1).
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 1,104
Messages: 2,113

Also, ARC is the cache that also holds the virtual memory pages for swapping. You need to check the file system with "zfs get ..." for the compression of the on-disc data.
 

PMc

Aspiring Daemon

Reaction score: 221
Messages: 618

Look how much things happen to be compressed on-disk: zfs get compressratio
Look how much of your arc is actually metadata: sysctl kstat.zfs.misc.arcstats.arc_meta_used
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 1,104
Messages: 2,113

Thank you for the details. These days I'm more often than not on my phone typing...
 
OP
OP
Bobi B.

Bobi B.

Well-Known Member

Reaction score: 155
Messages: 371

The compression for this pool is now enabled. Since that is a testing time-shifting storage server, all content in the pool is fully replaced in 24 hours.

Currently compressratio is 1.07x, which is roughly in the expected 95% (5% saved because of compression). Occupied space is 322G in 160K files. Yet
Code:
last pid: 16468;  load averages: 13.86, 13.46, 13.08          up 9+19:40:56  10:32:03
110 processes: 3 running, 107 sleeping
CPU:  1.2% user, 94.2% nice,  4.4% system,  0.2% interrupt,  0.0% idle
Mem: 391M Active, 267M Inact, 268M Laundry, 20G Wired, 43M Buf, 14G Free
ARC: 18G Total, 1532M MFU, 16G MRU, 25M Anon, 106M Header, 249M Other
     16G Compressed, 34G Uncompressed, 2.05:1 Ratio
and I wonder where this 2.05:1 compression ratio comes from. I simply cannot believe a compressed video and audio media can be additionally compressed nearly twofold. Finally kstat.zfs.misc.arcstats.arc_meta_used: 897269696.
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 1,104
Messages: 2,113

As I said, that cache not only holds your media but also swap cache and other data. This is NOT only your video data, and when that is written only once and never read back the ARC may not cache it at all.

The 5% you got from the ZFS are what your files are compressed.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,444
Messages: 32,482

Keep in mind that large, sequentially read, files aren't cached. That would be a waste of cache memory.
 
OP
OP
Bobi B.

Bobi B.

Well-Known Member

Reaction score: 155
Messages: 371

I saw what you've wrote. I feel stupid, however I simply cannot tell where did ZFS found those 34GB of compressible data in the pool. The pool contains only data that cannot be compressed to that extent. Even if we take into account the pool metadata, this ratio (nearly 2:1) seems far too large for the amount of data (34GB!) in the ARC.
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 1,104
Messages: 2,113

Do a 'swapoff' and see what happens. ARC may contain virtual memory pages, loaded binary data, web pages, you name it. It's not the file system. Do you swap on a vdev?
 
OP
OP
Bobi B.

Bobi B.

Well-Known Member

Reaction score: 155
Messages: 371

There is no swap on this server. Also there are no executables, web pages, whatever else stored on ZFS; apart from MPEG-TS media files. Even if there is something else it is in megabytes range, not gigabytes. OS itself is on UFS.
 

Eric A. Borisch

Well-Known Member

Reaction score: 282
Messages: 478

Do a 'swapoff' and see what happens. ARC may contain virtual memory pages, loaded binary data, web pages, you name it. It's not the file system. Do you swap on a vdev?
Do you have a reference for the ARC containing anything other than ZFS (file/meta) data? Thanks.
 

PMc

Aspiring Daemon

Reaction score: 221
Messages: 618

Well, yes, this would give me headaches, too. ;) But then, as long as the machine works well, maybe one should just not wonder about it. Usually, with ZFS, those things enravel to something sensible only if you dig quite deep into it.

Certainly all the things in ZFS can be observed. The usual way to do so is to engage dtrace, which implies to read and understand a bit of the source.
 
OP
OP
Bobi B.

Bobi B.

Well-Known Member

Reaction score: 155
Messages: 371

There you go:
Code:
kstat.zfs.misc.arcstats.demand_hit_prescient_prefetch: 0
kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 2472201
kstat.zfs.misc.arcstats.async_upgrade_sync: 974
kstat.zfs.misc.arcstats.arc_meta_min: 2284163584
kstat.zfs.misc.arcstats.arc_meta_max: 971147392
kstat.zfs.misc.arcstats.arc_dnode_limit: 913665433
kstat.zfs.misc.arcstats.arc_meta_limit: 9136654336
kstat.zfs.misc.arcstats.arc_meta_used: 765543112
kstat.zfs.misc.arcstats.arc_prune: 0
kstat.zfs.misc.arcstats.arc_loaned_bytes: 0
kstat.zfs.misc.arcstats.arc_tempreserve: 0
kstat.zfs.misc.arcstats.arc_no_grow: 0
kstat.zfs.misc.arcstats.memory_available_bytes: 0
kstat.zfs.misc.arcstats.memory_free_bytes: 0
kstat.zfs.misc.arcstats.memory_all_bytes: 0
kstat.zfs.misc.arcstats.memory_indirect_count: 0
kstat.zfs.misc.arcstats.memory_direct_count: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0
kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0
kstat.zfs.misc.arcstats.l2_write_pios: 0
kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0
kstat.zfs.misc.arcstats.l2_write_full: 0
kstat.zfs.misc.arcstats.l2_write_not_cacheable: 56613
kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0
kstat.zfs.misc.arcstats.l2_write_in_l2: 0
kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0
kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0
kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.l2_asize: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_evict_l1cached: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_write_bytes: 0
kstat.zfs.misc.arcstats.l2_read_bytes: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 559616
kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 9130213376
kstat.zfs.misc.arcstats.mfu_ghost_size: 9130772992
kstat.zfs.misc.arcstats.mfu_evictable_metadata: 11118080
kstat.zfs.misc.arcstats.mfu_evictable_data: 6265503744
kstat.zfs.misc.arcstats.mfu_size: 6378239488
kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 0
kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 6799832576
kstat.zfs.misc.arcstats.mru_ghost_size: 6799832576
kstat.zfs.misc.arcstats.mru_evictable_metadata: 228568064
kstat.zfs.misc.arcstats.mru_evictable_data: 11349404672
kstat.zfs.misc.arcstats.mru_size: 13574040064
kstat.zfs.misc.arcstats.anon_evictable_metadata: 0
kstat.zfs.misc.arcstats.anon_evictable_data: 0
kstat.zfs.misc.arcstats.anon_size: 43598336
kstat.zfs.misc.arcstats.other_size: 241845384
kstat.zfs.misc.arcstats.bonus_size: 57481280
kstat.zfs.misc.arcstats.dnode_size: 137455864
kstat.zfs.misc.arcstats.dbuf_size: 46908240
kstat.zfs.misc.arcstats.metadata_size: 419584000
kstat.zfs.misc.arcstats.data_size: 19576293888
kstat.zfs.misc.arcstats.hdr_size: 104113728
kstat.zfs.misc.arcstats.overhead_size: 1163627520
kstat.zfs.misc.arcstats.uncompressed_size: 35421092352
kstat.zfs.misc.arcstats.compressed_size: 18837722624
kstat.zfs.misc.arcstats.size: 20341837000
kstat.zfs.misc.arcstats.c_max: 36546617344
kstat.zfs.misc.arcstats.c_min: 4568327168
kstat.zfs.misc.arcstats.c: 20270898035
kstat.zfs.misc.arcstats.p: 14311930941
kstat.zfs.misc.arcstats.hash_chain_max: 4
kstat.zfs.misc.arcstats.hash_chains: 9541
kstat.zfs.misc.arcstats.hash_collisions: 2830590
kstat.zfs.misc.arcstats.hash_elements_max: 463440
kstat.zfs.misc.arcstats.hash_elements: 402691
kstat.zfs.misc.arcstats.evict_l2_skip: 0
kstat.zfs.misc.arcstats.evict_l2_ineligible: 7420379136
kstat.zfs.misc.arcstats.evict_l2_eligible: 4075799131648
kstat.zfs.misc.arcstats.evict_l2_cached: 0
kstat.zfs.misc.arcstats.evict_not_enough: 12330
kstat.zfs.misc.arcstats.evict_skip: 100193
kstat.zfs.misc.arcstats.access_skip: 3062267200
kstat.zfs.misc.arcstats.mutex_miss: 4174
kstat.zfs.misc.arcstats.deleted: 29996214
kstat.zfs.misc.arcstats.allocated: 191595884
kstat.zfs.misc.arcstats.mfu_ghost_hits: 1097813
kstat.zfs.misc.arcstats.mfu_hits: 2717236958
kstat.zfs.misc.arcstats.mru_ghost_hits: 5652
kstat.zfs.misc.arcstats.mru_hits: 63370989
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 1296
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 48769
kstat.zfs.misc.arcstats.prefetch_data_misses: 2530165
kstat.zfs.misc.arcstats.prefetch_data_hits: 1008485
kstat.zfs.misc.arcstats.demand_metadata_misses: 2543232
kstat.zfs.misc.arcstats.demand_metadata_hits: 2771486541
kstat.zfs.misc.arcstats.demand_data_misses: 5501
kstat.zfs.misc.arcstats.demand_data_hits: 8423339
kstat.zfs.misc.arcstats.misses: 5080194
kstat.zfs.misc.arcstats.hits: 2780967134
top(1) output:
Code:
last pid: 41029;  load averages:  5.08,  2.93,  1.43                                                                         up 12+00:50:14  15:41:21
110 processes: 1 running, 109 sleeping
CPU:  0.3% user, 27.9% nice,  3.7% system,  0.0% interrupt, 68.1% idle
Mem: 304M Active, 289M Inact, 261M Laundry, 20G Wired, 49M Buf, 14G Free
ARC: 19G Total, 6039M MFU, 13G MRU, 38M Anon, 99M Header, 231M Other
     17G Compressed, 33G Uncompressed, 1.89:1 Ratio
Swap:
 
Top