ZFS Disabling compression on user level

Since I use zfs with compression I start to think to disable compression everywhere on appplication/user level.
I found 3 non-needed compressions:
- freebsd packages are compressed. If fact only needed when using internet.
- logfiles are compressed
- man pages are compressed.
The interesting part is that i don't know how to configure for these things to be uncompressed by default.
Maybe you have other ideas of things which can be uncompressed using zfs filesystem.
 
I start to think to disable compression everywhere on user level.
You mean dataset level?
freebsd packages are compressed. If fact only needed when using internet.
/var/cache/pkg is not a separate dataset.

- logfiles are compressed
I would recommend not compressing the old log files. You can do that by removing the J at each line in /etc/newsyslog.conf. Uncompressed logs are easier to read and the compression is already done on the filesystem level.

man pages are compressed.
Not on a separate dataset.

The default lz4 compression is actually quite safe to turn on, it won't actually compress if there's no benefit. In any case, if you want it off, zfs set compression=off zroot/usr. But note that this will only work for new files, the original files will still be compressed.
 
I don't want to disable zfs compression where the data is already compressed.
I want to disable compression of data by the application/script where zfs already compress it.
For intance I don't need compressed man-pages. zfs compress already everything
 
I want to disable compression of data by the application/script where zfs already compress it.
You can't store packages "uncompressed", they're meant to be a single, compressed, archive. That's the whole package format. With regards to the log files, I already pointed you to the J option in /etc/newsyslog.conf.
 
- logfiles are compressed

The ones that were rotated and renamed are compressed but its far more important that CURRENT logs are compressed on the fly with ZFS because it makes them 5-20 times smaller - and that buys a lot of time before /var/log become full. Having transparent compression on /var/log is one of the best ZFS features. I really miss that on Linux boxes with EXT4 and XFS filesystems.

Also keep in mind that the default ZFS compression on zroot pool - lz4 - has 'early abort' mechanism which it will give up compression if it can not save 12.5% - so with lz4 you will ignore compressed files - best of both worlds.

That is important to remember - this means that you do not have to disable lz4 on certain datasets just because they have compressed files on them.

Hope that helps.
 
Having transparent compression on /var/log is one of the best ZFS features.
Yeah, that's why I turned off compression for the rotated logs too. Makes it easier to read them too and I don't have to resort to zless/ zgrep etc. LESSOPEN is useful but I always forget to set it.
 
When you set compression=off, you will waste loads of disk space on partially-filled records for any file that is larger than the dataset's recordsize. With the default of recordsize=128K, ten files of 130K will take up 2.5MiB instead of 1.3MiB.

The sane choice is to stop worrying and leave compression=on, or explicitly set compression=lz4 when worried about performance. If you absolutely do not want to compress any data, but you also do not want to waste space on partially-filled records, set compression=zle, which only compresses runs of zeroes, such as the padding added to partially-filled records.

If, for no good reason, you still insist on compression=off, you could crank your recordsize way up (when you're sure that files never grow beyond 1M, otherwise space waste will be huge), or crank it way down to correspond to your ashift (minimizing partial-record waste, but instead wasting space on metadata bloat, which is hard to quantify).
 
Compression at the "block" level is roughly what ZFS compression on a dataset is doing. Even if the data being stored is compressed, you have to keep in mind "block level".

Con: to do that you have cpu cycles and memory to compress/decompress
but:
Pro to that is you are transferring fewer bytes over the interface to the device. SATA is a fixed bandwidth to the device, but compressing data before you present it to the device is "turning it up to 11". You are effectively sending more than 100% of the bandwidth of real data.

Most computers probably have excess CPU and memory so the user never sees the cost of compressing/decompressing, especially for the default algorithm. My rule of thumb is "default is fine until I prove I need to change it".

Note: I accept that there are some situations (workloads, physical hardware) that are different and the cost to compress/decompress is outweighed by other factors.

The above comments about removing default compression on log files (SirDice and others) is more about usability than anything else. Sure it saves space, helps /var/log from overflowing if it's a distinct partition, but looking at the cost per TB of storage (with acknowledgement that it may be going up and everyone has their limits), I agree there are real gains by not compressing old log files.

Above is my opinion; feel free to disregard.
 
While I agree that disabling compression for rotated logs might be useful, it's for a different reason: ease of use ;)

There's no reason at all to avoid storing compressed files on a compressed ZFS dataset, as already explained above. Just don't worry about it.
 
Back
Top