In general, I almost always have lz4 enabled, there's very few disadvantages. At a minimum though, I'd use zle instead of off; it at least makes sure files will be sparse if they can be.
If your system has the zstd feature available, I'd highly recommend using that instead of gzip. It's basically better in every way (better ratio
and less CPU time). In some circumstances, it can replace lz4, especially if you have zstd-fast-1 or so (default is zstd-3).
My opinion of the compression algorithms available on ZFS:
off = guarantees files take up space and don't automatically become sparse on runs of zeros, just like UFS would do.
zle = blocks of zeros don't get stored to disk, otherwise equivalent to
off - I use this on my swap zvol.
lzjb = the original fast ZFS compression algorithm. Mostly useless now except on old pool versions.
lz4 = current default for compression=on with an extremely good ratio and CPU time tradeoff. Fully replaces lzjb and mostly a good idea to enable on a root dataset (if your pool is sufficiently general-purpose anyway)
gzip = old algorith offering decent ratio but severely expensive in CPU time. Mostly only good for old pool versions.
zstd = Extremely flexible algorithm nearly rivaling lz4 in speed (the fast levels can actually achieve similar benchmarks). Ratio is much better than any of the prior algorithms, even the default
zstd-3 is very fast and achieves better compression ratio than
gzip-9 can.
From this list, my main considerations in
any circumstance are zle, lz4, and zstd. It comes down a lot to the write performance characteristics desired. I do even have a
zstd-19 file system; very rarely do I write to it, so it achieves an extreme level of compression while not being demanding on read.
A large recordsize generally gives you better compression and less metadata overhead, but space is wasted on partly-filled records if compression is off
recordsize advises ZFS of the largest record size to use for file blocks, but small enough files can and will use the smallest unit available on the hardware (typically 512 or 4096 bytes).