ZFS ZFS record size 16M

[swegen]

I were benchmarking performance of UFS vs ZFS for a single USB 3.1 SSD. UFS seems to be a clear winner with more efficient handling of small files. Too bad TRIM cannot be enabled with a solid state drive inside a USB enclosure? At least camcontrol secure erase works well to clear memory cells to restore factory default write performance. This is with FreeBSD 11.1-RELEASE-p1.

When playing around with ZFS record size I noticed it is possible to go over 1M by adjusting vfs.zfs.max_recordsize. By trial and error I found that maximum ZFS record size is 16M: 'recordsize' must be power of 2 from 512B to 16384KB. It can be set like this:

sysctl vfs.zfs.max_recordsize=16777216
zfs set recordsize=16M pool

But then I noticed something weird. Directories cannot be created if the record size is larger than 4M:

# mkdir x
mkdir: x: File too large

I found related error on FreeBSD filesystems mailing list: ZFS: mkdir: File too large.

I even tried disabling compression and checksumming but result is the same. It seems that these larger record sizes 8M and 16M should be supported but there are errors?

 

[Eric A. Borisch]

Do you feel there is much to gain going above the (advertised in the zfs man page, but perhaps out of date?) max of 1MB?

 

[swegen]

Maybe it could be useful with very large pools. There would be up to 16 times less blocks for ZFS to keep track of and possibly the lz4 compression is more efficient with larger compressible blocks.

There is also this:

# sysctl -d vfs.zfs.max_recordsize
vfs.zfs.max_recordsize: Maximum block size.  Expect dragons when tuning this.

Best not to touch this tunable at least with production systems.

 

[Eric A. Borisch]

Yes, but you risk smaller blocks being stuck behind large blocks for decompression / processing. The incremental reduction in absolute % overhead is a diminishing return for ever larger blocks, too.