Microbenchmark collection


Well-Known Member

Reaction score: 159
Messages: 413

I regularly do some kind of microbenchmarks of various kinds. I wanted to share the outcomings, and maybe some of you have some results laying around, too. You are happily invited to contribute. Be it performance benchmarks of filesystem settings, database/network settings, tools, FreeBSD vs <other_os> etc. Hopefully this leads to a thread with an interesting discussion and a nice collection of data.

I was interested in how much faster zfs send/recv is compared to rsync. I used a 3-disk raidz1 (atime=off, compression=lz4) with 3 disks Toshiba MG07ACA12TE connected internally via SATA-3 (600 MB/s) to copy our code repository (7,297,671 files, ~373 GB) to a 3-disk raidz1 (atime=off, compression=lz4) with (atime=off, compression=lz4) connected to external drives via USB 3.0 (400 MB/s). ARC was limited to 8GB and before I did warmup the cache via "find /data/code". System has 2xnvme as root, Ryzen 7 3700X CPU with 64GB ECC RAM.
  • rsync -a /data/code /ext/code-rsync1: real 475m35.261s; user 5m57.723s; sys 28m35.892s
  • zfs send data/code@snap | zfs recv ext/code-zfssync1: real 86m17.891s; user 0m0.000s; sys 7m54.837s
  • zfs send data/code@snap | mbuffer -s 128k -m 2G -o - | zfs recv ext/code-mbuffer2: real 87m44.900s; user 0m15.359s; sys 11m7.539s
  • zfs send data/code@snap | mbuffer -s 128m -m 2G -o - | zfs recv ext/code-mbuffer3: real 82m9.039s; user 0m10.456s; sys 10m22.156s
Last edited by a moderator:


New Member

Messages: 3

just stumbled on this thread looking for ZFS checksum microbenchmark data.

i am running a TrueNAS 12.0 but i don't have /proc there so i cannot access /proc/spl/kstat/zfs - maybe you see some numbers there?
(adding /proc to /etc/fstab allows mounting /proc, but i do not see /proc/spl/kstat/zfs in there)
see also https://openzfs.github.io/openzfs-docs/Basic Concepts/Checksums.html

i was mainly interested in flechter4 vs sha-512 given the recent speedup of the latter


Well-Known Member

Reaction score: 159
Messages: 413

will try to benchmark this when I have time.

another rsync vs zfs send/recv benchmark:
~ 146 GB, 248118 files of jpgs with lower than 3.6 MP resolution; target is an encrypted zfs dataset.
rsync from one external 7.2k disk to the other: real 104m34.104s, user 1m33.669s, sys 5m52.857s
zfs send/recv: real 60m9.500s, user 0m0.000s, sys 1m11.733s


Aspiring Daemon

Reaction score: 334
Messages: 547

I think the time benefit of ZFS send/recv winds up in the incrementals. An initial snapshot and send/recv is going to take the longest because it has the most data.
From then on incremental snapshots and send/recv is only sending the changes.

Why is that important? I believe rsync processes things differently, as in still at least walks the full source to compare against destination.