ZFS Slow zfs send/receive on incremental snapshots

I am trying to migrate a zpool's (backups) dataset to znother zpool (backupR) . Both are on the same host

I did:
zfs snap -r backups@xfer
zfs send -R backups@xfer | zfs recv -Fuv backupR

receiving full stream of backups@xfer into backupR@xfer
received 49.9K stream in 1 seconds (49.9K/sec)
receiving full stream of backups/mfs@mfs-20231128182820 into backupR/mfs@mfs-20231128182820
received 1.05T stream in 2104 seconds (524M/sec)
receiving incremental stream of backups/mfs@xfer into backupR/mfs@xfer
received 312B stream in 1 seconds (312B/sec)
receiving full stream of backups/mail@20230806 into backupR/mail@20230806
received 136T stream in 200121 seconds (711M/sec)
receiving incremental stream of backups/mail@20230903 into backupR/mail@20230903


It copied the full stream fairly quickly with visible activity on the CPU/disks but now in the incremental which is only 5T or so, there's almost 0 activity on the CPU and disks and it's been at it for an entire day. What is it doing and why is it so slow ? The machine is fast enough as I maxed out the 6gbps backplane during the full stream transfer.

top just shows:

65796 root 2 20 0 18M 8464K q->bq_ 20 22.2H 1.46% zfs
65795 root 4 20 0 20M 9616K pipewr 4 24.5H 0.62% zfs
 
So it finally completed the incremental but @ 7x slower speed

receiving full stream of backups@xfer into backupR@xfer
received 49.9K stream in 1 seconds (49.9K/sec)
receiving full stream of backups/mfs@mfs-20231128182820 into backupR/mfs@mfs-20231128182820
received 1.05T stream in 2104 seconds (524M/sec)
receiving incremental stream of backups/mfs@xfer into backupR/mfs@xfer
received 312B stream in 1 seconds (312B/sec)
receiving full stream of backups/mail@20230806 into backupR/mail@20230806
received 136T stream in 200121 seconds (711M/sec)
receiving incremental stream of backups/mail@20230903 into backupR/mail@20230903
received 11.7T stream in 112911 seconds (108M/sec)
receiving incremental stream of backups/mail@20230910 into backupR/mail@20230910


Why would incrementals be slower than the full stream ?
 
Back
Top