ZFS ZFS receive hangs when receiving incremental stream

I have a weird situation where trying to receive an incremental stream generated with zfs send on a Linux system hangs. Specifically, it gets as far as processing the first snapshot, but not past that. The receive process is stuck in wait channel "rwa.cv", which I can't find any information about:

  0 2428 2421   0  21  0 40336 4060 rwa.cv D+    3  0:04.90 zfs receive...
This happened the first time over ssh, but I have also tried dumping the stream to a file, copying the file to the target system and receiving it from there, with the same result. I've left the process running for a few hours, but according to top and zpool iostat nothing is happening. The only way to kill the process is to reboot the machine.

I have run the same receive command with -n and that completed successfully in a few minutes (the size of the stream dump is about 10GB, I've done bigger ones before without problem). I have scrubbed the pool; no errors. Even with this hung process I can work with the pool, e.g. other backup streams (to other filesystems) work fine.

I've run ktrace on the receive process and the last output I get is

2428 zfs      GIO   fd 1 wrote 107 bytes
       "receiving incremental stream of ... into ..."
2428 zfs      RET   write 107/0x6b
2428 zfs      CALL  ioctl(0x3,0xc0185a1b,0x7fffffff71c8)

The stream is coming from a filesystem that I've been backing up this way successfully for months. It is located on an external disk and I upgraded to a new bigger disk, transferring the ZFS filesystem using zfs send/receive without problems. The error occurred after this. I have migrated the same filesystem from another smaller disk more than a year ago without any problems for the subsequent backups -- I was able to continue streaming incremental backups.

I'd be grateful for any suggestions. I don't know what else to try at this point. Happy to provide further information.
Last edited by a moderator: