I have a weird situation where trying to receive an incremental stream generated with zfs send on a Linux system hangs. Specifically, it gets as far as processing the first snapshot, but not past that. The receive process is stuck in wait channel "rwa.cv", which I can't find any information about:
This happened the first time over ssh, but I have also tried dumping the stream to a file, copying the file to the target system and receiving it from there, with the same result. I've left the process running for a few hours, but according to top and zpool iostat nothing is happening. The only way to kill the process is to reboot the machine.
I have run the same receive command with -n and that completed successfully in a few minutes (the size of the stream dump is about 10GB, I've done bigger ones before without problem). I have scrubbed the pool; no errors. Even with this hung process I can work with the pool, e.g. other backup streams (to other filesystems) work fine.
I've run ktrace on the receive process and the last output I get is
The stream is coming from a filesystem that I've been backing up this way successfully for months. It is located on an external disk and I upgraded to a new bigger disk, transferring the ZFS filesystem using zfs send/receive without problems. The error occurred after this. I have migrated the same filesystem from another smaller disk more than a year ago without any problems for the subsequent backups -- I was able to continue streaming incremental backups.
I'd be grateful for any suggestions. I don't know what else to try at this point. Happy to provide further information.
Code:
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 2428 2421 0 21 0 40336 4060 rwa.cv D+ 3 0:04.90 zfs receive...I have run the same receive command with -n and that completed successfully in a few minutes (the size of the stream dump is about 10GB, I've done bigger ones before without problem). I have scrubbed the pool; no errors. Even with this hung process I can work with the pool, e.g. other backup streams (to other filesystems) work fine.
I've run ktrace on the receive process and the last output I get is
Code:
2428 zfs GIO fd 1 wrote 107 bytes
"receiving incremental stream of ... into ..."
2428 zfs RET write 107/0x6b
2428 zfs CALL ioctl(0x3,0xc0185a1b,0x7fffffff71c8)The stream is coming from a filesystem that I've been backing up this way successfully for months. It is located on an external disk and I upgraded to a new bigger disk, transferring the ZFS filesystem using zfs send/receive without problems. The error occurred after this. I have migrated the same filesystem from another smaller disk more than a year ago without any problems for the subsequent backups -- I was able to continue streaming incremental backups.
I'd be grateful for any suggestions. I don't know what else to try at this point. Happy to provide further information.
Last edited by a moderator: