rsync running when ssh pipe broken

Greetings all,

I use rsync over ssh to back-up to a remote server. In irregular intervals the backup progress stalls, and the remote side reports:
Code:
packet_write_wait: Connection to XXX.XXX.XXX.XXX port XXXX: Broken pipe
However, when I check, judging by top and the exit status code, the rsync still runs. Consequently, my script that is watching for the rsync exit status code to report success or to restart rsync never executes.

I do not have root privileges at the remote side; however, I can ask to run specific commands if they could help to analyse the network problem. No irregularity appear on the client side.

Alternatively, is there a way to (1) detect that the ssh pipe is broken, so that rsync can be can killed and restarted, or (2) have the ssh pipe breaking automatically kill rsync so that this event could be detected and rsync restarted by the script?

Kindest regards,

M
 
No, that's the normal way. Just wanted to make sure.

How bizarre. I've yet to run into a situation like yours, when the ssh connection dies but rsync doesn't exit.
 
I have had problems with ssh when one side had lower MTU, and ssh packets got fragmented and it eventually failed: open sessions looked frozen.
Of course, I'm not sure whether that involved in your case, but you can try setting lower MTU manually, e.g. ifconfig ue0 mtu 1400.
 
Hi @phoenix,

yes, I though that since the rsync' transport depends on ssh, once the ssh pipe breaks, the rsync detests it and exits. But no. And it happened several times, so it is not a random occurrence.

Hi aragats,

I will try your suggestion, if my other idea I am testing will not have solved the problem.

Kindest regards,

M
 
Greetings all,

I use rsync over ssh to back-up to a remote server. In irregular intervals the backup progress stalls, and the remote side reports:
Code:
packet_write_wait: Connection to XXX.XXX.XXX.XXX port XXXX: Broken pipe
However, when I check, judging by top and the exit status code, the rsync still runs. Consequently, my script that is watching for the rsync exit status code to report success or to restart rsync never executes.

I do not have root privileges at the remote side; however, I can ask to run specific commands if they could help to analyse the network problem. No irregularity appear on the client side.

Alternatively, is there a way to (1) detect that the ssh pipe is broken, so that rsync can be can killed and restarted, or (2) have the ssh pipe breaking automatically kill rsync so that this event could be detected and rsync restarted by the script?

Kindest regards,

M

I do not know if it works for you but in my case I have mounted all my servers shares in a folder and then I use normal rsync from a folder to another.
 
Greetings all,

I have not solved the asked for problem, but solved the ssh pipe breakage by periodically sending a keep-alive packet to the server.

Kindest regards,

M
 
Hi Lamia,
Autossh might prove useful here!
I am aware that my solution does not address the underlying issue. However, it seems that your proposed solution does not either. I fail to see any advantage of installing additional software over my solution, which uses a configuration option of the rsync.

Could you please elaborate?

Kindest regards,

M
 
I have not solved the asked for problem, but solved the ssh pipe breakage by periodically sending a keep-alive packet to the server.
I suspect there's a firewall state timing out that's causing the connection to be dropped. If the firewall drops (instead of block) subsequent packets it can take a while for SSH to break off the connection as it will retry sending the packets a couple of times. The 'trick' is to send regular keep-alive messages which will prevent the firewall state from timing out. Setting ServerAliveInterval and ServerAliveCountMax can help detect those dropped sessions sooner.
 
Back
Top