The story goes like this:
We have a server with huge ZFS capacity in a DC. (server is being backed up)
We have many clients who use ZFS storage boxes ranging from 2TB - 4TB. All clients send a full first snapshot of their data to a USB disk which is being shipped to the DC. Their data then are being received from the server.
Now, the clients need to be able to send differential snapshots to that server. Their daily differential could be anything between a few Megabytes to 10 Gigabytes! The bottleneck is their upload bandwidth which ranges between 1Mbit to 2Mbit.
The problem here is not so much their upload speed but the fact that a ZFS send operation over ssh can stall, due to network issues, without the ability to resume. Meaning that they have to start the process again.
My thoughts and please comment!
I haven't found the way to do this with SFTP, FTPS or SCP. Please let me know if I am wrong.
So, I though of torrent. Torrent was designed as a means to reliable transfer files peer2peer. I think that this could be a solution.
I would appreciate your inputs, comments on the above. If what I am saying sounds stupid please don't hesitate to point it out.
If you are already doing this in different way then I would be glad to know how!
Thanks
We have a server with huge ZFS capacity in a DC. (server is being backed up)
We have many clients who use ZFS storage boxes ranging from 2TB - 4TB. All clients send a full first snapshot of their data to a USB disk which is being shipped to the DC. Their data then are being received from the server.
Now, the clients need to be able to send differential snapshots to that server. Their daily differential could be anything between a few Megabytes to 10 Gigabytes! The bottleneck is their upload bandwidth which ranges between 1Mbit to 2Mbit.
The problem here is not so much their upload speed but the fact that a ZFS send operation over ssh can stall, due to network issues, without the ability to resume. Meaning that they have to start the process again.
My thoughts and please comment!
- Save the differential snapshot to a compressed file.
- Use a reliable means of transferring that file to the server.
- Restore the snapshot once the transfer has been completed. (repeat the procedure if the snapshot fails)
I haven't found the way to do this with SFTP, FTPS or SCP. Please let me know if I am wrong.
So, I though of torrent. Torrent was designed as a means to reliable transfer files peer2peer. I think that this could be a solution.
I would appreciate your inputs, comments on the above. If what I am saying sounds stupid please don't hesitate to point it out.
If you are already doing this in different way then I would be glad to know how!
Thanks