Solved zfs replication speed is too slow.

[sdf]

Hello everyone,

When I use "zfs send" and "zfs receive" to make a snapshot copy between two servers, why is the transfer speed only 57.3K/s?

zfs send -Rv zroot@snap | ssh root@b.example.com zfs receive -dvF

20:10:13 57.3K zroot/ROOT@snap
20:10:14 57.3K zroot/ROOT@snap
20:10:15 57.3K zroot/ROOT@snap
...

Thanks.

 

[Terry_Kennedy]

When I use "zfs send" and "zfs receive" to make a snapshot copy between two servers, why is the transfer speed only 57.3K/s?

1) Use benchmarks/iperf or a similar tool to make sure you don't have an underlying network problem.
2) Try doing a # zfs send to /dev/null to make sure your pool can sustain the necessary read rate.
3) ssh is generally CPU bound when doing bulk data transfers. Use top to see what processes are using a lot of CPU.
4) # zfs receive is synchronous - it has to wait for data to come from the sending system, then write it to the pool (which may be done synchronously or asynchronously), then acknowledge that data and wait for the next packet.

You might want to look at the "Replication and backups" section of my RAIDzilla 2.5 page here, where I demonstrate 600+ Mbyte (yes, byte) / second transfers (over 10GbE, of course) and go into details on how to achieve it.

 

[ShelLuser]

This is one of the reasons why I never bother with zfs send in combination with zfs receive. Instead I prefer: # zfs send root@snap | ssh backup@example.com "dd of=/backups/snapshot_root.zfs".

That method also prevents not having to rely on a rather stupid security hazard regarding direct root logons (which is plain out bad no matter how you look at it).

 

[sdf]

1) Use benchmarks/iperf or a similar tool to make sure you don't have an underlying network problem.

Thank you. I tested on two VMware virtual machines on the same computer. And I used scp to copy test between the two virtual machines and the speed is normal.

Virtual machine RAM 1G

zfs send -Rv zroot@snap | ssh root@b.example.com zfs receive -dvF
Is there a problem with this command?

zfs send -Rv zroot@snap > /dev/null OK


zfs send -Rv zroot@snap | ssh root@b.example.com > /dev/null

....
total estimated size is 1.2G
TIME    SENT    SNAPSHOT
TIME    SENT    SNAPSHOT
Password for root@b.example.com:20:21:33    57.3K    zroot/ROOT@snap
20:21:34    57.3K    zroot/ROOT@snap
20:21:35    57.3K    zroot/ROOT@snap
...