Pros/cons of SSH / netcat ZFS replication?

littlesandra88

littlesandra88

Hi All =)

What are the pros and cons of ZFS replication over SSH and netcat?

I see about 5x speed improvement when using netcat, but are the other things I should be awere of? E.g. are one better then the other when it comes to fault tolerance?

Hugs,
Sandra =)
 
littlesandra88 said:
I see about 5x speed improvement when using netcat, but are the other things I should be awere of?
Click to expand...
It's unencrypted, which is why SSH is slower. There's also no authentication.
 
If running FreeBSD 8.x, install security/openssh-portable and enable HPN.

If running FreeBSD 9.x, add the following to /etc/make.conf and rebuild the world (or at least the OpenSSH parts):
Code: 
### HACK until PR 163095 is committed
### http://www.freebsd.org/cgi/query-pr.cgi?pr=163095
.if ${.CURDIR:M/usr/src/secure/*}
CFLAGS+=-DNONE_CIPHER_ENABLED
.endif

Then you can use the None cipher in SSH, which gets you the best of all worlds:
  • encrypted connection
  • encrypted authentication
  • unencrypted datastream
  • bigger buffers, and auto-tuning buffers, for SSH connection

Note: don't use an unencrypted transport medium across the Internet, or a network you don't control/trust.

Using HPN-enabled OpenSSH and the None cipher, we're able to saturate (980 Mbps) a gigabit link between two ZFS hosts for a ZFS send/recv operation. With HPN and normal encryption, we get around 400-ish Mbps. And with non-HPN (aka "normal) and normal encryption, we had trouble getting to 300 Mbps. Between the same two servers.
 
You can turn on compression with ssh (-C), which may help replication speed/bandwidth consumption over ssh, depending on your network bandwidth vs. CPU processing power.

As above, I wouldn't do unencrypted traffic between two machines for replication unless the network was local and/or otherwise secured such as via IPsec.
 
Hi All,

Thanks a lot for the many details solutions! I will try them out now.

It is on a 10Gbit closed network, so it should be interesting to benchmark =)
 
phoenix said:
Code: 
### HACK until PR 163095 is committed
### http://www.freebsd.org/cgi/query-pr.cgi?pr=163095
.if ${.CURDIR:M/usr/src/secure/*}
CFLAGS+=-DNONE_CIPHER_ENABLED
.endif
Click to expand...

I had no clue about this. I am so going to try this out on our machines to test the difference:) Thanks man!

/Sebulon
 
throAU said:
You can turn on compression with ssh (-C), which may help replication speed/bandwidth consumption over ssh, depending on your network bandwidth vs. CPU processing power.
Click to expand...

This really depends on the speed difference between your disks and your network. On our ZFS storage pools (4x 6-disk raidz2), SSH compression dropped throughput *a lot*. The pool is "multi-threaded" (striped across multiple vdevs) but the SSH compression is single-threaded. Even with a 2.0 GHz Opteron 6100-series CPU (8x magny-cours cores). Disabling SSH compression sped things up.
 
But if one is transferring between two systems on a home network, would forgoing both authentication and encryption by using netcat be a bad idea? I'd rather not slip a hack into my SSH server if I don't have to.
 
phoenix said:
This really depends on the speed difference between your disks and your network. On our ZFS storage pools (4x 6-disk raidz2), SSH compression dropped throughput *a lot*. The pool is "multi-threaded" (striped across multiple vdevs) but the SSH compression is single-threaded. Even with a 2.0 GHz Opteron 6100-series CPU (8x magny-cours cores). Disabling SSH compression sped things up.
Click to expand...

Yeah, as I said, depends on bandwidth vs CPU.

If you're doing replication over, say, a 512k frame-link (or over the internet) like I often have to deal with, compression is a good thing :D Also, if speed is not important but bandwidth cost is (like 3G networks, as I also have to deal with :D), compression helps.


On 10GB-e LAN, not so much.

:)
 
Hey all,

I just wanted to report that after recompiling two of our storage systems, sender and reciever, reconfigured sshd_config to allow the None cipher etc. and using the appropriate swtiches along with the ssh command, our zfs replications are now pushing 1GbE to the limit:)

# zfs send pool/fs@snap | ssh -oNoneEnabled=yes -oNoneSwitch=yes -oHPNBufferSize=8192 username@host zfs recv pool/bakfs

@phoenix

Is the HPN-switch really neccessary, since the buffer size is already specified in each sshd_config? And is there any way to write down the switches as default values in system´s, or user´s ssh_config so that you don´t have to specify the switches in the commands all of the time? I haven´t been able to find any examples of it "out there"...

/Sebulon
 
The buffer isn't needed, as it's listed in the config file.

The None options must be specified on the client command-line each time, to make sure you really want it. A shell alias could work around that, though.
 
to make sure you really want it
Click to expand...

Yeah, that´s what I thought as well. Thanks for the clarification! And thanks again for the hint, it´s working wonders for our backuping:)

/Sebulon
 
You must log in or register to reply here.
Back
Top