Odd scp/sftp performance issues with FreeBSD bare metal vs Linux bhyve instance on same hardware

fritz

fritz

Hi all.

TL;DR​

- scp from a MacBook to bare metal FreeBSD over 2.5G NIC (on Mac) to 10G interface on BSD gets ~250kb/s
- scp from same MacBook to Linux bhyve instance on the FreeBSD host gets 56MB/s
- iperf3 from MacBook to FreeBSD saturates the link (2.5Gbit/s)
- Both FreeBSD and Linux ssh/sftp sessions are using the same ciphers for transfer and same keys for key exchange.

Long form explanation​

I have an odd performance issue that I'm struggling to remedy. I have a MacBook with a 2.5G NIC, connected to an EPYC 9254-based FreeBSD 15 machine (generic kernel, no changes to default sshd configs or tcp tunables). That same machine has a Linux VM running in bhyve using virtio networking and a bridge.

When trying to scp a 200MB file from the MacBook to the FreeBSD host, my throughput is about 250kb/s, which slowly creeps up to 1-2MB/s. When I scp the same file to the Linux VM that is running on the host, I move the file at 56MB/s.

When I scp the file between the FreeBSD host and the Linux VM (in either direction), it moves the file at about 500MB/s (faster than ssh/sftp can reliably measure I'm sure).

It doesn't immediately look like a networking issue. iperf3 between the MacBook and FreeBSD host saturate the MacBook's 2.5G NIC (see below), when using 6 threads.

It also appears that both FreeBSD and the Linux VM are using the same ciphers for everything ( scp -vv doesn't look like it advertises what ciphers it settled on, but ssh -vv indicates both are using chacha).

Additionally, I have several jails on the host that exhibit the same behavior.

Any thoughts would be appreciated. Thanks!

Supporting documentation​


Relevant scp output for Mac->FreeBSD host:

Code: 
iperf3 -c brunt -P 6
Connecting to host brunt, port 5201
[  7] local 10.0.8.101 port 54941 connected to 10.0.8.10 port 5201
[  9] local 10.0.8.101 port 54942 connected to 10.0.8.10 port 5201
[ 11] local 10.0.8.101 port 54943 connected to 10.0.8.10 port 5201
[ 13] local 10.0.8.101 port 54944 connected to 10.0.8.10 port 5201
[ 15] local 10.0.8.101 port 54945 connected to 10.0.8.10 port 5201
[ 17] local 10.0.8.101 port 54946 connected to 10.0.8.10 port 5201

<data elided>

[SUM]   0.00-10.01  sec  2.71 GBytes  2.33 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  2.71 GBytes  2.32 Gbits/sec                  receiver

iperf Done.

Relevant scp output for Mac->Linux bhyve:

Code: 
Executing: program /usr/bin/ssh host zek, user (unspecified), command sftp
debug1: OpenSSH_10.0p2, LibreSSL 3.3.6
debug1: Reading configuration data /Users/fritz/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to zek port 22.

debug2: KEX algorithms: mlkem768x25519-sha256,sntrup761x25519-sha512,sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com

<data elided>

scp between Mac->FreeBSD host:

Code: 
Executing: program /usr/bin/ssh host brunt, user (unspecified), command sftp
debug1: OpenSSH_10.0p2, LibreSSL 3.3.6
debug1: Reading configuration data /Users/fritz/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to brunt port 22.

debug2: KEX algorithms: mlkem768x25519-sha256,sntrup761x25519-sha512,sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 

<data elided>
 
cracauer@ said:
What are the filesystems involved?
Click to expand...
The host is using ZFS with an array of NVME backing the pool. The Linux VM is using a zvol cut from the same pool.

I reran the tests using a tmpfs with the same result, so I don't think the filesystem or disks are the culprit.

As an aside -- when I kill a file transfer, the sftp process stays running and keeps the file handle on the destination file open. I didn't give the process enough time to cleanup, so maybe it would have, but is this expected? Is sftp keeping the file open in case the client reconnects to continue the transfer?
 
fritz said:
As an aside -- when I kill a file transfer, the sftp process stays running and keeps the file handle on the destination file open. I didn't give the process enough time to cleanup, so maybe it would have, but is this expected? Is sftp keeping the file open in case the client reconnects to continue the transfer?
Click to expand...

Is the leftover process killable?

I don't know much about sftp. Can you try rsync?
 
cracauer@ said:
Is the leftover process killable?

I don't know much about sftp. Can you try rsync?
Click to expand...
It is killable, and the ssh session it's sitting on cleans up fine when I do. rsync moves the file at near wire speed on both FreeBSD and Linux.

JohnK said:
Newer scp(1) uses sftp(1) because the old scp code is depreciated. I believe my mac still has the old scp(1).

Try a tar pipe over ssh.
Click to expand...
My issue here is that scp is a pretty common approach, and both the Linux vm and FreeBSD host are using sftp for the transport, just with wildly different performance.

It's clear at this point that it's sftp-server as the culprit, as I can move data over ssh proper at full speed. Curious...
 
fritz said:
My issue here is that scp is a pretty common approach, and both the Linux vm and FreeBSD host are using sftp for the transport, just with wildly different performance.

It's clear at this point that it's sftp-server as the culprit, as I can move data over ssh proper at full speed. Curious...
Click to expand...
Correct it is and I'm not all that excited about scp being depreciated and essentially sftp now.

For what it's worth: I also have this (which uses a tar pipe over ssh) which you can add to your test (*my* test results: "super fast"). Please try if you want (don't forget to like and subscribe [that's bad humor]).
github.com

GitHub - JohnKaul/drist: A reliable and auditable tool to synchronize configurations to and/or run scrips on remote host(s) via ssh(1).

A reliable and auditable tool to synchronize configurations to and/or run scrips on remote host(s) via ssh(1). - JohnKaul/drist
github.com github.com
 
You must log in or register to reply here.
Back
Top