Slow network (KB/s) between FreeBSD to Solaris combination

A

Airell

Hi there,

I'm working on a network issue which involves FreeBSD 14.3 (-p2), supplied via the OCI (Oracle Cloud Infrastructure) Marketplace (https://cloud.oracle.com/marketplace/application/187135972/overview). This is an official App by The FreeBSD Foundation.

We have a Solaris 11.3 VM running on SPARC emulation software in an Linux VM in OCI and we want to send ZFS snapshots from the FreeBSD VM directly into the 'Solaris SPARC VM'. This is incredibly slow (less than 30 KB/s when testing with scp). Now networking to this Solaris VM is not great, but we can get ~25 MB/s transfer rate when testing with scp from a Linux (OEL) box to the Solaris VM. Although not much as well, but we know the limitations of the VM withing Emulation on a VM ... However, when we scp from FreeBSD to Linux, no issues (250 MB/s). This is all located in the same OCI Region and Fault Domain.

I just can't figure out why we only get less than 30 KB/s to -stalled- from FreeBSD to Solaris, while we get 25 MB/s from OEL to Solaris. Also from FreeBSD to OEL is great (250 MB/s). When 'ssh-tunneling' from a FreeBSD 14.3-p2 from home directly into this Solaris VM (From Netherlands to Singapore), I get 1 MB/s continuously, no stalling.

Tried many things, including disabling HW offloading and flow control on the FreeBSD interface, comparing kernel and network related sysctl parameters etc... I do see `sbwait` in the process status so I think FreeBSD is pushing to hard somehow.

I admit we can not rule out things on the Solaris VM, but because the network connection from FreeBSD to Solaris VM is about 1000 times slower than others to the Solaris VM, I'm trying to get any clues from the FreeBSD box why it is so slow in this combination.

Any ideas to try is appreciated.

Regards, Airell
 
Have you tried standard performance-measurement tools like iperf3. You should first compare TCP and UDP performance, over IPv4 and IPv6 with something like iperf3 before trying more complicated technologies like scp.

Looking at the various statistics reported by netstat -s can also give you a clue.

Another thing to try is a FreeBSD VM and a Linux VM on the same physical machine. Measure their speed to the Solaris VM.

If you can't figure it out yourself, capture packet traces of the slow transfers from both ends and upload them somewhere for other people to take a look.

I'm curious why you run Solaris on emulated SPARC instead of on x64 directly. The more layers you have, the more problems you get. One more thing you can try is putting a Solaris-on-x64 VM on the same host as the Solaris-on-emulated-SPARC VM specifically for testing the network performance.
 
You must log in or register to reply here.
Back
Top