Slow Download Speeds from AWS S3

sydney6

Member


Messages: 22

Hi there,

I'm seeing continuous slow download speeds from Amazon S3, but only on FreeBSD. Other OSes saturate the connection without problems.

This happens with 10.3-RELEASE and 11.0-RELEASE and only with AWS S3 in different regions (Ireland, London, Frankfurt, US Standard) whilst using fetch, curl, et. al.

I have tested this on multiple machines, bare metal, bhyve, Xen and VMware VMs, routed setups and direct pppoe links.. all the same.

Anyone seeing similar issues?
 
OP
OP
S

sydney6

Member


Messages: 22

Here's a url to try:

http://s3.eu-central-1.amazonaws.com/4f48caf1d8bcbef8/c5b38f8b3625d2b6/zerofile.raw

Also, when doing double NAT, i.e. VMware Fusion FreeBSD Guest with "Share with my Mac"-Interface, the machine is doing completely fine, as in saturates the link, 90Mbps otherwise between 12-15Mbps..

I also switched the FreeBSD Routers with Linux-based ones and with the isp-provided router box, with the same result.

Edit: I have launched VMs with Digital Ocean to "rule out" my ISP and there seems to be the same issue. Downloading from S3 is multiple times slower than any other connection to services outside of S3 or on any other OS.
 

Tabs

Member

Reaction score: 21
Messages: 65

I haven't had much time to look at this, but the link you provided downloads a lot slower on my FreeBSD 11.0 droplet than the 100MB speedtest file.

Code:
curl -o /dev/null "http://s3.eu-central-1.amazonaws.com/4f48caf1d8bcbef8/c5b38f8b3625d2b6/zerofile.raw"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100M  100  100M    0     0  3715k      0  0:00:27  0:00:27 --:--:-- 3757k
Code:
curl -o /dev/null "http://speedtest.wdc01.softlayer.com/downloads/test100.zip"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100M  100  100M    0     0  10.2M      0  0:00:09  0:00:09 --:--:-- 16.8M
 
OP
OP
S

sydney6

Member


Messages: 22

Hi,

thanks for your results..

FYI, i have also sent this to the freebsd-net mailing list, in case someone would like to follow up on the issue.

If i have news on this one, i will update this forum post.
 

Tabs

Member

Reaction score: 21
Messages: 65

Hello,

I took this issue to the -net mailing list and thanks to Steve Hartland the issue is resolved.

https://lists.freebsd.org/pipermail/freebsd-net/2017-February/047234.html

The problem was/is (probably) lacking TCP timestamps in AWS S3 traffic resulting in a failure of FreeBSDs TCP window scaling mechanism to increase the TCP receive buffer size and therefore resulting in lower troughput over higher latency links.
Very interesting - thanks for posting the outcome.
 
Top