Slow ftp but fast fetch or curl

athan

New Member


Messages: 12

#1
Putting or getting files using amd64 FreeBSD 11.1rel stock cli ftp client I'm experiencing very slow speed (4MB/s).
Same files using either fetch, curl or wget are transfered at almost 50MB/sec, still using exactly the same settings and ftp protocol.

Tried with various other ftp servers getting exactly the same results. Builtin ftp client is 10x slower than either fetch, curl or wget.

Anybody has any idea why is that?

PS: Generic kernel and no firewall or packet filter
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,332
Messages: 27,504

#2
It might be caused by a difference in active or passive FTP. I'm not entirely sure but I believe the ftp(1) command uses active FTP and fetch(1) passive FTP by default. It's worth trying to see if that makes a difference.
 
OP
OP
athan

athan

New Member


Messages: 12

#3
Thanks SirDice but I'm afraid it's not that simple.
All ftp servers I tried are passive - only thus ftp is not even connecting when using -A option

Btw no other ftp client I tried so far has this speed issue, only the stock one. Really weird situation...
 

PacketMan

Aspiring Daemon

Thanks: 139
Messages: 868

#4
I've never tried to performance test FTP, but I wonder, do you log in once, or per file. When I transfer files using a utility the utility logs in once and the file transfer rate seems to be pretty fast. However some IP cameras I have that use FTP, authenticate on a per file basis I'm pretty sure. I gotta belive that will hurt the throughput rate. I have this vague memory of using a utility program one time that did the same thing, authentication on a per file basis, and I was thinking "well now that's just yucky".
 

VladiBG

Well-Known Member

Thanks: 99
Messages: 272

#5
Search the problem somewhere else it's not in the native ftp client. It utulize the 100Mbps line (11.25MiB/s). Check your router for ftp inspection.
 
OP
OP
athan

athan

New Member


Messages: 12

#6
Check your router for ftp inspection.
I think that if router was the problem any ftp client using same ports and same mode should be affected not only the stock one.
Try using the stock client with some distant server with enough latency and compare results with other ftp clients.
 

Datapanic

Well-Known Member

Thanks: 134
Messages: 257

#7
I tried /usr/bin/ftp with FreeBSD 11.1-p9 amd64 between a VM and my NAS box (all FreeBSD) on the same LAN and got 52 MiB/s for a 620M file.

But going from the same VM to ftp.freebsd.org and downloading FreeBSD-11.1-RELEASE-amd64-dvd1.iso, I am only getting about 2 MiB/s.
 
OP
OP
athan

athan

New Member


Messages: 12

#8
I tried /usr/bin/ftp But going from the same VM to ftp.freebsd.org and downloading FreeBSD-11.1-RELEASE-amd64-dvd1.iso, I am only getting about 2 MiB/s.
Could you please try the same download from ftp.freebsd.org using fetch this time and tell us the difference if any.
 

Datapanic

Well-Known Member

Thanks: 134
Messages: 257

#9
Could you please try the same download from ftp.freebsd.org using fetch this time and tell us the difference if any.
With /usr/bin/fetch, it was only a little better with an average of 3200 kBps which should be around 3.2 MiB/s:

Code:
[honkycat@chateau01 ~]$ fetch -h ftp.freebsd.org -f /pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-dvd1.iso
FreeBSD-11.1-RELEASE-amd64-dvd1.iso            31% of 3193 MB 3600 kBps 11m02s
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,332
Messages: 27,504

#10
Are these different?

fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso
fetch http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso
 
OP
OP
athan

athan

New Member


Messages: 12

#11
Yes they're different, only the first uses ftp protocol. The second one is http..

BTW, 2MB/s to 3.2MB is not only "a little better", it's about 60% faster; a huge difference considering that both use the same protocol.
 

VladiBG

Well-Known Member

Thanks: 99
Messages: 272

#12
If you want to test the speed then you have to make a test setup between two computers connected on the same switch. On the public ftp/http servers there's a lot of QoS polices that restrict the bandwidth per connection and protocol to ensure that one client won't take the entire bandwidth and to protect itself from DoS. That's why those test are irrelevant. Also there's a difference in the protocols and send/receive buffers that also have some impact of the transfer speed.

So you are testing nothing more than your Internet connection speed not the ftp client itself. If you still want to test it over the Internet then you can use some of the test servers that can provide bigger throughput per client. For example you can try the speedtest.tele2.net that claim to support up to 10Gbps.
the addresses are:
ftp://speedtest.tele2.net
http://speedtest.tele2.net

where you can select to download/upload a test file.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,332
Messages: 27,504

#13
Yes they're different, only the first uses ftp protocol. The second one is http..
Yes, I know that. I was wondering if you experience a difference in speed between HTTP and FTP.
 

VladiBG

Well-Known Member

Thanks: 99
Messages: 272

#14
I don't see any difference in the download speed between HTTP and FTP.
Here's the test with ftp and fetch:

1524207632145.png
 
OP
OP
athan

athan

New Member


Messages: 12

#15
So you are testing nothing more than your Internet connection speed not the ftp client itself. If you still want to test it over the Internet then you can use some of the test servers that can provide bigger throughput per client. For example you can try the speedtest.tele2.net that claim to support up to 10Gbps.
the addresses are:
ftp://speedtest.tele2.net
http://speedtest.tele2.net
I know that VladiBG and it's exactly what I'm doing.

Take a look

scr.jpg
 

Sensucht94

Well-Known Member

Thanks: 326
Messages: 345

#18
Putting or getting files using amd64 FreeBSD 11.1rel stock cli ftp client I'm experiencing very slow speed (4MB/s).
Same files using either fetch, curl or wget are transfered at almost 50MB/sec, still using exactly the same settings and ftp protocol.

Tried with various other ftp servers getting exactly the same results. Builtin ftp client is 10x slower than either fetch, curl or wget.

Anybody has any idea why is that?

PS: Generic kernel and no firewall or packet filter
Hi, I had never realized it, but after performing some tests it looks like you're right: the ftp client included in base system is ~ 4 times slower than any other ftp/http download client I tried. Keeping in mind that FTP as proto should be slightly faster in download than HTTP given same hardware and configuration (I assume both FTP and HTTP FreeBSD servers physically reside on the same machine), here's my tests (speed peaks) performed on FreeBSD 11.1-p9 RELEASE, custom kernel, IPFW enabled (http, ftp, ftp-data, dns ports naturally opened), Ethernet stable dsl connection with Intel (R) PRO/1000 Gigabit (2016) using em(4) driver:

- FreeBSD's ftp(1) getting ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso: 1,1 Mb/s

- FreeBSD's fetch(1) getting http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso: 4,1 Mb/s

At which point i thought there was rather an issue with the FreeBSD server. However:

- ftp/lftp getting disc1.iso from ftp FreeBSD server: 4,5 Mb/s

- ftp/axel download accelerator getting disc1.iso from http FreeBSD server: 4,45 Mb/s

- ftp/axel getting disc1.iso from ftp FreeBSD server: 4,57 Mb/s
 
Top