Very slow download for packages but not ports

V

vpcd

I have been using 10.2R without (m)any problems as my desktop workstation. Some time ago the download speed for binary packages deteriorated significantly, I am talking 50-120 KiB/s on a link that can achieve around 4 MiB/s. Since I don't install new and big packages very often I have put up with it, hoping that it will fix itself due to magic. It didn't, and today I needed to install a ~2 GiB package which would take 3 hours to complete, so I decided to investigate a bit and here is what I found:
  • Changing the PACKAGESITE to the one closer to me added some 30 KiB/s to the peak,
  • Downloading the package manually with a browser, wget, etc., is also very slow,
  • make fetch achieves full speed with no problems.
I found a thread where the similar problem was described and supposedly fixed with a mirror change, but that does not work for me. My configuration is standard, I didn't change or create any .../pkg/*.conf files.
 
Ports fetch their distfiles from the original upstream sources, packages are fetched from the FreeBSD repository servers. This can account for the differences in speed. The package repositories are all balanced based on GeoIP data, so you're bound to end up on the same, slow, server each time.

Can you tell use which server you end up on?
 
pkg.freebsd.org chooses a mirror near you.

drill _http._tcp.pkg.freebsd.org SRV shows all available mirrors. You could try to set a fixed one in /etc/pkg/FreeBSD.conf as an experiment, but if this problem persists maybe open a bug report on https://bugs.freebsd.org (because uptime of individual mirrors is not guaranteed).
Code: 
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 4703
;; flags: qr rd ra ; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; _http._tcp.pkg.freebsd.org.	IN	SRV

;; ANSWER SECTION:
_http._tcp.pkg.freebsd.org.	1800	IN	SRV	50 10 80 pkg0.ydx.freebsd.org.
_http._tcp.pkg.freebsd.org.	1800	IN	SRV	10 10 80 pkgmir.pkg.freebsd.org.
_http._tcp.pkg.freebsd.org.	1800	IN	SRV	50 10 80 pkg0.nyi.freebsd.org.
_http._tcp.pkg.freebsd.org.	1800	IN	SRV	50 10 80 pkg0.bme.freebsd.org.

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 35 msec
;; SERVER: 192.168.0.1
;; WHEN: Thu Feb 18 16:32:43 2016
;; MSG SIZE  rcvd: 206
pkg ends up using pkgmir.pkg.freebsd.org here.
 
vpcd said:
I did it with the above mentioned mirror and the problem persists.
Click to expand...
pkgmir.pkg.freebsd.org and pkg0.bme.freebsd.org seem to point to the same IP address...
Code: 
% host pkg0.bme.freebsd.org
pkg0.bme.freebsd.org has address 213.138.116.73
pkg0.bme.freebsd.org has IPv6 address 2001:41c8:112:8300::50:1
pkg0.bme.freebsd.org mail is handled by 0 .
% host pkgmir.pkg.freebsd.org
pkgmir.pkg.freebsd.org has address 213.138.116.73
pkgmir.pkg.freebsd.org has IPv6 address 2001:41c8:112:8300::50:1
pkgmir.pkg.freebsd.org mail is handled by 0 .
Maybe try one of the other mirrors.
 
tobik said:
Maybe try one of the other mirrors.
Click to expand...

Russian one gets me ~50 KiB/s, Europe ~150 KiB/s, Asia ~250 KiB/s, USA ~1500 KiB/s. That is if I use the web browser to download the package, if I change /etc/pkg/FreeBSD.conf then the speed stays the same for all mirrors, i.e. jumps all over the place in 50-200 KiB/s range.
 
JamesElstone said:
It could be that they are experiencing a higher than expected demand right now?
Click to expand...

The problem first appeared at least two months ago, it was persistent since then and does not seem to change during the day. Also, download speed is very low (50-150 KiB/s) when using pkg(8) regardless of the mirror I select. The fact that I only get ~150 KiB/s from the European mirror when accessing it with a browser may have to do with Bytemark having problems, esp. since I get much better results from the USA mirror, but I think that there is something more going on for me here. Nevertheless, thanks for the info!
 
Last edited by a moderator:
vpcd said:
I would just like to report that upgrading to 10.3-RELEASE solved this problem.
Click to expand...
That's probably a coincidence. All versions of FreeBSD use the exact same ports tree and package repository servers.
 
vpcd: I see the exact same phenomena you do, also starting at about the same time, but given that a server of mine located in France has no problems at all with the bme0-mirror and gets its 8-10MB/s easily, I came to the conclusion that my ISP is at fault. Other users with the same ISP see roughly the same pattern, while everyone else seemed to have no problems with the pkg-mirrors. My ISP is Deutsche Telekom, infamous for pour peering with other ISP's networks (unless they pay them obscene amounts of money).
Do you have access to other FreeBSD computers? A small VPS would be enough, then you could test the connection issues and see whether it might be your local uplink.
 
yggdrasil said:
Deutsche Telekom, infamous for ...
Click to expand...

My ISP is a subsidiary of DT so it might be that we are both victims of the same corporate policy! That certainly wouldn't surprise me given their ongoing practice of cheating customers by providing sub par services and charging a small fortune for it, along with many other wonderful things ISPs with a monopoly like to do, such as renting you the crappiest of all "routers" and then making you pay extra for the one that can actually handle the service level that you pay for...

However, why did the problem go away when I upgraded to 10.3? Does it mean it might come back?

I currently don't have access to other FreeBSD machines but I had one, located in Germany, at the time that this started happening and it worked normally.
 
Your problem went away? Are you sure? I've been in contact with DT for a few weeks now, pressuring them about the issue (including showing them traces and speedtest logs from different locations to bme0), and it got a bit better, but is still only at best at 2MB/s, instead of the ~10 I should see. And at times, it might still be as bad as 50kB/s.
 
Right now I fetched a large package from bme0 and achieved the expected average of 4 MB/s for my link. The issue seems to be completely gone after upgrading the kernel and userland; it was quite bad just before that and throughout the last half of year, at best 200 kB/s but at times ~50 kB/s.

Although I would like to, I really don't know why the upgrade removed the issue.
 
On 10.3-release I've had the same issue (and previously with 10.2-release). All pkg mirrors fetch at 10.2kB/s. Very rarely they will start fetching faster, but eventually drop to 10.2kB/s. Can anyone help with this? pkg upgrades are pretty much not possible anymore. When I first had the issue, specifying a mirror (pkg0.nyi.FreeBSD.org) solved the problem, but now it doesn't.


Manual fetching of packagesite.txz is very slow:

> fetch http://pkg0.nyi.freebsd.org/FreeBSD:10:amd64/latest/packagesite.txz
> packagesite.txz 1% of 5767 kB 1902 Bps 51m16s
 
I suggest trying a different mirror, preferably one that's, geographically, close. If that also downloads slowly it may just be caused by your own internet connection, your ISP should be able to help out with that.
 
Thanks for the reply.

All mirrors are slow (pkg0.bme.freebsd.org, pkg0.isc.freebsd.org, pkg0.twn.freebsd.org, pkg0.ydx.freebsd.org, pkg0.nyi.freebsd.org).

The server is located in a datacentre with a gigabit connection.

FreeBSD update works fine, so do large file uploads to the server. Manual fetch of anything also works fine unless it is on the freebsd.org domain.

For example:

> fetch http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.0/FreeBSD-11.0-RELEASE-amd64-bootonly.iso only runs at ~20kBps

whereas

> fetch http://mirrors.evowise.com/linuxmint/stable/18.1/linuxmint-18.1-mate-64bit.iso runs at ~1400kBps
 
I have exactly the same problem recently on my FreeBSD servers on OVH.
Actually, even fetching a page from this forum is incredibly slow, while I download any other webpage quickly, as long as the domain doesn't end with freebsd.org.
Do you think OVH would limit download from FreeBSD servers? Why would they do that? Or does FreeBSD blacklist some range of IPS that happen to belong to OVH servers?
 
# pkg search python3
hangs for 90 seconds before returning results. This is really slowing me down.
Any idea how to improve this?
 
krtvsky said:
I have the same problem in ovh, pkg search and pkg install take 2-3 minutes to start.
Click to expand...
This sounds like a badly configured IPv6. That would cause an initial delay of several seconds when it tries to connect using IPv6 first, fails and then falls back to IPv4.
 
Where OVH offers VPS with FreeBSD :)? On other machine i have virtualization and when i create own vps with freebsd all is ok.
 
SirDice said:
This sounds like a badly configured IPv6. That would cause an initial delay of several seconds when it tries to connect using IPv6 first, fails and then falls back to IPv4.
Click to expand...
That sounds coherent. I've postponed configuring IPv6 for years because there's always something else I have to do. Maybe it's time. I'll try to configure it and I'll report if that makes a difference.
 
You must log in or register to reply here.
Back
Top