pkg slow downloas speed

usakhncit · Sep 26, 2019

Hi
This question is already asked on this forum, but still I am forced to ask it, due to severity of problem.
The problem I am facing is that pkg downloads packages very slow. The average download speed by pkg is between 8-16kB/s. Please check recent download sample:

Code:

Proceed with this action? [Y/n]: 
[1/35] Fetching ja-font-ipa-00303_6.txz: 100%    8 MiB  21.4kB/s    06:21    
[2/35] Fetching ja-font-ipa-uigothic-00203_4.txz: 100%    2 MiB  14.6kB/s    02:35    
[3/35] Fetching ja-font-ipaex-00301.txz: 100%    7 MiB  27.0kB/s    04:50    
[4/35] Fetching ja-font-kochi-20030809_5.txz: 100%    7 MiB  19.3kB/s    05:59    
[5/35] Fetching ja-font-migmix-1.0.20130617_4.txz: 100%    5 MiB  27.4kB/s    03:09    
[6/35] Fetching ja-font-migu-1.0.20130617_4.txz: 100%    5 MiB  24.1kB/s    03:27    
[7/35] Fetching ja-font-mona-ipa-1.0.8_7.txz: 100%    6 MiB  34.0kB/s    03:06    
[8/35] Fetching ja-font-motoya-al-0.0.20100921_5.txz: 100%    3 MiB  36.3kB/s    01:14    
[9/35] Fetching ja-font-mplus-ipa-1.0.20060520.p1_5.txz: 100%    3 MiB  27.0kB/s    02:08    
[10/35] Fetching ja-font-sazanami-20040629_6.txz: 100%    6 MiB  45.3kB/s    02:12    
[11/35] Fetching ja-font-shinonome-0.9.11.p1_6.txz: 100%    5 MiB  42.1kB/s    01:57    
[12/35] Fetching ja-font-takao-003.02.01_5.txz: 100%   10 MiB  25.4kB/s    07:13    
pkg: http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly/All/ja-font-takao-003.02.01_5.txz: Operation timed out

Although SirDice in his following post:

Very slow download for packages but not ports

I have been using 10.2R without (m)any problems as my desktop workstation. Some time ago the download speed for binary packages deteriorated significantly, I am talking 50-120 KiB/s on a link that can achieve around 4 MiB/s. Since I don't install new and big packages very often I have put up...

forums.freebsd.org

mentioned that one could end up on same slow server due to geolocation, but is it possible to use some custom settings in /etc/pkg/FreeBSD, instead of auto geo-location? Kindly guide me in this respect.
Thanks

Grzegorz Wiktorowski · Sep 27, 2019

I've pointed the my nearest server for EU region (pkg0.bme.freebsd.org) in hosts file:

Code:

# pkg.freebsd.org
213.138.116.73    pkg.freebsd.org

D-FENS · Sep 27, 2019

Maybe change to a mirror nearer to you. This is how I do it:

Code:

#############
# pkg mirrors
#####
man pkg

Repository    configuration can be stored in /usr/local/etc/pkg/repos/FreeBSD.conf which overrides the values in /etc/pkg/FreeBSD.conf in    the
     following format:

       FreeBSD: {
         url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
         mirror_type: "srv",
         signature_type: "none",
         fingerprints: "/usr/share/keys/pkg",
         enabled: yes
       }


mcedit /usr/local/etc/pkg/repos/FreeBSD.conf
#-----------8<------------------
  # mirror in UK
FreeBSD: {
  url: "pkg+http://pkg0.bme.freebsd.org/${ABI}/latest"
}
#-----------8<------------------

I believe you can get a list of mirror sites here: https://pkg.freebsd.org/

SirDice · Sep 27, 2019

roccobaroccoSC said:
I believe you can get a list of mirror sites here: https://pkg.freebsd.org/

Yes, that should have a list of mirrors. But, caveat emptor:

IMPORTANT: We do NOT guarantee uptime of any particular mirror. We provide SRV fallbacks for redundancy.

usakhncit · Sep 27, 2019

roccobaroccoSC said:

Maybe change to a mirror nearer to you. This is how I do it:

Code:

#############
# pkg mirrors
#####
man pkg

Repository    configuration can be stored in /etc/pkg/FreeBSD.conf in    the
     following format:

       FreeBSD: {
         url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
         mirror_type: "srv",
         signature_type: "none",
         fingerprints: "/usr/share/keys/pkg",
         enabled: yes
       }


mcedit /etc/pkg/FreeBSD.conf
#-----------8<------------------
  # mirror in UK
  url: "pkg+http://pkg0.bme.freebsd.org/${ABI}/latest",
#-----------8<------------------

I believe you can get a list of mirror sites here: https://pkg.freebsd.org/

Here is my /etc/pkg/FreeBSD:

Code:

| cat /etc/pkg/FreeBSD.conf 
# $FreeBSD: releng/12.0/usr.sbin/pkg/FreeBSD.conf 340161 2018-11-05 21:28:32Z gjb $
#
# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
#   mkdir -p /usr/local/etc/pkg/repos
#   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
#

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

I am using:
url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
but you are using:
url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
Why is the difference? Quarterly VS Latest? Are you using "CURRENT" branch?

SirDice · Sep 27, 2019

Don't modify /etc/pkg/FreeBSD.conf. Create a /usr/local/etc/pkg/repos/FreeBSD.conf and put your changes there.

zetrotrack000 said:
Why is the difference? Quarterly VS Latest? Are you using "CURRENT" branch?

All -RELEASE versions have the quarterly branch set as default. -STABLE and -CURRENT have latest by default. Any version (including the -RELEASE versions) can be changed from one branch to the other.

The latest package repositories follow the ports tree as fast as it can. The quarterly repositories are updated once every three months and only receive security updates in the mean time. Some time early October the 2019Q4 branch will be branched off.

PacketMan · Sep 27, 2019

Would be nice if:

the pkg system also provided the use of bittorrent to distribute the packages.
FreeBSD users could install a port/package that would let their server use bittorrent to receive and send copies of these packages.
and this torrent system package would provide users with options so that issues around disk storage capacity can be addressed.

So a guy like me for example, would install the port/package, configured it to get all the packages, or a subset (say part of the tree), and configure FreeBSD to use its local repository first. And having package management tools like ports-mgmt/synth to use this local repository would be bang on. So when I run Synth, and tell it to fetch pre-built packages when possible, it will use the torrent respository on my machine as a first choice, and the non-torrent repository mirrors 2nd. For those users that are hard-core package users, and do not use tool like Synth, then same applies for them too; FreeBSD will use the torrent respository on his/her machine as a first choice, and the non-torrent repository mirrors 2nd.

usakhncit · Sep 27, 2019

PacketMan said:
Would be nice if:

the pkg system also provided the use of bittorrent to distribute the packages.

FreeBSD users could install a port/package that would let their server use bittorrent to receive and send copies of these packages.

and this torrent system package would provide users with options so that issues around disk storage capacity can be addressed.

So a guy like me for example, would install the port/package, configured it to get all the packages, or a subset (say part of the tree), and configure FreeBSD to use its local repository first. And having package management tools like ports-mgmt/synth to use this local repository would be bang on. So when I run Synth, and tell it to fetch pre-built packages when possible, it will use the torrent respository on my machine as a first choice, and the non-torrent repository mirrors 2nd. For those users that are hard-core package users, and do not use tool like Synth, then same applies for them too; FreeBSD will use the torrent respository on his/her machine as a first choice, and the non-torrent repository mirrors 2nd.

Interesting suggestion. Let's see whats experts say about that

SirDice · Sep 27, 2019

PacketMan said:
the pkg system also provided the use of bittorrent to distribute the packages.

So you run pkg install whatever and then have to wait anywhere between 2 minutes to 2 days for it to complete? While torrents are good to provide the big installation images for example, I don't think it's good to use for things that need to be done 'instantly'.

usakhncit · Sep 27, 2019

SirDice said:
So you run pkg install whatever and then have to wait anywhere between 2 minutes to 2 days for it to complete? While torrents are good to provide the big installation images for example, I don't think it's good to use for things that need to be done 'instantly'.

Isn't that possible that FreeBSD stress 'slow server owners' to fix their server, and if they don't then block them?

SirDice · Sep 27, 2019

It's usually not the servers themselves that are slow but the infrastucture between the server and consumer.

usakhncit · Sep 27, 2019

SirDice said:
It's usually not the servers themselves that are slow but the infrastucture between the server and consumer.

Consider my case, I usually get 450KB/s download speed. On my Linux box, the download speed from repos is also good. But here, on FreeBSD, the download speed from repos is very slow, 8-16KB/s. So, in my case, the real culprit is slow server!

SirDice · Sep 27, 2019

zetrotrack000 said:
On my Linux box, the download speed from repos is also good. But here, on FreeBSD, the download speed from repos is very slow, 8-16KB/s. So, in my case, the real culprit is slow server!

Not if it comes from different servers that are on different parts of the internet. The "internet" doesn't really exists, it's just a huge collection of local networks that are tied together through peering contracts. Some peers are good, some peers are bad. So some connections to parts of the world are good while the bad peers will give you bad connections to the other parts.

usakhncit · Sep 27, 2019

SirDice said:
Not if it comes from different servers that are on different parts of the internet. The "internet" doesn't really exists, it's just a huge collection of local networks that are tied together through peering contracts. Some peers are good, some peers are bad. So some connections to parts of the world are good while the bad peers will give you bad connections to the other parts.

SirDice Can you look at my last post at:

Mounted directory showing empty in Dolphin file-manager

Hi I have mounted my linux partition as (/linux-home) with the help of following fstab entry: /dev/ada0s5 /linux-home fuse rw,mountprog=/usr/local/bin/lklfuse,type=ext4,allow_other,late 0 0 And I can read-write in this (/linux-home) directory as normal user: | whoami zetro | cd...

forums.freebsd.org

PacketMan · Sep 27, 2019

SirDice said:
So you run pkg install whatever and then have to wait anywhere between 2 minutes to 2 days for it to complete? While torrents are good to provide the big installation images for example, I don't think it's good to use for things that need to be done 'instantly'.

SirDice said:
Not if it comes from different servers that are on different parts of the internet. The "internet" doesn't really exists, it's just a huge collection of local networks that are tied together through peering contracts. Some peers are good, some peers are bad. So some connections to parts of the world are good while the bad peers will give you bad connections to the other parts.

True and true there Sir ole freind. But bittorent algorithms prefer the faster/quicker 'seeds'. Which is why I suggest that the FreeBSD mirrors would also be running bittorrent. And the hope would be that those of us blessed with good upload speeds would contribute to this system. And if those 'in need' of the snappiest response felt that the torrent system was not up to par, they simply could remove the port/package that makes it work, and go back to the 'good ole; way'.

Its Friday, I can dream a bit right?

SirDice · Sep 27, 2019

zetrotrack000 said:
Can you look at my last post at:

I've seen it. If I have something to add I would, but I don't so I won't.

SirDice · Sep 27, 2019

Please keep everything in the same threads. It's going to get utterly confusing for everybody else if your responses are spread over different threads (with different subjects).

I read pretty much everything (as much as possible in any case) and will respond if I have anything meaningful to add.

usakhncit · Sep 27, 2019

SirDice said:
Please keep everything in the same threads. It's going to get utterly confusing for everybody else if your responses are spread over different threads (with different subjects).

I read pretty much everything (as much as possible in any case) and will respond if I have anything meaningful to add.ou

Oh, sorry. Both tab were opened side-by-side and I did not pay attention to title and thought that I was writing in right thread.

usakhncit · Sep 28, 2019

roccobaroccoSC said:

Maybe change to a mirror nearer to you. This is how I do it:

Code:

#############
# pkg mirrors
#####
man pkg

Repository    configuration can be stored in /etc/pkg/FreeBSD.conf in    the
     following format:

       FreeBSD: {
         url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
         mirror_type: "srv",
         signature_type: "none",
         fingerprints: "/usr/share/keys/pkg",
         enabled: yes
       }


mcedit /etc/pkg/FreeBSD.conf
#-----------8<------------------
  # mirror in UK
  url: "pkg+http://pkg0.bme.freebsd.org/${ABI}/latest",
#-----------8<------------------

I believe you can get a list of mirror sites here: https://pkg.freebsd.org/

Now I am getting a little bit reasonable download speed of 100KB/s.

usakhncit · Sep 29, 2019

zetrotrack000 said:
Now I am getting a little bit reasonable download speed of 100KB/s.

But this reasonable speed is also occasional. Most of the times I am getting 8-16KB/s. I am stuck with it :-(

D-FENS · Sep 29, 2019

Well, why don't you open the mirror URL in your browser and try to download the file manually? Also you can try from the command line using the command fetch(1) (this is what pkg does).
If your download speed is too low, then it's probably your Internet connection and not the OS. Get in touch with the Internet service provider and ask.

Also, could you be using a proxy, VPN or a Tor network, which could slow things down?
Also, do a speed test online.

big.fat.panda · Oct 20, 2021

I am facing the same issue and I have tried every single mirror there is to try. I am not crossing 40 KBPS anywhere. How can this be? How can every single server be so slow? I am getting at least 3 MBPS from everywhere else (I am on a 30 MbPS / 3.5 MBPS connection).

seirios · Oct 21, 2021

I want to know your environment, because FreeBSD 13-Release on Hypervisor like xcp-ng or any other qemu based vm.
if your environment runnibg on Qemu based vm, so you try to followings.

ifconfig xxx -txcsum -rxcsum - lro -tso

I checked by tcpdump, then I found checksum error at the communicatoion link.
I found this on FreeBSD 9.* , 10.*, 11.*, and 13.*.
I hope to solve your problem.

pkg slow downloas speed

usakhncit

Very slow download for packages but not ports

Grzegorz Wiktorowski

D-FENS

SirDice

Administrator

usakhncit

SirDice

Administrator

PacketMan

usakhncit

SirDice

Administrator

usakhncit

SirDice

Administrator

usakhncit

SirDice

Administrator

usakhncit

Mounted directory showing empty in Dolphin file-manager

PacketMan

SirDice

Administrator

SirDice

Administrator

usakhncit

usakhncit

usakhncit

D-FENS

big.fat.panda

seirios