How to configure mirror in FreeBSD 10

[duleshi]

I'm an intermediate Linux user and new to FreeBSD. What I have found the most daunting thing is the slow downloading speed for ports.

In all the Linux distros I have used, it's very convenient and FAST to install binary packages. For example, Fedora and CentOS can automatically find the fastest mirror site for you, others can achieve this by simply editing a file such as sources.list in Debian. But in FreeBSD, it's a great pain to install new software, whether by pkg install <binary> or make install clean in ports.

I have searched the internet, but most of the FreeBSD articles are about FreeBSD 8. I found the ports-mgmt/fastest_sites in this forum. But the mirrors in my country are not listed in bsd.port.mk.
Any one please help me configuring the mirror? Below is one of the mirrors in my country:
http://mirrors.tuna.tsinghua.edu.cn/freebsd/

 

[SirDice]

PKGNG is still relatively new and we also have to build the "old-style" packages. Hopefully when the old package system gets deprecated the freed up resources will be used for PKGNG. That could provide more (local) mirrors.

 

[duleshi]

PKGNG is still relatively new and we also have to build the "old-style" packages. Hopefully when the old package system gets deprecated the freed up resources will be used for PKGNG. That could provide more (local) mirrors.

I'm not sure if I fully understand what you mean. Do you mean the legacy releases have a higher priority than the the current one? And there's nothing I can do to make my download faster? Do you recommend me to use FreeBSD 8? I personally don't like old releases.

 

[SirDice]

No, I'm saying there are only a finite number of machines that build and distribute packages for FreeBSD. Most of them are used for the "old-style" packages, because that was all we had. PKGNG (what FreeBSD 10.0 uses by default) is a new way of packaging that tries to solve a lot of the issues we had with the old system. Until we can get rid of the old package system we can't use those resources for PKGNG.

 

[duleshi]

No, I'm saying there are only a finite number of machines that build and distribute packages for FreeBSD. Most of them are used for the "old-style" packages, because that was all we had. PKGNG (what FreeBSD 10.0 uses by default) is a new way of packaging that tries to solve a lot of the issues we had with the old system. Until we can get rid of the old package system we can't use those resources for PKGNG.

Don't be misled by my Linux examples. I only want to emphasize the ease of using a local mirror in Linux. It's ok for me if you can't provide binaries for FreeBSD 10, since I can build them myself in ports. The ports way is one of the reasons FreeBSD attracts me.
But can the source code downloading be faster when I input "make install"?
Does the mirror site I mentioned contain the sourece codes? I don't understand the structure of the FreeBSD mirrors.

 

[SirDice]

A lot of the source files for ports aren't stored on FreeBSD servers, they're fetched directly from the original (upsteam) sites. There are a lot of ports that fetch sources from SourceForge sites for example. Only if it can't find it there would it fall-back to a couple of FreeBSD servers that mirrored the file.

Any port in particular that downloads slowly?

 

[duleshi]

A lot of the source files for ports aren't stored on FreeBSD servers, they're fetched directly from the original (upsteam) sites. There are a lot of ports that fetch sources from SourceForge sites for example. Only if it can't find it there would it fall-back to a couple of FreeBSD servers that mirrored the file.

Any port in particular that downloads slowly?

Almost all the ports downloading speed is very slow. About 20K/s~30K/s. Never above 100K/s. For example, installing gnome2 took me several hours' time.
When I use local mirrors for Linux, the speed is is about 500K/s~1.xM/s.
ps: This forum don't have a notification machanism like StackOverflow? I always have 0 new messages even after you answered my question.

 

[SirDice]

Well, I think that may be something local. Most ports download their sources quite fast here. Even if I have to get them from the other side of the world.

Here's one example (from print/cups-client):

dice@molly:~ % fetch http://www.cups.org/software/1.7.1/cups-1.7.1-source.tar.bz2
cups-1.7.1-source.tar.bz2                     100% of 8533 kB  933 kBps 00m09s

 

[duleshi]

Well, I think that may be something local. Most ports download their sources quite fast here. Even if I have to get them from the other side of the world.

Here's one example (from print/cups-client):

dice@molly:~ % fetch http://www.cups.org/software/1.7.1/cups-1.7.1-source.tar.bz2
cups-1.7.1-source.tar.bz2                     100% of 8533 kB  933 kBps 00m09s

Then I must say you have a fast Internet connection. I in fact installed FreeBSD as a virtual machine, and I don't have that computer with me now. But I'm certain I can't download it that fast.
I think FreeBSD mainly consider users in Europe and North America. Although most of its users indeed reside there. Think of the other places where internet connection is not that good. What a pain for them to download all kinds of packages from an oversea server!
I'm not blaming anybody. But Linux's popularity has a reason. They are more convenient to use for many people.

ps: I download in Linux using wget. Here is the result:
10% [===> ] 939,556 11.1KB/s eta 12m 42s

 

[SirDice]

Then I must say you have a fast Internet connection.

Well, yes. I am blessed with a 120 Mbit/s cable internet connection.

I in fact installed FreeBSD as a virtual machine, and I don't have that computer with me now. But I'm certain I can't download it that fast.

I understand that, not everybody has a big internet pipe. But there should be no reason why FreeBSD downloads significantly slower than Linux. Downloading the same file from the same source should, at the very least, have similar speeds.

ps: I download in Linux using wget. Here is the result:
10% [===> ] 939,556 11.1KB/s eta 12m 42s

Ok, I can't compare that to my speed of course but do you think that's a reasonable speed for your connection?

But you mentioned running FreeBSD in a VM. Is the Linux you tested this with on a similar VM? The bad speeds might be VM related, what VM software are you using and what (network) interfaces do you have in the virtualized FreeBSD?

 

[duleshi]

But you mentioned running FreeBSD in a VM. Is the Linux you tested this with on a similar VM? The bad speeds might be VM related, what VM software are you using and what (network) interfaces do you have in the virtualized FreeBSD?

As you see, the Linux is not in a VM, but it still downloads very slowly. So it must not be the VM's fault.
I'm using VMware, and I use bridged pattern to connect to internet for VM.
This is really a normal speed. Connecting to local servers in my country is fast, but is very slow when requesting for some foreign servers. People in developed countries are used to fast internet connections, but that's not the case for most other countries, I guess.

To summarize, can I say that:
1) Resources are limited and the legacy versions have a higher priority so binary packages for FreeBSD 10 is unavailable now. And there's no schedule for that.
2) FreeBSD doesn't hold its ports sources in its server, so I can't download ports sources from my local mirror either.

And the conclusion is for people don't have a good Internet connection as those in developed countries, there's no way to use FreeBSD 10 very conveniently for the time being.

 

[SirDice]

1) Resources are limited and the legacy versions have a higher priority so binary packages for FreeBSD 10 is unavailable now. And there's no schedule for that.

No, binary packages are available. There just are no Chinese or Asian PKGNG site mirrors.

2) FreeBSD doesn't hold its ports sources in its server, so I can't download ports sources from my local mirror either.

Looks like there are a lot of distfiles here: ftp://ftp.cn.freebsd.org/pub/FreeBSD/ports/distfiles/ No idea how complete or how up to date it is though.

 

[duleshi]

No, binary packages are available. There just are no Chinese or Asian PKGNG site mirrors.

Looks like there are a lot of distfiles here: ftp://ftp.cn.freebsd.org/pub/FreeBSD/ports/distfiles/ No idea how complete or how up to date it is though.

Could you please give an example using your local mirror? For both pkgng and distfiles.
I just don't know which config file to edit, and what the format is. As I said before, I don't find any doc for configuring mirror in detail for FreeBSD 10. FreeBSD's doc is the only one I find can match ArchLinux's wiki. But it certainly doesn't do well in teaching newbies how to configure the mirror.
I will find the local mirror myself and do the replacement, if possible.
ps: I'm using FreeBSD 10, amd64 version.

 

[SirDice]

Could you please give an example using your local mirror? For both pkgng and distfiles.

For ports I just use whatever the ports system uses by default. And my PKGNG configuration isn't going to do you any good because I build my own repositories

I just don't know which config file to edit, and what the format is.

For pkg(8) see pkg.conf(5) and the default repository file /etc/pkg/FreeBSD.conf.
For building ports try adding to /etc/make.conf:

MASTER_SITE_OVERRIDE=ftp://ftp.cn.freebsd.org/pub/FreeBSD/ports/distfiles/

I'm not sure if this is going to work but according to ports(7) the ports system should now try that site first to download sources and patches.