How do you make a local ports package collection?

B

baronobeefdip

We are just diving into FreeBSD and the way we have everything set up is that we have all our software packages (repositories) on a local server within the LAN. It doesn't connect to the internet. What we want to do is have a way for us to retrieve the entire ports source code and package collection and store them on one of our systems; that way we don't need the internet to install ports and packages.
 
This is quite unlikely. You would have better luck downloading the entire internet onto a floppy drive :)

My attempt would be to:-

Code: 
# cd /usr/ports
# make -DBATCH fetch-recursive

So should now fetch every single distfile in the ports collection (though will probably fail on a few of the strange chinese ports)

Though if you do manage it, let me know :)

edit:
You might be able to fetch the whole distfiles directory from the FreeBSD ftp. This wont include 100% everything, but will contain the majority of stuff you want.
 
We want our own local package repository for FreeBSD software like we have for Debian and fedora. We also want our repository to be able to be updated when we decide to connect to the internet to do so. When there are updates for the software packages we want to be able to download the packages and replace what we already have with the new ones.

When they are being fetched where will they be stored on the computer we're running the command on?
 
Downloaded distribution source files all end up in /usr/ports/distfiles/. Pre-built packages end up in /usr/ports/packages/. It's easy to share those using read-only NFS. You could also set up an FTP or web server and use pkg_add -r to install them.
 
I want all of the packages off of the port collection online so I can have my own package collection. And just to be clear, will the command that kpederson mentioned download all of the packages from the ports mirror? After all of the packages have been downloaded, how do I make all the BSD boxes fetch packages from the collection I just made?
 
baronobeefdip said:
And just to be clear, will the command that kpederson mentioned download all of the packages from the ports mirror
Click to expand...
No, it downloads all the distribution sources.
 
Code: 
ftp ftp.freebsd.org
cd pub/FreeBSD/ports/<arch>/packages-<release>/All
get *

At the moment, AFAIK, you cannot create a package for a port which is not installed on your system, without first installing that port.
 
When all of the packages are downloaded with the commands that da1 has suggested, will they be put in their designated directories (like xorg, gnome, etc)? What's the directory that all of the packages end up in after running those commands?

Since the packages are always fetched from an ftp server, does the operating system that the ftp server is installed and configured on matter? But it now raises questions on how to download all the packages through another OS.

But instead of downloading it from the all directory, can I have it to where it downloads to directories where they are categories like they are on the ftp site? Because the ports installer will fetch packages from directories on the ftp site in categories not in the ALL directory but correct me if I'm wrong.
 
baronobeefdip said:
When all of the packages are downloaded with the commands that da1 has suggested, will they be put in their designated directories (like xorg, gnome, etc)?
Click to expand...
No.
What's the directory that all of the packages end up in after running those commands?
Click to expand...
They end up in the current directory.

Since the packages are always fetched from an ftp server, does the operating system that the ftp server is installed and configured on matter?
Click to expand...
No.

But it now raises questions on how to download all the packages through another OS.
Click to expand...
Not sure if I follow but if you wish to download all packages to, say, server1, and "feed" them to server2, then on server2 you need to set the PACKAGEROOT variable in make.conf to point to server1 (the dir where you have all the packages) but before this, make sure you have a ftp server installed on server1 (anonymous).

But instead of downloading it from the all directory, can I have it to where it downloads to directories where they are categories like they are on the ftp site?
Click to expand...
Sure. Just go 1 directory up from "All" and you will see all the normal dirs you see in the ports tree. Download those ones :).

Because the ports installer will fetch packages from directories on the ftp site in categories not in the ALL directory but correct me if I'm wrong.
Click to expand...
Wrong. It fetches from "LATEST" but this behavior can be changed via the PACKAGEROOT variable in make.conf.
 
Okay here's the situation. After entering those commands you told me about in a previous post I'm starting to get errors. I entered this command.
Code: 
#ftp ftp.freebsd.org
Then I get the following message.
Code: 
Trying 149.20.64.73...
Connected to ftp.freebsd.org
220 Welcome to freebsd.isc.org
Name (ftp.freebsd.org:root)
I was guessing that the last prompt meant that I had to enter my root password but then I get this
Code: 
530 This FTP server is anonymous only
ftp> Login failed
I did some research and found a solution that might have fixed it but I'm not entirely sure. Here's what I entered.
Code: 
#ftp -a ftp.freebsd.org
then I get this
Code: 
Trying 149.20.64.23...
Connected to ftp.freebsd.org.
220 Welcome to freebsd.isc.org.
331 Please specify the password.
230 Login successful
Remote system type is UNIX
Using binary mode to transfer files
Where it says to specify a password, I never had to specify a password and it logged into the ftp server successfully. After that is where all of the trouble begins. I entered the cd command
Code: 
cd pub/FreeBSD/ports/i386/packages-8.2-release/All
Then I ran your get* command and this is what happened.
Code: 
ftp> get *
local: * remote *
229 Entering Extended Passive Mode (|||26839|)
550 Failed to open file.
Nothing was copied and there was an error. What did I do wrong here?
 
There are two things you need to do and they are:

1 - Use the 'i' switch to turn off interactive prompting so you will not have to confirm the transfer of every file, or you can select the 'a' option at the first prompt.
# ftp -ai ftp.freebsd.org

2 - Use mget, which transfers multiple files, instead of get, which transfers just one file.
[CMD="ftp> "]mget *.tbz[/CMD]

If, when you start your ftp session, you are not in the directory where you want the files transferred then you will have to change it.
[CMD="ftp>"]lcd /new/local/directory[/CMD]

P.S. I googled "ftp get wildcard" and the first link was The Basics of FTP, which is worth bookmarking.
 
Thanks man.

I made the decision to use FileZilla to retrieve those packages instead for efficiency reasons (progress bars and knowing which packages were skipped). If possible do you by chance know how to skip all of the packages that weren't missed when I go back I perform the retrieval again. I want the program it to skip the ones that already exist on the box and re-try downloading the ones that were.
 
I do not know if there is a way to skip files that already exist, but I get the feeling it is all-or-nothing. I went to the FileZilla web site and saw it has a "Directory Comparison" feature that will allow you to hide "identical" files so you should be able to just select the ones that are not hidden and complete your goal.
 
Alright, my professor is confusing me. He is telling me that not only should we obtain all of the files but we need to replicate what the directories are and what names they have on the ftp server. (basically copy the entire "8.2-release-packages" directory and use it as a root directory.) That's as close to replication as I think that we can get. But there is always the sysinstall method of installing packages. By that I mean that we can go into configure and tell it to retrieve from an ftp source and manually set it to fetch from our packages repo, And scan it, find everything and hopefully mark all of the dependancies after marking a program. this is what I think will work best.

Will it replicate the repository if we ran the commands like this?
Code: 
#ftp ftp.freebsd.org
#cd pub/FreeBSD/ports/i386/packages-8.2
#mget *.tbz
Unless you think going with filezilla is easier. He doesn't like filezilla so we need to know how to download them exactly as they are on the ftp server via command line. I thought that it can be run like this.
Code: 
#ftp ftp.freebsd.org && cd pub/FreeBSD/ports/i386/packages-8.2 && mget * .tbz
If this doesn't work then what is the command for copying everything no matter what type of files they are?
 
Alright now I'm confused again. How are we suppossed to install ports without access to the internet? What do we need to have downloaded and set up for it to work? I was told that installing a package alone probably won't get the program running. Like when I was trying to install and run Samba. I used sysinstall to download and install packages through an FTP site, I checked off samba35 and pressed install. The smb.conf file was never created and the smbd wasn't to be found either. What did we miss here because we want to know how to install these programs without having access to the internet.
 
smb.conf.default should be somewhere under /usr/local/share/examples. There are quite a few ports that do not create a .conf file but put one under the /examples/ directory. And smbd should be under /usr/local/sbin/ according to the port's pkg-plist.
 
So a quick re-cap.

Port Makefiles are instruction manuals (basically) that tell the machine where to fetch the source code. (I think that's what it is). And the locations where it searches for the code are ftp servers. But by default it retrieves the files from this ftp server and directory (when using 8.2).

Code: 
ftp://ftp.freebsd.org/pub/FreeBSD/ports/distfiles/

And I can override this by typing this into the make.conf file to get the ports to fetch from my local repository.

Code: 
MASTER_SITE_OVERRIDE=<address to my repo>

Where all of my distfiles are stored.

And packages are a completely different story. Some aren't on the DVD. I just noticed that recently. And in order to have a package repository, you not only need the packages, you need the INDEX file in the ftp server so the machine knows what's in the directory.

What would you call the contents of the distfiles directory? Is it source code, or is it something different?
 
We're downloading all of the distfiles. It's now settled.

Now how do we change the default ftp url for packages? (We plan on using the sysinstall method for installing packages)
 
I just tried to put in something in the MASTER_SITE_OVERRIDE field. Then when I ran the make install command to install a port it didn't go into the directory that I wanted it to.

For example I put in this.
Code: 
MASTER_SITE_OVERRIDE=ftp.192.168.1.1/FreeBSD/ports/distfiles
For a test I tried to install Pidgin by running this:
[cmd=]#cd /usr/ports/net-im/pidgin && make install[/cmd]
Instead of looking in this directory.
Code: 
ftp.192.168.1.1/ports/distfiles/net-im/pidgin/pidgin.tbz
It looks here.
Code: 
ftp.192.168.1.1/ports/distfiles/pidgin.tbz
How do I get it to automatically find and install the files from their designated directory on the ftp server?
 
The directory structure of your ftp server has to match the directory structure of the official MASTER_SITE(s). You can't have a different directory structure.
 
Well one more thing I did research on this and turned up some dead ends. how do you change the main FTP site for the package system. The main site that it will retrieve the packages for installation when we use the sysinstall method. The sysinstall method is a lot easier than command line. We don't want to punch in the address to our package server every time we decide to download and install a package.

We did download all of the packages from the official ftp site. it we downloaded the entire "packages-8.2-release" directory using filezilla. It took about a week and a half but it got done, There were just a few packages that didn't make it because filezilla keeps disconnecting from FreeBSD's ftp site and we have no idea why. But they were mostly packages from the "ALL" directory i don't think it will matter that much. All of the packages in the sub-directories are all present.

You guys have been real all this time thanks.
 
You must log in or register to reply here.
Back
Top