[Guide] Building a package repository with Portmaster

ShelLuser

Son of Beastie

Reaction score: 1,820
Messages: 3,600

Hi gang!

Editorial

When it comes to maintaining a ports tree and setting up a package repository most people rely on software such as ports-mgmt/poudriere or ports-mgmt/synth. Interesting and definitely impressive projects for sure, but to be perfectly honest I never liked them myself. For the simple reason that I think they're creating an abstract layer between the OS (and its capabilities) and you. For example: when a jail, managed by Poudriere, needs to be upgraded then you're no longer looking for a solution on how to upgrade that jail. No, you're looking for a way to upgrade that jail using Poudriere. Because many times (but not always) these programs introduce their own workflow which doesn't necessarily have to correspond with that of FreeBSD itself. Another side effect is that some people actually seem to think that setting up and maintaining a package repository is a difficult task. After all: just look at how complex both Poudriere and Synth are!

Of course it doesn't have to be that way at all. Even with a rather trivial tool such as ports-mgmt/portmaster, which is basically just a huge shell script, can you maintain your own repository. The main difference is that it'll be a little more work to set up (so I assume) and you'll need to have some basic understanding of FreeBSD itself, in this case its package management system (pkg).

But the flexibility this gives you is endless, and it easily allows for expanding on it with your own shell scripts and customized solutions. You're no longer depending on 3rd party software perse: you're purely depending on the OS itself, and nothing more.

And for the record: just because I don't like working with programs such as Poudriere or Synth doesn't automatically imply that I also think badly about them. Quite the contrary even. Seriously: it's not that hard to dislike something yet still respect and admire the project for what it is and can do.

Note that this guide is (trying to) go all the way, if you're already familiar with some topics you can just skip them according to their headers; I made sure to keep chapters pretty much self-contained so it shouldn't be too problematic.

Introduction: ports & packages

Installing software on FreeBSD can be done in two ways. The easiest and more straight forward way is to use pkg to download and install binary ("precompiled") packages from an official repository. It's simple: # pkg install <name of package>. This will tell pkg to update its information on any configured repositories (the default config file is /etc/pkg/FreeBSD.conf) and then it'll check if 'package' is actually available. If so it will be downloaded and installed.

The second way is to rely on the so called Ports collection. Now you don't grab a binary package anymore but the actual source code of whatever software you're trying to install after which it gets compiled on your system. The advantage of using ports is that you can finetune a software package to your exact needs; because the software gets build on your system it will also make sure that it will be adapted to your system.

For example: the default Samba version for FreeBSD is (at the time of writing) 4.7. So what do you think would happen if you had 4.6 installed, and then build something which uses Samba? Simple really: unless it has a specific need for the latest version then it will most likely build cleanly against your installed version and you'd end up with a software package which uses Samba 4.6 instead of 4.7.

This flexibility is the major advantage of the ports collection. ...and also its curse.

Just keep one very important detail in mind: A port is something you use to build (and eventually install) a package. You build a port which compiles some software and stores the result into a package which in its turn gets installed onto your system. So: you build ports in order to install packages.

Mixing ports & packages?

As mentioned above: Once you build a port you'll end up with a package. And packages have specific dependencies:
Code:
peter@zefiris:/usr/ports/packages/All $ pkg info -dF squid-4.3_1.txz
squid-4.3_1:
        heimdal-7.5.0_2
        openssl-1.0.2p_1,1
        p5-DBI-1.641
        pcre-8.42
        perl5-5.26.2
        samba46-4.6.16_1

This Squid package was build on my server which still uses Samba 4.6, as you can see. So what do you think would happen if you'd install this package onto a server which has 4.7 installed? Well, then you'd end up with a conflict, probably resulting in the removal of Samba in favor for the previous version. Which in its turn would result in the removal of every other package which relied on that Samba version (4.7). Which, depending on your setup, could have quite a cascading effect:
Code:
peter@zefiris:/home/peter $ pkg info -rx samba
samba46-4.6.16_1:
        fusefs-smbnetfs-0.6.1_1
        kde-runtime-kde4-16.12.3_13
        squid-4.3_1
>       ffmpeg-4.1,1
        mplayer-1.3.0.20180920_2
        kodi-17.6_7
        libxine-1.2.9_4
        vlc-2.2.8_14,4
peter@zefiris:/home/peter $ pkg info -rx ffmpeg-
ffmpeg-4.1,1:
        thunderbird-60.2.1_3
        firefox-63.0_3,1
        seamonkey-2.49.4_15
        alsa-plugins-1.1.1_4
>       gegl-0.4.8_2
        kdemultimedia-ffmpegthumbs-kde4-4.14.3_7
        kfilemetadata-kde4-4.14.3_19
        libstreamanalyzer-0.7.8_13
        mplayer-1.3.0.20180920_2
        nepomuk-core-kde4-4.14.3_20
        opencv-3.4.1_9
        openjfx8-devel-8.u172.b11_5,1
        gallery2-2.3.2_8
        libxine-1.2.9_4
        vlc-2.2.8_14,4
peter@zefiris:/home/peter $ pkg info -rx gegl
gegl-0.4.8_2:
>       gimp-app-2.10.6_1,1
        py27-gimp-2.10.6_1
peter@zefiris:/home/peter $ pkg info -rx gimp-app
gimp-app-2.10.6_1,1:
        gimp-focusblur-plugin-3.2.6_6
        gimp-data-extras-2.0.2_7
        gimp-gutenprint-5.2.14
        gimp-beautify-plugin-2012.08.12.00_7
        py27-gimp-2.10.6_1

See what I mean? Samba is required by ffmpeg, which in its turn is required by gegl which in its turn is required by gimp-app. So by removing Samba I'd also lose Gimp (and my entire KDE4 installation).

This is the main reason why mixing ports and (precompiled) packages to install software is a bad idea: packages were build with specific (static) dependencies whereas ports basically 'adapt' themselves to your system. So mixing those two together can (and will) cause a conflict of interest, resulting in dependencies which are not compatible with each other and it will break things.

The best way to counter for this? Well, simple: don't mix ports and packages.

Of course sometimes you simply have to. Take Apache: its default SUEXEC path (= path from which you can execute CGI programs using another UID, see also suexec(8)) may not be what you'd like to use. So the solution should be obvious: define SUEXEC_DOCROOT in /etc/make.conf, build Apache using the ports collection and you're done. This is one of the reasons why I rely on the ports collection for all my servers myself: the massive amount of flexibility which you get.

However, there are 2 problems with this approach where the first is solved by Portmaster; building ports can be quite a hassle at times.

What is Portmaster?

Portmaster is a shell script which can make it easier to install and maintain ports on your system. Although building a port is relatively easy (basically it only requires you to run make install clean from within the port directory) it can become quite an issue if it has many dependencies which need to be configured.

If you forgot to sort this out right away by running make config-recursive and instead used the make command I mentioned earlier then you may end up with a new config screen after 20 minutes of compiling. You know: around the time when you already decided to do something else, like making a cup of tea and doing the dishes. Only to come back 30 minutes later to suddenly discover that you could have sped up the building process if you had sorted out that config screen right away. This is why Portmaster always shows unconfigured ports before it starts the building process.

But it doesn't stop there. Portmaster can also make a backup of a currently installed package right before you upgrade it with a newer version. So if the new version somehow messes up your system then you can simply uninstall it and re-install the backup package. Which obviously leads to yet another option: copying the package you just build & installed to a specific location: /usr/ports/packages/All.

Portmasters behavior can be configured using its config file, you'll find an example in /usr/local/etc.

A package repository?

The second of the two problems I mentioned above manifests itself if you have more than one server to maintain. Sure: Portmaster can make building ports a lot easier but wouldn't you agree that building ports on both machines is basically a waste of resources? Especially if they both use the same architecture.

Wouldn't it be nice if you could somehow build your ports on the fastest of the two servers and then point the other to all those packages you already build and stored away? Guess what? You can by setting up your own package repository! And trust me: it's much easier than it might seem.

So what exactly is a package repository? Obviously a collection of packages (duh!) but what else would we need? If we take a look at /etc/pkg/FreeBSD.conf we'll come across an URL entry: url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly". So what is this ${ABI} section? Obviously something that can separate our FreeBSD version from the rest, and according to pkg.conf(5) it is "derived from the ABI of the /bin/sh binary", which still doesn't tell us anything useful ;)

I started some more digging and eventually ended up on pkg-config(8) which finally gave me the answer (in its EXAMPLES section even!):
Code:
peter@zefiris:/home/peter $ pkg config abi
FreeBSD:11:amd64

Why bother going through all this trouble you wonder? Because this allows us to look at a repository with our own eyes: just point your browser to http://pkg.freebsd.org/FreeBSD:11:amd64/quarterly. Notice the All directory in there? Unsurprisingly enough it is the same name which Portmaster uses to store its own package files, and it is the key to creating our own repository.

But there's more: you'll also notice several files in the root directory: digests.txz, packagesite.txz and meta.txz, these files tell your client(s) all they need to know about your repository: the packages it contains, (cryptographic) checksums for those packages (which allow clients to check the validity) and of course all the information about the packages (such as its dependencies).

Now that we know what we're up against and have identified all our requirements and the tools at our disposal it's time to get some work done!

Setting up our own package repository!

You know why I love FreeBSD so much and why it quickly became my de-facto favorite operating system? Because it's real, and build for you (us) to use, it provides you with everything you need to truly make it your operating system. You'll see...

So: what we're trying to achieve here is to provide an easier way to replicate our collection of software to be used on another server. We could just start building & installing new packages and providing them in our repository but that creates a new problem: dependencies.

Remember that Squid package and its dependencies I showed earlier? It was build against Samba 4.6 even though the default version (time of writing!) is 4.7. So if you'd install that on a server which doesn't have Samba installed then how is it supposed to gain access to a version which is no longer officially supported?

Therefor we're going to start by populating our repository with a copy of all the packages we currently have installed. It's easy, no worries: # pkg create -ao /usr/ports/packages/All.

This basically grabs the software as it is currently installed on our server, re-packages it and then places that in /usr/ports/packages/All. So if you somehow changed (and thus invalidated) some of your packages then that could be a problem.

Example: I despise the info system, truly loathe it.

(warning: heavily opinionated bias follows! :rude:)

To me info truly showcases the immaturity of the Linux community. Instead of working within the limits of the manual page system and showing the discipline to adapt to an already set out standard which outdates Linux by years some (lazy?) people decided that it wasn't good enough and thus info was born. Awesome: now us users need to search for our information in 2 different places, such a display of userfriendlyness. Unix manualpages have apropos (or man -k <keyword> (see man(1) (or: man man (I love that!))), this makes searching for keywords (even in specific sections) quite easy. How it works? Using makewhatis(8) (mentioned in apropos(8)).

info has... -k which is said to allow a look up using indices, just a shame though that even info info doesn't mention anything about how to actually create those monstrosity indexes.

</rant>. Sorry, had to be done :what:

Given the above it shouldn't come as a surprise that I actually trashed /usr/local/info, Ran pkg which -o `which info`, discovered that it was print/texinfo which provided this mess, but unfortunately removing that package didn't have any effect on /usr/local/info.

As a result I had invalidated plenty of packages which resulted in pkg-create(8) having to give up on trying to (re)create those packages because of their incomplete ("corrupted") state. Consider yourself warned :)

But for every other sane admins who don't enjoy trashing their own system this shouldn't be a problem at all :D

Providing the repository to the network

"The network is the computer", an opinionated statement by Sun Microsystems which I think still holds true today, especially for Unix environments such as FreeBSD.

When checking out /etc/pkg/FreeBSD.conf you can easily see that providing a repository can be easily done through both FTP and HTTP. Because my server already has an Apache instance running I prefer using HTTP.

Now, you might be considering to simply provide /usr/ports/packages to the network but that's actually not a good idea, especially if you're using Portmaster. See: Portmaster can also maintain backups for any packages which you upgrade, and it keeps this collection in /usr/ports/packages/portmaster-backup. A location which would automatically be included.

Instead I prefer creating /usr/local/www/repo, then adding a symbolic link using: # ln -s /usr/packages/All /usr/local/www/repo. If you still want to include the Portmaster backup into your repository then you can: just create another symlink. The key here is that you are in control of things.

Of course we do need to keep the use of links in mind with our web server. This is the configuration I use on my network (I maintain my own local domain):
Code:
<VirtualHost 10.0.1.5:80>
        ServerName pkg.intranet.lan
        ServerAdmin pl@intranet.lan
        CustomLog "/var/log/httpd/pkg.log" combined
        ErrorLog "/var/log/httpd/pkg_error.log"

        DocumentRoot "/usr/local/www/repo"
        #
        <Directory /usr/local/www/repo>
                AllowOverride none
                Options Indexes SymlinksIfOwnerMatch
                Require ip 10.0.0.0/23
        </Directory>
</VirtualHost>



Making sure new packages get added

There are a lot of different ways in which you can do this, but I personally prefer to fully use my 'test' or 'stage' server as the primary source for the other. Meaning that ports get build, installed & used on this server and once I'm convinced that things work as expected then they're pushed onto the repository and provided to the rest of the network. For me it's an issue of control and reliability.

But that doesn't mean that you can't deviate from this. It's really not that hard to configure a Jail dedicated to building ports. However, I'm not going to address this just yet (maybe in an upcoming part) because of my own methedology.

So what I prefer to do is instruct Portmaster to add any packages I installed on the server. Couldn't be easier, just copy /usr/local/etc/portmaster.rc.sample to portmaster.rc (in the same directory of course) and then enable this:
Code:
# Make and save a package of the new port (-g)
MAKE_PACKAGE=gopt

And you're done! :cool:

Repository maintenance

We are now 2 steps away (excluding client configuration) from fully using our new repository. First we'll need to create the index and manifest files which will allow clients to retrieve an overview of the packages which our repository provides.

This is easily done by using pkg-repo(8) which will create all the index files for us. But... Security is a concern too. How will your other servers determine that the packages from your repository are truly yours and not something that has been tampered with? Simple: by telling pkg to sign the fingerprints using a private key and then providing the public key to our other servers.

As such we're going to need to create a public/private key pair and what better way than letting openssl handle this for us?
Code:
# mkdir /root/repo && chmod 600 /root/repo
# openssl genrsa -out /root/repo/repo.key 2048
# openssl rsa -pubout -in /root/repo/repo.key -out /root/repo/public.key

Next stop: generating the system files (indexes, manifest, etc.) and sign them so that our clients will know what to expect from our repository and can ensure themselves that it's truly our work. Prepare for an incredibly difficult command:
# pkg repo /usr/local/www/repo /root/repo/repo.key.

It's honestly that easy! :confused: Need to update your repository? Just dump in new packages and then re-run the above command, done! Of course you can expand on the encryption if you'd like but for a straight forward repository this should do nicely.

At this point we're done setting up our repository but also created a bit of a problem. See: new packages get added whenever you upgrade an existing package (or install a new one). But what about those old ones which got replaced?

Portmaster can automatically clean up your distfiles (= original sourcecode files which got downloaded to build a port) and keep that in check with stuff you actually have installed. It can also clean up your packages to clean up stale entries. And it can even clean up port options (/var/db/ports) and the package database in /var/db/pkg. There's a reason why I love this stuff ;)

But it cannot (and won't) clean up your repository, because that's not its job in the first place. Now what?

This is why I prefer to roll it all up into one "maintenance script": check for duplicate packages and move those out of the way, then update the actual index files. Hmm, I did a little bit more because I also kept manual updates in mind but.. this should be usable nonetheless:

Code:
#!/bin/sh

## Maintenance.repo v1.0
##
## This script removes double entries from the repository
## directory and then re-builds the index files.

### Init section

IFS=$'\n'
dir="/usr/ports/packages/All"
backup="/usr/ports/packages/Double"

### Functions

prune_repo() {
# Move double entries out of the way
for a in $(ls $dir | sed -E 's/-[0-9]+\..*\txz//g' | uniq -cd); do
        amount=`echo $a | cut -w -f2`;
        entry=`echo $a | cut -w -f3`;

        mv $(ls $dir/$entry* | head -n $(($amount-1))) $backup
done
}

rebuild_repo() {
# Rebuild repository index
pkg repo $OPT /usr/local/www/repo /root/repo/repo.key
}

### Main section

if [ "$1/" == "/" ]; then
        OPT="-q"
        prune_repo;                     > /dev/null
        rebuild_repo;                   > /dev/null
else
        if [ $1 == "u" ]; then
                rebuild_repo;
        fi
fi

In case you didn't notice: in order to make this work you'll also need to create /usr/ports/packages/Double, this is the location where older ('double') packages from the repository get moved to so that you can perform a manually check if needed.

This script, and the actual cleaning, is started through a cron job (what else?):

Code:
# m h d m dow

# Perform pkg repository maintenance/every 4 days
30 5 * * */4    /root/bin/maintenance.repo

# Remove outdated (= age > 1 month) backup packages/once per month
30 6 * */1 *    /usr/bin/find /usr/ports/packages/Double -ctime +4w -delete

Don't know how to use this crontab? :rolleyes: Oh dear: # crontab -e, then paste in the above and you're all set.

Sure... Using Poudriere and/or Synth may have been much easier on you, definitely not going to deny that possibility. But I will say that setting all of this up was a lot more fun for me.

And there you have it!

And now a word from my sponsors! :D

opensource.jpg
This post was written using exclusively open source products. I know that might not sound as a big deal, but it's the beginning of the weekend and I'd like to share some appreciation for all the hard work put into these projects by their authors, developers and fellow fans ;)
  • This post was mostly written, edited and proof read (famous last words) using Kate, the KDE Advanced Text Editor.
  • Access to this forum was handled by Seamonkey; an integrated Internet suite providing a browser, e-mail client, IRC client and an HTML editor.
  • The above software was running on KDE4, courtesy of the KDE project. An open source (duh!) desktop environment which most likely even feels good for the die hard Windows users.
  • And all of the above ran on none other than FreeBSD 11.2; the operating system which all of us came to love & respect (or so I assume).
    • Fun fact: all the software mentioned above is available through FreeBSD's ports collection and/or binary software repository.
Thanks for reading!
 
Last edited:
OP
ShelLuser

ShelLuser

Son of Beastie

Reaction score: 1,820
Messages: 3,600

And what about the other servers (clients)?

So while re-reading and fixing some typoes I figured that I should also cover the client side in order to fully complete this tutorial. A short post but this way I won't risk going over the maximum post limit (which I think I might have almost reached).

The first thing you'll need to do is provide the public key you made to those other servers. You know: /root/repo/public.key. I tend to keep these public keys in /usr/local/etc/pkg/keys.

Next is to disable the official (default) repository. You know: /etc/pkg/FreeBSD.conf? You can do that by creating a file in /usr/local/etc/pkg/repos which contains this:
Code:
FreeBSD: {
        enabled: no
}


And then finally you'll need to add your own repository. You will need to provide the location from which the repository can be accessed, the method (or protocol) which is being used, point the system to the public key you just downloaded and finally you'll need to enable the repository.

Something like this:
Code:
## IntraNet PKG repository

Zefiris: {
  url: "http://pkg.intranet.lan",
  mirror_type: "http",
  signature_type: "pubkey",
  pubkey: "/usr/local/etc/pkg/keys/zefiris.pub",
  enabled: yes
}

I think this pretty much speaks for itself, right?

Update your setup using # pkg update -f and then check to see if you can access specific packages:

Code:
root@breve:/home/peter # pkg update -f
Updating Zefiris repository catalogue...
Fetching meta.txz: 100%    560 B   0.6kB/s    00:01
Fetching packagesite.txz: 100%  388 KiB 397.0kB/s    00:01
Processing entries:   0%
Newer FreeBSD version for package zziplib:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1102000
- running kernel: 1101001
root@breve:/home/peter # pkg search -d squid-
squid-4.3_1
Comment        : HTTP Caching Proxy
Depends on     :
        openssl-1.0.2p_1,1
        heimdal-7.5.0_2
        samba46-4.6.16_1
        perl5-5.26.2
        pcre-8.42
        p5-DBI-1.641
root@breve:/home/peter # pkg info -x samba
samba46-4.6.15
root@breve:/home/peter # pkg search -q samba
samba46-4.6.15_1

Notice Samba 4.6 up there? Now where have we seen that before ;)

For the record: yeah, I have an issue right now with regards to a version mismatch, but that will be resolved soon enough when my breve server gets a full update.
 
Top