Poudriere repository management

[nanotek]

I've installed Poudriere and started building my own repository. How do you properly use the default FreeBSD repo and your own repo seamlessly?

For example, here is my repo.conf file on the local machine:

cat /usr/local/etc/pkg/repos/repo.conf

repo: {
    url             : "file:///zbank/poudriere/data/packages/10x64-default",
    enabled         : yes,
    signature_type  : "PUBKEY",
    mirror_type     : NONE
    pubkey          : "/usr/local/etc/ssl/certs/repo.crt"
  }

If I keep the default FreeBSD.conf repo file enabled:

cat /etc/pkg/FreeBSD.conf

# $FreeBSD: releng/10.0/etc/pkg/FreeBSD.conf 258710 2013-11-28 14:24:26Z gjb $
FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

How do I direct pkg to install which packages from which repository? Or do you have to manually disable (by editing the ?.conf file and changing enabled to no) whichever repo you don't want the packages to be installed from before issuing any pkg install commands?

For example, say I want to install nginx-devel from my own repo (because I've built the package with custom options) but install perl5 from the default FreeBSD repo (because I use the default options) how is this achieved?

I see no point in building custom packages with the default build options as they're readily available from the default FreeBSD repo, so would like to use both repositories on an as needed basis. But remembering and having to manually disable and enable the right one before each pkg install seems inefficient. Further, it seems pkg upgrade could turn into a real mess when using multiple repositories.

From pkg(8):

install
	     Install a package from a remote package repository.  If a package
	     is	found in more than one remote repository, then installation
	     happens from the first one.  Downloading a	package	is tried from
	     each package repository in	turn, until the	package	is success-
	     fully fetched.

How do you set your own repository as the "first one"?

Additionally, the custom built packages and repository are stored on the host despite being built in the jail. What's the most effective way of making this repository accessible from a web server inside a jail on the host system?

 

[SirDice]

How do you properly use the default FreeBSD repo and your own repo seamlessly?

You disable the FreeBSD repository. You don't want to mix them, especially not if you set custom options on your own packages. It's going to get very messy if you try to mix them.

 

[nanotek]

That's quite surprising. I would have expected the next generation package management utility to not be so rudimentary. I expected a switch to specify which repository to use (e.g. pkg install -R $REPO1 bind98 or pkg install -R $REPO2 portaudit). As it is, to use your own Poudriere repository you need to build all ports that use default options -- this is remarkably inefficient.

I understand many features of pkg aren't yet implemented, do you know if this ability is under development?

 

[kpa]

It is under development. It's not as trivial to implement as one would expect because again it's all about dependency and conflict resolving and that's one area where PKGNG system is still far from complete. The old package system had a very simple and naive concept of what a dependency to another port is and the PKGNG system doesn't yet try to do anything fancier than what the old package format did (actually PKGNG is aware of few more things like shared libraries provided by packages but they are not yet used to their full potential). Basically the only kind of dependency that is supported is one where the package name and the version must match exactly. For example net/mtr requires devel/pcre. Now when the first one is built into a package a dependency to devel/pcre is included in form of name equals pcre and version equals 8.34 (with the ports tree of this date). Obviously the only way to satisfy this dependency is to install a pcre-8.34 package in one way or another, no other version of devel/pcre will satisfy this dependency. There's no way to express things like "version of devel/pcre must be greater or equal to 8.0" in a package and that's exactly kind of thing you'd want to be able to express as a port maintainer because it would mean the user could have a pcre package from some other repository that is not necessarily of the newest version (or the version of devel/pcre the net/mtr port was compiled against to produce the mtr package) and it would be compatible as long as the version number in it satisfies this condition.

Debian's dpkg packaging system has known these tricks for a long time and that's why you can mix packages from different sources in Debian and all Debian-based Linuces with some freedom and the system usually does the right thing for you.

 

[dpejesh]

It's not that its rudimentary, it's that all you're asking for is trouble. Have you ever tried to manage multiple package repositories with apt or yum that had overlapping packages? It gets real ugly, real quick. You can see some of the troubles people are having with trying to build custom ports while trying to use binary packages at the same time, its not much different with other package tools when mixing and matching repositories. One of the things they have that pkg doesn't which would probably be useful in your situation is repository priorities, even then though its very easy to get into a mess. I've been at places where just dealing with our custom packages and fixing them against updates coming from CentOS was nearly a full time job at times.

The upside to using ports-mgmt/poudriere to manage your own complete package repository that you don't get with other package systems is that you have full control over everything that gets built; which is something you can't easily do, if at all, on some Linux distros. Which is why they normally default to enabling every feature in their packages. With the tools we now have in FreeBSD it's a completely different methodology to managing systems. Instead of thinking of it as managing individual packages and putting them in a separate repository, you're managing an entire package set which has been built together as a unified unit. You get all the benefits and flexibility of ports while at the same time having full binary package management. It gets even better once you start using ports-mgmt/portshaker to overlay your own custom ports tree on top of the official one. Instead of managing your custom packages and trying to inject them into the packaging tool with a separate repository, you're injecting them into the ports tree and they get built along with everything else at the same time. Since they're all built together you will never run into a problem with random packages trying to get pulled in or random conflicts due to dependencies and resort to playing games with the packaging tool (eg. pkg lock). All those problems are being pushed into poudriere at build time. You find problems during the build process when something is broken instead of at install/runtime. A simple example of what I'm talking about, is if you have your own packages for software your company has built in house and upstream decided to update a library that you depend on, when the library gets upgraded on your systems your custom built package sees that the new version is greater than what it expects and thinks all is fine, but when your app runs it can't find the library and fails to start. If you use portshaker to overlay your ports tree, poudriere to build everything together, then when you perform an upgrade not only does the library get updated but everything that depended on it including your custom software. The recent devel/icu problem is somewhat related to what I'm referring to. The issue they had just didn't exist for poudriere users who were using it to manage a complete package set.

Hopefully this makes sense. The point I'm trying to make is you should be looking at poudriere a little different than what I think you're expecting out of it. Use it to build a unified package set instead of as a tool to generate one off packages. Once you use it that way you'll really start to understand how awesome it is and how there's really nothing else like it (at least not that I've found).

 

[nanotek]

It's not that its rudimentary, it's that all you're asking for is trouble. Have you ever tried to manage multiple package repositories with apt or yum that had overlapping packages? It gets real ugly, real quick. You can see some of the troubles people are having with trying to build custom ports while trying to use binary packages at the same time, its not much different with other package tools when mixing and matching repositories. One of the things they have that pkg doesn't which would probably be useful in your situation is repository priorities, even then though its very easy to get into a mess. I've been at places where just dealing with our custom packages and fixing them against updates coming from CentOS was nearly a full time job at times.

I invariably mix repositories on Debian, OpenSUSE and, in the past, on my LMDE and Ubuntu notebooks, without this "trouble" that is a concern on FreeBSD; a problem I thought pkg had as one of its objectives to resolve. More importantly, however, I don't look to Linux package management utilities for comparison: PKGNG is a very recent development and despite the shortcomings of pkg_* when compared to Linux package management, pkg should be more concerned with introducing new abilities for FreeBSD -- not simply trying to catch up with Linux. An ability to mix repositories is not only a realistic option, but an extremely efficient one and, in my opinion, not one that should be summarily dismissed to the 'kill' bin on the PICK chart. It's only "trouble" because the program is incapable of doing things more efficiently.

The upside to using ports-mgmt/poudriere to manage your own complete package repository that you don't get with other package systems is that you have full control over everything that gets built; which is something you can't easily do, if at all, on some Linux distros. Which is why they normally default to enabling every feature in their packages.

And it's remarkable that FreeBSD doesn't do this vice some of the absurd options they choose for some packages. Making a package available with all options is more prudent than making a package available absent any one option. Why do I need full control over compiling a binary package if that binary package is already available with desirable options? I only need that control for binary packages that are not available with desirable options.

With the tools we now have in FreeBSD it's a completely different methodology to managing systems. Instead of thinking of it as managing individual packages and putting them in a separate repository, you're managing an entire package set which has been built together as a unified unit. You get all the benefits and flexibility of ports while at the same time having full binary package management.

All ports do is provide the machine with binary packages built with desired options because the FreeBSD repository doesn't provide them. If said binary packages are already available in the FreeBSD repository with said desired options, you don't need the flexibility of building from ports. Building them with Poudriere is inefficient as you're not getting the benefits of port flexibility if using packages with default options: you're double-handling, so to speak. If Poudriere is about providing binary packages, why build binary packages that are freely available? That's poor management.

It gets even better once you start using ports-mgmt/portshaker to overlay your own custom ports tree on top of the official one. Instead of managing your custom packages and trying to inject them into the packaging tool with a separate repository, you're injecting them into the ports tree and they get built along with everything else at the same time. Since they're all built together you will never run into a problem with random packages trying to get pulled in or random conflicts due to dependencies and resort to playing games with the packaging tool (eg. pkg lock). All those problems are being pushed into poudriere at build time. You find problems during the build process when something is broken instead of at install/runtime. A simple example of what I'm talking about, is if you have your own packages for software your company has built in house and upstream decided to update a library that you depend on, when the library gets upgraded on your systems your custom built package sees that the new version is greater than what it expects and thinks all is fine, but when your app runs it can't find the library and fails to start. If you use portshaker to overlay your ports tree, poudriere to build everything together, then when you perform an upgrade not only does the library get updated but everything that depended on it including your custom software. The recent devel/icu problem is somewhat related to what I'm referring to. The issue they had just didn't exist for poudriere users who were using it to manage a complete package set.

Hopefully this makes sense. The point I'm trying to make is you should be looking at poudriere a little different than what I think you're expecting out of it. Use it to build a unified package set instead of as a tool to generate one off packages. Once you use it that way you'll really start to understand how awesome it is and how there's really nothing else like it (at least not that I've found).

I understand your explanation, which is fine if you're responsible for many systems. If you're managing a handful of machines that are not uniform in specifications, it's a waste of time and resources. Poudriere is good to build customized packages that are not available in the FreeBSD repository. It is inefficient to build packages with the default options when these packages are readily available. It's not good management at all.

If Poudriere is so awesome because it provides functionality the base system lacks, the (now default) package management utility should be improved to capitalize on the infrastructure that already exists. PKGNG is capable of doing this, whether the developers want to put in the work to do it is up to them. But it's not unrealistic and offers a high impact/reward outcome if implemented.

 

[SirDice]

The problem is that once you built a port with certain options it's going to depend on other packages. Unfortunately these dependencies are set in stone and are only changeable when you build from ports. It's not so much PKGNG that's incapable, it's simply the way the ports system currently works.

As for turning on all options all the time, why would I want to have Gnome compatibility when I don't even use Gnome? Sure you can build Application A with all the bells and whistles but then it's going to depend on Application B to make that happen. If Application B isn't installed Application A will simply refuse to work. So you'll need to have two versions of Application A, one with the dependency on B and one without. You can build both without problems but the resulting packages will be named exactly the same in both cases making it very difficult to distinguish between the two.

 

[nanotek]

The problem is that once you built a port with certain options it's going to depend on other packages.

I understand this, but how often are these other packages that a custom built port depends on simply built with the default options? Quite often. Further, what about the origin ports (i.e. not dependencies) that are simply built with default options. In the case of the latter, there is no insurmountable reason why they should not be able to be installed with pkg(!) alongside port built or Poudriere built binaries. And in the case of the former, improvements in both the ports tree and PKGNG would also make such efficient system management possible. To be fair, regarding the former scenario, these dependencies will be built automatically by Poudriere anyway, and this is not the core of my argument. It's the latter scenario that definitely requires attention.

Unfortunately these dependencies are set in stone and are only changeable when you build from ports. It's not so much PKGNG that's incapable, it's simply the way the ports system currently works.

See above. If dependencies only require default options (i.e. that which the FreeBSD repository binary packages come with) why do they need to be built? In theory, they don't.

As for turning on all options all the time, why would I want to have Gnome compatibility when I don't even use Gnome? Sure you can build Application A with all the bells and whistles but then it's going to depend on Application B to make that happen. If Application B isn't installed Application A will simply refuse to work. So you'll need to have two versions of Application A, one with the dependency on B and one without. You can build both without problems but the resulting packages will be named exactly the same in both cases making it very difficult to distinguish between the two.

Fair point. However, thousands of packages (if provided with all options) don't invoke such a reponse. In any event, PKGNG intends to (eventually) make this type of functionality a reality, so it's already agreed that improved package management is needed.

(!) ETA:
with pkg from the default FreeBSD repository.

 

[SirDice]

In any event, PKGNG intends to (eventually) make this type of functionality a reality, so it's already agreed that improved package management is needed.

Definitely. Keep in mind that packages have always been somewhat of an afterthought on FreeBSD. Most of us have been happily building from ports since the beginning. It's only in the last couple of years packages have become more commonly used. The current PKGNG, with all it's missing drawbacks, is already a huge improvement over the old system. But there's still a lot of work to be done (which is one of the reasons why it's still a port and not yet part of the base OS).

 

[nanotek]

I'm definitely impressed and equally looking forward to what is still to come. I'm mostly thinking aloud and enjoy the comments from everyone here who has immeasurably more experience and insight than I do with FreeBSD. Plus, if developers read these forums, user feedback (both right, wrong, feasible and downright impossible) might add to their ideas and one day become something beyond what was even wished for in the first place.

You make a very valid point regarding ports vice packages in FreeBSD. It is very interesting reading the distinct positions the different BSD communities have regarding packages. OpenBSD devs laugh at the postulation that compiling programs offer any improved performance or usability and wholeheartedly advocate binaries. I think FreeBSD might be moving toward this position, albeit slowly, as the cost:reward ratio certainly favours binary packages and most end users realize this.

 

[SirDice]

I have been building my own packages for quite a number of years now. With the old system it was very difficult to update/upgrade packages once they were installed. You had to resort to sysutils/bsdadminscripts and even that was a pain. So I usually ended up doing a pkg_delete -a and re-adding all the packages I needed, just to prevent the "dependency-hell". Now I can do pkg upgrade and the system takes care of everything. It even notices option changes in dependencies and re-installs them when needed.

Regardless of the state of the official package repositories, I'm probably going to continue building my own. There's just something strangely hypnotic about watching compiler output scrolling over your screen

 

[dpejesh]

I invariably mix repositories on Debian, OpenSUSE and, in the past, on my LMDE and Ubuntu notebooks, without this "trouble" that is a concern on FreeBSD; a problem I thought pkg had as one of its objectives to resolve. More importantly, however, I don't look to Linux package management utilities for comparison: PKGNG is a very recent development and despite the shortcomings of pkg_* when compared to Linux package management, pkg should be more concerned with introducing new abilities for FreeBSD -- not simply trying to catch up with Linux. An ability to mix repositories is not only a realistic option, but an extremely efficient one and, in my opinion, not one that should be summarily dismissed to the 'kill' bin on the PICK chart. It's only "trouble" because the program is incapable of doing things more efficiently.

I didn't intend to imply that pkg should be trying to catch up to Linux in any way. I was just referring to packaging tools that most people would be familiar with. I also wasn't trying to say this shouldn't be a supported feature, just explaining how it can quickly get tricky even with systems that don't have as much volatily that the ports tree does. Distros like Debian which stick with the same package set for many years and only updates them for security vulnerabilies can still become problematic when trying to replace a specific package with a newer version. Hell, they even started a team dedicated to it to make it easier for people.

And it's remarkable that FreeBSD doesn't do this vice some of the absurd options they choose for some packages. Making a package available with all options is more prudent than making a package available absent any one option.

In theory it'd be nice to enable all options in a package, but in practice it's not realistic. People would complain about why mysql and postgresql are being installed when all they want is the embedded sqlite option. Some ports aren't even capable of supporting multiple database backends like zabbix for example. Or should git by default enable the gui option which pulls in tk, which then pulls in a bunch of xlibs when it's primarily a CLI utility. In general some of the options probably could be enabled by default and it would be a good idea for there to be a guideline that porters should follow, but everybody's opinions will differ depending on how they use the port or what gets pulled in as a dependency. If you can improve that then I implore you to contribute to the ports team. It's not an easy problem to solve once you start looking at each port and deciding what should be enabled. There will always be someone who thinks its assbackwards because it doesn't fit their specific needs or pollutes their system with cruft they don't want.

Why do I need full control over compiling a binary package if that binary package is already available with desirable options? I only need that control for binary packages that are not available with desirable options.

All ports do is provide the machine with binary packages built with desired options because the FreeBSD repository doesn't provide them. If said binary packages are already available in the FreeBSD repository with said desired options, you don't need the flexibility of building from ports. Building them with Poudriere is inefficient as you're not getting the benefits of port flexibility if using packages with default options: you're double-handling, so to speak. If Poudriere is about providing binary packages, why build binary packages that are freely available? That's poor management.

It is inefficient to build packages with the default options when these packages are readily available. It's not good management at all.

This gets a little more complicated with how you can manage the package builds. Just because the options haven't changed in a specific package, the way the dependencies are built gets tricky. Like setting APACHE_PORT or PHP_VER in your make.conf will completely change your binary package dependencies while leaving the options unchanged. If I remember right, this type of situation could be fixed with changes to the ports tree but would end up breaking the old pkg_* tools which is why it's not supported yet.

If Poudriere is so awesome because it provides functionality the base system lacks, the (now default) package management utility should be improved to capitalize on the infrastructure that already exists. PKGNG is capable of doing this, whether the developers want to put in the work to do it is up to them. But it's not unrealistic and offers a high impact/reward outcome if implemented.

poudriere is awesome, but the pkg limitations aren't related to it. Don't forget that this is still a relatively new package utility, one that is working around the legacy pkg_* tools, and doing a decent job of it given the circumstances. Can you imagine what would happen if some Linux distro tried to switch their packaging tool and support both at the same time? The fallout would blow reddit up overnight. I don't see how you can claim the developers don't 'want to put in the work' to improving pkg. I've had very few problems with it and can't thank them enough for their efforts. Sure there are some things that are missing and some growing pains, but to expect everything to work while being limited in what they can actually do to the ports tree to support every feature from the start is unreasonable.