Pkg package repository using ports-mgmt/poudriere. With or without ZFS.

K

kpa

This is a HOWTO on how to build your own ports-mgmt/pkg packages in a jail(8) using ports-mgmt/poudriere.

Few notes about notation:
  • First few mentions of a ported program will be a "port link", like net/mtr. I will not repeat the link too many times however.
  • I will use a link to a manual page when introducing a new command, like mtr(8). Unfortunately the official manual page are lagging badly behind what's actually available on a recent FreeBSD and many of the links below don't work. You can however take the name of the command from the link and check the manual page on your own system that will have up to date manual pages.
  • I will use a couple of variables to avoid repeating paths like /usr/local/poudriere. I will use ${LOCALBASE} for the path of the default location for installed ports, /usr/local. I will use ${BASEFS} for the root folder where poudriere stores ports trees, jails, packages and few other things, default ${LOCALBASE}/poudriere (but see below for the BASEFS setting in the configuration file). I will use ${jailname} in few places where file names vary depending on the name of jail used.

What is ports-mgmt/poudriere? The port description reads as follows:

Code: 
poudriere is a tool primarily designed to test package production on
FreeBSD. However, most people will find it useful to bulk build ports
for FreeBSD.

WWW: https://github.com/freebsd/poudriere/wiki

This description is now quite out of date since the official FreeBSD binary packages are now built with it. Poudriere is a package building system intended for anyone who wants to build their own binary pkg packages for FreeBSD. Its design is as follows:
  • Mostly sh(1) scripts for portability.
  • Uses jail(8)s for creating clean build environments for ports building.
  • Makes heavy use of nullfs(5) to avoid unnecessary copying.
  • Makes heavy use of ZFS features like snapshots and cloning if ZFS is used.
  • Has built-in utilities for setting up and updating ports(7) trees and jail(8)s.
  • Customizable trough a configuration file (${LOCALBASE}/etc/poudriere.conf, ${LOCALBSE} defaults to /usr/local) and a dedicated settings directory (${LOCALBASE}/etc/poudriere.d).
  • Can be run as a daemon that serves build requests.

Why is poudriere needed in the first place? The ports infrastructure suffers from numerous design faults that are not easy to overcome without a package building tool like poudriere. These faults include:
  • Building of ports as outlined in ports(7) is performed on the live system. Many build failures stem from this when an already installed port or leftover files from an old version causes conflicts with the to be built new version of the port. Poudriere avoids this problem by starting every build from a clean state as if the ports were installed for the first time on the system.
  • Building ports on the live system pulls in a large number of build time dependencies that are not required to be installed after the builds have finished. These build time dependencies can cause configuration issues and trigger build failures when port dependencies and options change. Again, poudriere avoids this by starting every build from a clean state.
  • Building ports on the live system may cause service interruptions while the ports are being built and the situation may not be corrected until all of the ports have been rebuilt. Poudriere uses jails that are separated from the live system for building the packages and therefor can not disrupt the running services on the host.
  • There might not always be an easy update path when an important port gets updated because of complicated dependencies between the updated port and its dependents. Usually in such cases there are instructions in the /usr/ports/UPDATING file to remedy the situation but to apply the instructions manual intervention is needed. Poudriere nullifies the need for manual intervention and the need to follow the instructions in first place because the repository gets rebuilt starting from the lowest level of dependencies and all dependents of the changed port/ports that need rebuilding are properly rebuilt resulting in a consistent repository that will result in a consistent system when the updated packages get installed.
  • When building individual ports one by one with # make install or for example ports-mgmt/portmaster there are no opportunities to make use of the conflict resolution feature of the new pkg system. It is only available when installing sets of packages from a remote repository with pkg-install(8) or pkg-upgrade(8).
  • Building for a different target system that uses different defaults and options can be next to impossible because of conflicts between already installed ports and the build time dependencies the differing options would pull in to be installed at the same time
  • Build once, distribute to many client hosts is cumbersome to implement with only the basic ports system.

What poudriere is not:
  • It's not a replacement for any of the existing tools in the FreeBSD base or the ports(7) system. It doesn't modify the standard ports infrastructure in any way to achieve what it does.
  • It's not in competition with the few other ports building tools like ports-mgmt/portmaster or ports-mgmt/portupgrade because its goals are different. The two mentioned tools are nice frontends to the ports system but are limited to building the ports on the live system just as the basic ports system is.
  • It can't be used as a generic jail(8) management system like sysutils/ezjail. The jails created by poudriere are only suitable for building ports into binary packages.

Requirements:
  • Any supported version of FreeBSD installed on the build host system, amd64 or i386. Amd64 very much recommended over i386. The so called stable versions stable/10 and stable/11 are supported. The head branch a.k.a 12-CURRENT should also work also but see here first. Operating system versions installed in the jails used to build packages should not be more recent than the operating system installed on the build host system, jails that have older operating system versions work as long as the kernel on the build host has the proper compatibility options set.
  • 64- and 32-bit versions can be mixed but only in a way where the build host system is a 64-bit version and the jail has a 32-bit version of FreeBSD. The kernel on the build host must include the 32-bit compatibility layer for this to work.
  • An up to date version of ports-mgmt/poudriere, latest version is 3.2.7 at the time of the last edit of this HOWTO. The development version ports-mgmt/poudriere-devel can be also used if bleeding edge features are needed.
  • An up to date version of ports-mgmt/pkg, latest version is 1.10.5 at the time of the last edit.
  • Enough memory to build the largest packages. This depends on what you're building, things like www/firefox may take lots of memory. 4GBs or more to be safe (amd64 version of FreeBSD needed if you intend to utilize more that 4GBs of memory).
  • Enough free disk space. As above, depends on what you're building. 10GBs or more to be safe.
  • Support for the ZFS filesystem on the build host. ZFS makes jail cloning very fast and efficient and in general should be used if possible. I will note below the settings that need to be changed in case you don't want to use ZFS for some reason.
  • The kernel on the build host must have support for jail(8) and nullfs(5).

Before proceeding take note:
  • All of this assumes that you have a working internet connection right from the start.
  • Commands listed as # command should be run with root privileges. Other commands work without the root privileges.
1. Install ports-mgmt/pkg.

This step is only needed on a freshly installed system that has no ports-mgmt/pkg installed yet.

On any supported version of FreeBSD there is bootstrap utility for pkg installed as /usr/sbin/pkg that can be used to install ports-mgmt/pkg. Simply run this on a freshly installed system and you'll be prompted if you want to install pkg(8):

# pkg


2. Install ports-mgmt/poudriere.

Install poudriere from the official binary packages:

# pkg install -y poudriere

If you need any of the so called bleeding edge features of poudriere, install the ports-mgmt/poudriere-devel port instead.

3. Configure ports-mgmt/poudriere.

First copy the sample configuration file ${LOCALBASE}/etc/poudriere.conf.sample to ${LOCALBASE}/etc/poudriere.conf and make sure the copy is writable:

# cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf
# chmod 640 /usr/local/etc/poudriere.conf

Edit the following entries in ${LOCALBASE}/etc/poudriere.conf (some already have the desired values, leave them untouched):

Code: 
#### ZFS
# The pool where poudriere will create all the filesystems it needs
# poudriere will use ${ZPOOL}/${ZROOTFS} as its root
#
# You need at least 7GB of free space in this pool to have a working
# poudriere.
#
ZPOOL=zroot

### NO ZFS
# To not use ZFS, define NO_ZFS=yes
#NO_ZFS=yes

# root of the poudriere zfs filesystem, by default /poudriere
ZROOTFS=/poudriere

# the host where to download sets for the jails setup
# You can specify here a host or an IP
# replace _PROTO_ by http or ftp
# replace _CHANGE_THIS_ by the hostname of the mirrors where you want to fetch
# by default: ftp://ftp.freebsd.org
#
# Also note that every protocols supported by fetch(1) are supported here, even
# file:///
# Suggested: https://download.FreeBSD.org
FREEBSD_HOST=https://download.FreeBSD.org

# The directory where poudriere will store jails and ports
BASEFS=/usr/local/poudriere

# If set the given directory will be used for the distfiles
# This allows to share the distfiles between jails and ports tree
# If this is "no", poudriere must be supplied a ports tree that already has
# the required distfiles.
DISTFILES_CACHE=/var/cache/distfiles

# If set the ports tree or source tree marked to use svn will use the defined
# mirror (default: svn.FreeBSD.org)
# The SSL fingerprints are published here:
# https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html#svn-mirrors
SVN_HOST=svn.FreeBSD.org

# Disable linux support
NOLINUX=yes

Create the /var/cache/distfiles directory, you can put this somewhere else as well but this path is the most compliant one with hier(7):

# mkdir /var/cache/distiles

The comments in the above are verbatim from the sample configuration file, I have chosen to disable Linux support because I don't build anything that requires the Linux emulation layer. There are a number of different configuration options that I'm leaving uncovered here because the defaults are fine for most cases, refer to the configuration file and the documentation for more information.

If you want to disable the use of ZFS leave the ZPOOL and ZROOTFS lines commented out and uncomment the NO_ZFS=yes line. None of the steps below need to be modified if ZFS is not used, poudriere will use alternate methods for jail creation/cloning and handling of ports trees and package repositories that are somewhat slower than with ZFS but the end results are the same.


4. Set up the ports tree for poudriere.

Setting up and managing of ports(7) trees is done with the poudriere-ports(8) command.

This will create a new ports tree located at ${BASEFS}/ports/default using the SVN method of checking out and updating of the ports tree:

# poudriere ports -c -m svn

After this running poudriere ports -l should output:

Code: 
PORTSTREE METHOD TIMESTAMP           PATH
default   svn    2018-08-16 14:29:52 /usr/local/poudriere/ports/default

It's now possible to use poudriere-ports(8) to update the ports tree:

# poudriere ports -u

It's also possible to register an existing ports tree into poudriere, however this will leave the updating of this ports tree for you to handle:

# poudriere ports -c -f none -m null -M /usr/ports -p test


5. Set up the jail for building packages.

The command to set up and manage jails in poudriere is poudriere-jail(8). The simplest way to create a jail is to use the http method (note: the manual page will claim that http is the default but the default is in fact ftp) to fetch a FreeBSD distribution file set to be extracted in the new build jail. The http method will use the FREEBSD_HOST setting from the configuration file as the mirror site for downloading the necessary files. HTTPS is supported.

This is how an amd64 11.2-RELEASE jail would be created on an amd64 system, substitute amd64 with i386 if creating an i386 jail on an i386 build host:

# poudriere jail -c -m http -v 11.2-RELEASE -j releng112amd64

The jail identifier for the -j option is for you to decide, I used "releng112amd64" to signify that it's 11.2-RELEASE but can be updated with release engineering updates and security fixes. In other words the jail follows the official 11.2-RELEASE branch with updates.

After creation the jail should be in the poudriere jail -l listing:
Code: 
JAILNAME       VERSION                 ARCH  METHOD       TIMESTAMP           PATH
releng112amd64 11.2-RELEASE-p1        amd64 http         2018-08-16 18:21:38 /usr/local/poudriere/jails/releng112amd64

Creating an i386 jail on an amd64 build host is done as follows:

# poudriere jail -c -m http -a i386 -v 11.2-RELEASE -j releng112i386

If you're tracking a stable or current branch of FreeBSD the easiest way to keep your poudriere jails up to date is to use the -m src=/usr/src method for creating and updating the jails. This is how a stable/11 amd64 jail would be created, it is assumed that make buildworld buildkernel has been done in /usr/src and the build host has been updated to use the new version of the OS with make installkernel installworld followed by a reboot.

# poudriere jail -c -m src=/usr/src -v stable/11 -j stable11amd64

It is also possible to use the -m svn method that fetches the sources for the requested version of FreeBSD, builds a new world using those sources and installs the results of the build as the new jail.

Jails that were installed using the -m ftp or -m http method are updated using freebsd-update(8). Jails that were installed with the -m src=/usr/src method are updated using the sources and objects in /usr/src and /usr/obj of the build host. Jails that were installed using the -m svn method that involves building a world, a new world build is performed with the updated sources.
Updating of a jail regardless of the method of their creation is done with:

# poudriere jail -u -j releng112amd64


6. Create a list of ports to be built into packages.

Create a new file, for example /usr/local/etc/ports.txt (can be also located at your home directory just as well) and put all the ports you want to build in there.

Example:

Code: 
ports-mgmt/pkg
devel/git
devel/subversion
editors/vim
www/apache22

There is no need to put the dependencies of the built ports in the list, they are built automatically.

Note that you should include the ports-mgmt/pkg port in the list because there are no ports that depend on it. If you don't include it in your list of ports to be built it won't be updated when there's a new version of it.

IMPORTANT: In order to be sure that your repository stays in a consistent state, always try to run poudriere(8) with this full list of ports to build. If you build only a port here, port there, only parts of the repository are updated and many ports that should have been recompiled are not recompiled and that can lead to shared library mismatches.
 
Last edited:
7. Customize the build settings.

Poudriere can use a global make.conf or a jail specific ${jailname}-make.conf to customize build settings. These files go into ${LOCALBASE}/etc/poudriere.d/ directory. Note that the build host's own /etc/make.conf is not used at all because it's assumed to be "tainted" by build host specific settings.

Setting options for ports can be done with the global or jail specific make.conf and can be easier than using the saved options that are shown in the next section.

Here's an example what could go into the global ${LOCALBASE}/etc/poudriere.d/make.conf:

Code: 
# No CUPS DBUS or X11. These are unset globally for every port.
OPTIONS_UNSET= CUPS DBUS X11

# Force default postgresql version
WITH_PGSQL_VER=92

# Use the port version OpenSSL where possible, note the previous way of
# using WITH_OPENSSL_PORT=yes is no longer recommended.
DEFAULT_VERSIONS+= ssl=openssl


# Examples of port specific options. Note that the prefix to be used is the
# $OPTIONS_NAME for a port. In the example below for instance 'security_pinentry' is the
# $OPTIONS_NAME for security/pinentry.

# I want ZSH completions for ports-mgmt/poudriere-devel
ports-mgmt_poudriere-devel_SET= ZSH

# security/pinentry. Sets NCURSES option, unsets the other options.
security_pinentry_SET= NCURSES
security_pinentry_UNSET= GTK GTK2 QT3 QT4

# ftp/curl. Avoid linking to base system OpenSSL libraries.
ftp_curl_SET= GSSAPI_NONE
ftp_curl_UNSET= GSSAPI_BASE GSSAPI_HEIMDAL GSSAPI_MIT

Note that options set in this way are not saved under /var/db/ports or anywhere else. Refer to the next section for saved options for ports.

An example for a jail specific ${LOCALBASE}/etc/poudriere.d/release93amd64-make.conf file if the jail is a 9.3-RELEASE amd64 jail. This sets clang(1) as the default compiler and CPUTYPE for Intel's core duo CPUs.

Code: 
CPUTYPE?= core2
CC=clang
CXX=clang++
CPP=clang-cpp

Note that the above example is for an outdated and unsupported release of FreeBSD that didn't yet default to clang(1) as the C compiler. None of these settings are needed on currently supported releases.


8. Setting saved options for the ports to be built

In addition to specifying the options in the make.conf or in the jail specific ${jailname}-make.conf the options can be set interactively like you would with make config for a port. Setting options for the ports is done with the poudriere-options(8) command and the options are saved at ${LOCALBASE}/etc/poudriere.d/options/${OPTIONS_NAME}/options. You can make use of an existing /var/db/ports hierarchy by copying the contents of it to the ${LOCALBASE}/etc/poudriere.d/options directory. Note that poudriere-options(8) uses the config-conditional target by default that only sets options for ports that have no options set yet. Use # poudriere options -c category/port to force overwriting of saved options. Examples:

# poudriere options www/apache22

# poudriere options -c lang/python27

It is also possible to use a jail specific options directory. If you use the -j jailname option as part of the poudriere options command line the options will be set in ${LOCALBASE}/etc/poudriere.d/${jailname}-options directory and those options will override the default options for packages built using that jail.


9. Building packages

Package building is done with the poudriere-bulk(8) command. Putting together everything from the previous steps, building is done as follows:

# poudriere bulk -f /usr/local/etc/ports.txt -j releng112amd64

Built packages are stored at the ${BASEFS}/data/packages/releng12amd64-default directory. The -default ending signifies that the packages were built using the "default" ports tree.

Build logs are stored at the ${BASEFS}/data/logs directory in HTML format and can be viewed with a locally running web browser or published with a web server such as www/nginx.

It is possible to force rebuilding of every package in the repository by using the -c option with poudriere-bulk(8):

# poudriere bulk -c -f /usr/local/etc/ports.txt -j releng112amd64.

Using the -C option instead of -c rebuilds only the ports listed in the file specified by the -f option or given on the command line. For example this would forcibly rebuild the www/apache22 package and no other package:

# poudriere bulk -C -j releng112amd64 www/apache22


10. Using the package repository with pkg(8)

The packages can be installed from the local ${BASEFS}/data/packages directory with pkg-add(8).

# pkg add /usr/local/poudriere/data/packages/releng112amd64-default/All/apache22.txz

A better way to use the packages is to use the pkg remote repository feature (see pkg-repository(5)) by creating a repository configuration file /usr/local/etc/pkg/repos/myrepo.conf and specifying the repository address as a file:// URL (to use the local file system) in this configuration file.

Code: 
myrepo: {
    url             : "file:///usr/local/poudriere/data/packages/releng112amd64-default",
    enabled         : yes,
    mirror_type     : NONE
}

Details of the syntax of this file can found in the pkg.conf(5) manual page.

Note that the repository URL does not end in /Latest like it used to be with the old pkg_* tools.

Packages can be now installed with pkg-install(8):

# pkg install apache22

Updating the system when updated packages have been built is done with pkg-upgrade(8):

# pkg upgrade

Note that # pkg-upgrade command updates the repository metadata automatically before any other operations, there is no need to run pkg-update(8) first (unless the automatic repository update has been turned off).

Note that you probably want to disable the use of the official package repository by creating a /usr/local/etc/pkg/repos/FreeBSD.conf file with these contents:

Code: 
FreeBSD: {
  enabled: no
}


Sharing the repository to other machines can be done with a web server, FTP server, NFS etc. The only requirement is that the repository files and directories are fetch(1)able.

For example this is how the repository would be shared using www/apache22. Rough guideline what to put in the httpd.conf file:

Code: 
...
<Directory "/usr/local/poudriere/data/packages">
    Options Indexes FollowSymLinks

    AllowOverride None

    Order allow,deny
    Allow from all
</Directory>
...

<IfModule alias_module>
Alias /packages /usr/local/poudriere/data/packages
</IfModule>

With those settings a client machine can access the package repository with this repository configuration (save this as /usr/local/etc/pkg/repos/myrepo.conf):

Code: 
myrepo: {
  url: "http://webserver.mydomain.tld/packages/releng112amd64-default"
  enabled : yes,
  mirror_type : NONE
}


11. RSA signing key for cryptographic verification of the repository. Optional.

This information is largely from Glen Barber's page https://glenbarber.us/2012/06/11/Maintaining-Your-Own-pkgng-Repository.html

# mkdir -p /usr/local/etc/ssl/keys
# chmod 600 /usr/local/etc/ssl/keys
# mkdir -p /usr/local/etc/ssl/certs
# openssl genrsa -out /usr/local/etc/ssl/keys/pkg.key 4096
# openssl rsa -in /usr/local/etc/ssl/keys/pkg.key -pubout > /usr/local/etc/ssl/certs/pkg.cert

Add this line to /usr/local/etc/poudriere.conf:

Code: 
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/pkg.key

On any system that uses the packages you'll have to copy the /usr/local/etc/ssl/certs/pkg.cert file over and put it in the same place (DO NOT COPY THE pkg.key FILE ANYWHERE!) and add these lines to the /usr/local/etc/pkg/repos/myrepo.conf:

Code: 
signature_type  : "PUBKEY",
pubkey: "/usr/local/etc/ssl/certs/pkg.cert",

Now the downloaded packages are verified that they have the correct cryptographic signatures before installation.

There is also a different method for signing the packages that uses an external signing command that can be used to keep the private key separate from the package build host. Refer to pkg-repo(8) for details.
 
Last edited:
Set this up today. I used http://www.allbsd.org for a specific version, ie
# poudriere jails -c -j 9-STABLE -f none -M /export/jails/9-STABLE -m allbsd -v r248915 -a amd64
I set things up with the
Code: 
USE_TMPFS=all
option in /usr/local/etc/poudriere.conf. It really sped things up :e
 
Yes, it does, except the jail setup doesn't get much faster unfortunately, that's the penalty you pay for not using ZFS. I tested
Code: 
USE_TMPFS=all
on a machine with 4 GB of RAM and it didn't seem to choke on editors/vim.
 
I have downgraded my only remaining FreeBSD system (a firewall) to an i386 machine with only 1 GB or RAM. I can report that I can still use ports-mgmt/poudriere-devel for building packages as I did on the amd64 system that had four times the memory and somewhat faster CPU.
 
Works like a charm. Thank you very much.

I did hit one little snag when using PKGNG, it doesn't seem to create the needed repo.txz and you need to run # pkg repo /usr/local/poudriere/data/packages/release90amd64-default by hand. Can't this be automatically generated after the build run? Or did I miss something?
 
It should be generated automatically but here are some very recent changes to the repository format and there might be bugs in the intergration with poudriere. I think the repo.txz file is now deprecated and the repository metadata is now split into multiple files. I don't have my poudriere build system set up atm so I can't check.

Edit: I found this bit in the ports-mgmt/pkg commit log:

http://www.freshports.org/commit.php?message_id=201306240602.r5O62L91054147@svn.freebsd.org

Code: 
- pkg repo produce a repo.txz in pkg 1.0 (legacy) fromat

So it looks like ports-mgmt/poudriere-devel has been changed to produce the newer type of repository metadata that does not include the repo.txz file anymore.
 
kpa said:
It should be generated automatically but here are some very recent changes to the repository format and there might be bugs in the intergration with poudriere.
Click to expand...
Ah, it's not really an issue to do it by hand but it would be nice to automate.

I think the repo.txz file is now deprecated and the repository metadata is now split into multiple files. I don't have my poudriere build system set up atm so I can't check.
Click to expand...
Right! I noticed that too, but it seems to be a bit wonky at the moment. Sometimes it creates the 'old' repo.txz and at other times I get the new files. Maybe it's PKGNG that's not working correctly. I'm sure it'll be fixed soon, it's updated quite frequently.
 
Is anyone using the "sets" feature with poudriere, and if so, how are you dealing with maintaining repos across boxes that have different requirements? Let's take the example of three poudriere buildlists:

  • packages common to ALL hosts (common)
    • rsync
    • sshguard
    • vim
    • bash
    • apache22
    • php5
  • host group A (group-a)
    • perl 5.16
    • mysql51
    • postfix
  • host group B (group-b)
    • mysql50
    • perl 5.14
    • php53

Using the "sets" feature (the -z option to the bulk and options subcommands) would allow for each group specified above to have a custom set of options. The sets however also leave you with a package repo for each set, and a duplication of the common ports amongst all sets.

For example, building the three sets above:

Code: 
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common -f /usr/local/etc/poudriere.d/buildlists/group-a
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common -f /usr/local/etc/poudriere.d/buildlists/group-b

And then you would have three repos:

/poudriere_data/packages/common
/poudriere_data/packages/group-a
/poudriere_data/packages/group-b

And all packages in the "common" buildlist would be built three times and the resulting packages would also exist in all three repos. You would also need to define a different repo in each group of hosts in pkg.conf. I'm fine with that, but I wonder if there's a more clever way to do this without using sets to avoid the duplicate builds (both saving the time it takes to build a package multiple times and saving disk space).

Perhaps sets aren't needed and this can all be done with only a supplemental buildlist for each group of hosts? I'm not sure what poudriere does if you try to build packages that conflict with each other in the same poudriere bulk run (in the example above, MySQL, PHP, and Perl have conflicting versions).

Anyone doing something similar?

TL;DR: What's the intended use-case for poudriere sets?
 
I haven't had time to look into sets yet and I don't really need them because at the moment I'm building the packages for just one system. Feel free to post your findings on this thread.
 
@kpa, thank you for the great post!

I'm currently switching to ports-mgmt/poudriere for building packages for desktop, but I have a question. Can I use it for incremental updates of my packages without rebuilding them all every time? If yes, what should I do in case if there is a port that have libraries version bumped, so everything dependent on it should be rebuilt?
 
Last edited by a moderator:
hedgehog said:
Can I use it for incremental updates of my packages without rebuilding them all? If yes, what should I do in case if there is a port that have libraries version bumped, so everything dependent on it should be rebuilt?
Click to expand...
Yes, that's possible. And you don't really need to do anything, poudriere already does it by default. If you want to make 100% sure your dependencies are correctly built you can always add the -c switch to the bulk command, that removes all previously built packages and starts fresh. If you don't use -c it will do an 'incremental', updating only the packages that are newer. It will also take care of any dependent packages.
 
SirDice said:
Yes, that's possible. And you don't really need to do anything, poudriere already does it by default. If you want to make 100% sure your dependencies are correctly built you can always add the -c switch to the bulk command, that removes all previously built packages and starts fresh. If you don't use -c it will do an 'incremental', updating only the packages that are newer. It will also take care of any dependent packages.
Click to expand...

Does that mean that, for instance, if the devel/gettext has it's version bumped, all the ports depending on devel/gettext will get rebuilt automatically, even if their version remained the same? Do you happen to know how is that achieved?
 
The dependency solver takes care of that and my experience with poudriere is that it works correctly in almost all cases. I yet have to see a case where a version bump failed to rebuild a dependency correctly. A package is also considered to be changed if it's options have changed and you have enabled the options checking in poudriere.conf.

The PKGNG pkg(8) client also tracks the shared library versions and considers a package changed if there are changes in the shared library dependencies. This should catch ports that didn't get their port versions bumped but have a dependency to a shared library that got a version bump.
 
Thank you for answers.

Sorry for bombarding you with questions. Do I still need to manually request of reinstalling all packages that depends on devel/gettext in that case on the target system? Like that:
# pkg install -R gettext

Will the ports-mgmt/poudriere take care of dependencies that have they origins changed as well? For instance:
Code: 
20121002:
  AFFECTS: users of devel/clanlib
  AUTHOR: jhale@FreeBSD.org

  devel/clanlib has been updated to 2.3.6 and the legacy version has been
  moved to devel/clanlib1 and updated to 1.0.0.  Although both versions
  may be installed at the same time, you should update your port origins
  first since many ports that depend on clanlib will not work with the
  new version.

  Update your port origins as follows:

  # portmaster -o devel/clanlib1 devel/clanlib
  or
  # portupgrade -fo devel/clanlib1 devel/clanlib
  or
  # pkg set -o devel/clanlib:devel/clanlib1

I use Kubuntu at work and the only repository advantage I see there is that you don't have to care about dependencies at all. You just issue apt-get upgrade command and it takes care of everything. I think it would be great if you could add some metadata for packages in repository that will, for example, ask a user if he wants to automatically reinstall all the packages that depends on the specific package while updating it to current version. Same with changing ports origins.
 
Changed port origins are not a problem as long as the ports that you want get actually built by a build run. If the changed port is a dependency of a port in your list of ports that pass to poudriere then there's no problem, the whole chain of dependencies gets rebuilt if needed. If the port is a "leaf port" (that no other port depends on) that is directly listed in your list then you have to rename the port in the list. (I have to check if MOVED is consulted in the latter case, it would be really great if it was).
 
hedgehog said:
Do I still need to manually request of reinstalling all packages that depends on devel/gettext in that case on the target system? Like that:
# pkg install -R gettext
Click to expand...
Just run # pkg upgrade. It will correctly figure out which packages to update and/or replace. I haven't had any major issues doing this.
 
Great, thank you guys. Looks like it always rebuilds all dependencies automatically and updates the repository, so # pkg upgrade will know that the packages were rebuilt.

Quite a smart solution to avoid conflicts and make it even more automated. The only drawback is the longer time required for incremental builds, but that's the cost of stability.
 
The non-development version of ports-mgmt/poudriere seems have support for the NO_ZFS option starting from version 3.0.1 or 3.0.2. I have tested building a few packages with ports-mgmt/poudriere-devel replaced with ports-mgmt/poudriere without changing anything in my configuration files and everything seems to work. I'll add a notice to his HOWTO as soon as I'm satisfied that the non-development version can be used without issues.

Edit: Changes committed. I have built enough packages now to be confident that the non-development version works as well as the development version.
 
rusty said:
I set things up with the
Code: 
USE_TMPFS=all
option in /usr/local/etc/poudriere.conf. It really sped things up :e
Click to expand...

It gives insane speed boost because ports-mgmt/poudriere uses disk a lot when installing dependencies. Before having everything in tmpfs, I saw build-depends status all the time and it required more time than actually building ports.

Now I build everything in tmpfs and it's really fast. Also, I noticed that I usually have 7-10GB RAM free, out of 16GB, when using two builders, so probably it would be a good idea to increase builders number to 3.
 
A new configuration option was added a while ago to automatically detect dependency changes.

This is the full description from poudriere.conf.sample:

Code: 
# Automatic Dependency change detection
# When bulk building packages, compare the dependencies from kept packages to
# the current dependencies for every port. If they differ, the existing package
# will be deleted and the port will be rebuilt. This helps catch changes such
# as DEFAULT_RUBY_VERSION, PERL_VERSION, WITHOUT_X11 that change dependencies
# for many ports.
# Valid options: yes, no
#CHECK_CHANGED_DEPS=yes
 
You must log in or register to reply here.
Back
Top