• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Pkg package repository using ports-mgmt/poudriere. No ZFS required.

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#1
This is a HOWTO on how to build your own pkg packages in a jail(8) using ports-mgmt/poudriere without having to use ZFS.

What is ports-mgmt/poudriere? The port description reads as follows:

Code:
poudriere is a tool primarily designed to test package production on
FreeBSD. However, most people will find it useful to bulk build ports
for FreeBSD.

WWW: https://github.com/freebsd/poudriere/wiki
This description is now quite out of date since the official FreeBSD binary packages are now built with it. Poudriere is a package building system intended for anyone who wants to build their own binary pkg packages for FreeBSD. Its design is as follows:

  • Mostly /bin/sh scripts for portability.
  • Uses jail(8)s for creating clean build environments for ports building.
  • Makes heavy use of nullfs(5) to avoid unnecessary copying.
  • Has built-in utilities for setting up and updating ports(7) trees and jail(8)s.
  • Customizable trough a configuration file (${PREFIX}/etc/poudriere.conf, ${PREFIX} defaults to /usr/local) and a dedicated settings directory (${PREFIX}/etc/poudriere.d).
  • Can be run as a daemon that serves build requests.


Why is ports-mgmt/poudriere needed in the first place? The ports(7) infrastructure suffers from numerous design faults that are not easy to overcome without a dedicated package building tool like ports-mgmt/poudriere. These faults include:

  • Building of ports as outlined in ports(7) happens on the live system. Most often many build failures stem from this when an already installed port or leftover files from an old version causes conflicts with the to be built new version of the port. Poudriere avoids this problem by starting every build from a clean state as if the ports were installed for the first time on the system.
  • Building ports on the live system pulls in a large number of build time dependencies that are not required to be installed after the builds have finished. These build time dependencies can cause configuration issues and trigger build failures when port options are changed.
  • Building ports on the live system may cause service interruptions while the ports are being built and the situation may not be corrected until all of the ports have been rebuilt. Poudriere uses jails for building the packages and therefor can not disrupt the running services on the host.
  • The ports(7) infrastructure alone can not guarantee that the set of installed ports stays consistent with dependencies. For example, it is not possible to guarantee that all installed ports use a certain version of the PHP port. The conflicts mechanism will prevent conflicting ports to be installed at the same time but it doesn't offer any way to correct the problems. Poudriere tries to keep a consistent repository where every built package has satisfiable dependencies.
  • The ports system can not answer rather simple sounding questions like what would happen to the installed packages if one of the packages is updated to a new version with different set of dependencies than what the existing one has.
  • When building individual ports one by one with make install or ports-mgmt/portmaster there are no opportunities to make use of the conflict resolution feature of the new pkg system. It is only available when installing sets of packages from a remote repository with pkg-install(8) or pkg-upgrade(8).
  • There is an unmentioned assumption to using ports and packages. This is the requirement that all installed ports come from a ports tree that is in the same state meaning that ports that have changed since the last update are rebuilt consistently and also the ports that haven't changed are not rebuilt or replaced with different versions by any means. Mixing the official packages and self-built ports breaks this assumption very easily leading to dependency problems.
  • Building for a different target system that uses different defaults and options can be next to impossible because of conflicts between already installed ports and the build time dependencies the differing options would pull in to be installed at the same time
  • Build once, distribute to many client hosts is cumbersome to implement with only the basic ports(7) system.

What poudriere is not:
  • It's not a replacement for any of the existing tools in the FreeBSD base or the ports(7) system. It doesn't modify the standard ports(7) infrastructure in any way to achieve what it does.
  • It's not in competition with the few other ports building tools like ports-mgmt/portmaster or ports-mgmt/portupgrade because its goals are different. The two mentioned tools are nice frontends to the ports(7) system but are limited to building the ports on the live system just as the basic ports system is.
  • It can not be used as a generic jail(8) management system like sysutils/ezjail. The jails created by poudriere are only suitable for building ports into binary packages with it.


Requirements:

  • Any supported version of FreeBSD installed on the build host system, amd64 or i386. So called stable versions stable/9, stable/10 and stable/11 are well supported. The head branch a.k.a 12-CURRENT should also work also but see here first. Operating system versions installed in the jails used to build packages should not be more recent than the operating system installed on the build host system, jails that have older operating system versions work as long as the kernel on the build host has the proper compatibility options set.
  • 64- and 32-bit versions can be mixed but only in a way where the build host system is a 64-bit version and the jail has a 32-bit version of FreeBSD.
  • ports-mgmt/poudriere version 3.1.4+ (latest version is 3.1.14 at the time of the last edit). The development version ports-mgmt/poudriere-devel can be also used if bleeding edge features are needed.
  • ports-mgmt/pkg version 1.9+ (latest version is 1.9.3 at the time of the last edit).
  • Enough memory to build the largest packages. This depends on what you're building, things like www/firefox may take lots of memory. 4GBs or more to be safe (amd64 recommended for building large ports).
  • Enough free disk space. As above, depends on what you're building. 10GBs or more to be safe.
  • ZFS is not needed. Note: If your system has a ZFS pool you're better off using it by not using the NO_ZFS setting in the configuration. ZFS makes jail cloning very fast and efficient but I chose not to use ZFS on my system and that's why this HOWTO is written for non-ZFS systems.
  • The kernel on the build host must have support for jail(8) and nullfs(5).

Before proceeding take note:



1. Install ports-mgmt/pkg.

This step is only needed on a freshly installed system that has no ports-mgmt/pkg installed yet.

make -C /usr/ports/ports-mgmt/pkg install clean

On FreeBSD 9.3/10.0 or later there is bootstrap utility for pkg installed as /usr/sbin/pkg that can be used to install ports-mgmt/pkg without having to use the ports tree. Simply run pkg as root on a freshly installed system you'll be prompted if you want to install pkg(8).


2. Install ports-mgmt/poudriere.

make -C /usr/ports/ports-mgmt/poudriere install clean

If you need any of the so called bleeding edge features of poudriere, install the ports-mgmt/poudriere-devel port instead.




3. Configure ports-mgmt/poudriere.

First copy the sample configuration file /usr/local/etc/poudriere.conf.sample to /usr/local/etc/poudriere.conf and make sure the copy is writable:

cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf
chmod 640 /usr/local/etc/poudriere.conf

Edit the following entries in /usr/local/etc/poudriere.conf:

Code:
# Comment out ZPOOL
# ZPOOL=tank

# Add NO_ZFS=yes
NO_ZFS=yes

# Edit suitable FREEBSD_HOST
FREEBSD_HOST=ftp://ftp.freebsd.org

...

# Use devel/ccache to speed up building, optional.
CCACHE_DIR=/var/cache/ccache

# I don't build any Linux packages so I'm disabling support for building them,
# optional
NOLINUX=yes


4. Set up the ports tree for poudriere.

This will set up an existing ports(7) tree located at /usr/ports as the default ports tree:

poudriere ports -c -F -f none -M /usr/ports -p default

After this running poudriere ports -l should output:
Code:
PORTSTREE            METHOD     PATH
default              portsnap        /usr/ports
It's now possible to use poudriere(8) to update the ports tree:
poudriere ports -u

You can also let poudriere(8) to create the ports tree at the default location of /usr/local/poudriere/ports/default with (using the SVN method, substitute -m portsnap for portsnap(8)):

poudriere ports -c -m svn

This way the ports tree used for building the ports is completely separated from the build host and its /usr/ports tree. The build host can even have an empty /usr/ports turning it into a client that can not build ports itself but only uses the packages built by poudriere(8).


5. Set up the jail for building packages.

This will set up a 9.3-RELEASE amd64 jail for package building. Substitute i386 for amd64 if on i386.

The simplest way is to use the -m ftp (use -m http in case the ftp method fails) method to fetch a FreeBSD distribution to be installed in the build jail. All the hard work is done by the poudriere jail command. This is how an amd64 9.3-RELEASE jail would be created on an amd64 system:

poudriere jail -c -m ftp -v 9.3-RELEASE -f none -j release93amd64

The name after the -j flag is for you to decide, I used "release93amd64" to identify it here.

After creation the jail should be in the poudriere jail -l listing:
Code:
JAILNAME             VERSION              ARCH    METHOD
release93amd64       9.3-RELEASE          amd64   ftp
Creating an i386 jail on amd64 is done as follows:

poudriere jail -c -a i386 -m ftp -v 9.3-RELEASE -f none -j release93i386


If you're tracking a stable or current branch of FreeBSD the easiest way to keep your poudriere jails up to date is to use the -m src=/usr/src method for creating and updating the jails. This is how a stable/10 amd64 would be created, it is assumed that make buildworld buildkernel has been done in /usr/src and the build host has been updated to use the new version of the OS with make installkernel installworld followed by a reboot.

poudriere jail -c -j stable10amd64 -m src=/usr/src -v stable/10

It is also possible to use the -m svn method that fetches the sources for the requested version of FreeBSD, builds a new world and kernel using those sources and installs the results of the build as the new jail.

Jails that were installed using the -m ftp or -m http method are updated using freebsd-update(8). Jails that were installed with the -m src=/usr/src method are updated using the sources and objects in /usr/src and /usr/obj of the build host. Jails that were installed using the -m svn method that involves building a world, a new world build is performed with the updated sources. Updating of a jail is done with:

poudriere jail -u -j release93amd64


6. Create a list of ports to be built into packages.

Create a new file, for example /usr/local/etc/ports.txt and put all the ports you want to build in there.

Example:

Code:
ports-mgmt/pkg
devel/git
devel/subversion
editors/vim
www/apache22
There is no need to put the dependencies of the built ports in the list, they are built automatically.

Note that you should include the ports-mgmt/pkg in the list because there are no ports that depend on it. If you don't include it in your list of ports to be built it won't be updated when there's a new version of it.

IMPORTANT: In order to be sure that your repository stays in a consistent state, always run poudriere with this full list of ports to build. If you build only a port here, port there, only parts of the repository are updated and many ports that should have been recompiled are not recompiled and that can lead to shared library mismatches.
 
Last edited:

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#2
7. Customize the build settings.

Poudriere can use a global make.conf or a jail specific jailname-make.conf to customize build settings. These files go into /usr/local/etc/poudriere.d/ directory.

Setting options for ports can be done with the global or jail specific make.conf and can be easier than using the saved options that are shown in the next section.

Here's an example what could go into the global /usr/local/etc/poudriere.d/make.conf:

Code:
# No CUPS DBUS or X11. These are unset globally for every port.
OPTIONS_UNSET= CUPS DBUS X11

# Force default postgresql version
WITH_PGSQL_VER=92

# Use the port version OpenSSL where possible, note the previous way of
# using WITH_OPENSSL_PORT=yes is no longer recommended.
DEFAULT_VERSIONS+= ssl=openssl


# Examples of port specific options. Note that the prefix to be used is the
# $OPTIONS_NAME for a port. In the example below for instance 'security_pinentry' is the
# $OPTIONS_NAME for security/pinentry.

# I want ZSH completions for ports-mgmt/poudriere-devel
ports-mgmt_poudriere-devel_SET= ZSH

# security/pinentry. Sets NCURSES option, unsets the other options.
security_pinentry_SET= NCURSES
security_pinentry_UNSET= GTK GTK2 QT3 QT4

# ftp/curl. Avoid linking to base system OpenSSL libraries.
ftp_curl_SET= GSSAPI_NONE
ftp_curl_UNSET= GSSAPI_BASE GSSAPI_HEIMDAL GSSAPI_MIT
Note that options set in this way are not saved. Refer to the next section for saved options for ports.

An example for the jail specific /usr/local/etc/poudriere.d/release93amd64-make.conf. This sets clang(1) as the default compiler and CPUTYPE for Intel's core duo CPUs.

Code:
CPUTYPE?= core2
CC=clang
CXX=clang++
CPP=clang-cpp


8. Setting saved options for the ports to be built

In addition to specifying the options in the make.conf or in the jail specific jailname-make.conf the options can be set interactively. Setting options for the ports is done with the poudriere options command and the options are saved at /usr/local/etc/poudriere.d/options/${OPTIONS_NAME}/options. You can make use of an existing /var/db/ports hierarchy by copying the contents of it to /usr/local/etc/poudriere.d/options. Note that poudriere options uses the config-conditional target by default that only sets options for ports that have no options set yet. Use poudriere options -c category/port to force overwriting of saved options. Examples:

poudriere options www/apache22

poudriere options -c lang/python27

It is also possible to use a jail specific options directory. If you use the -j jailname option as part of the poudriere options command line the options will be set in /usr/local/etc/poudriere.d/${jailname}-options directory and those options will override the default options for packages built using that jail.



9. Building packages

Package building is done with the poudriere bulk command. Putting together everything from the previous steps, building is done as follows:

poudriere bulk -f /usr/local/etc/ports.txt -j release93amd64

Built packages are stored at the /usr/local/poudriere/data/packages/release93amd64-default directory. The -default ending signifies that the packages were built using the "default" ports tree.

Build logs are stored at the /usr/local/poudriere/data/logs directory in HTML format and can be viewed with a locally running web browser or published with a web server such as www/nginx.

It is possible to force rebuilding of every package in the repository by using the -c option with poudriere bulk:

poudriere bulk -c -f /usr/local/etc/ports.txt -j release93amd64.

Using the -C option instead of -c rebuilds only the ports listed in the file specified by the -f option or given on the command line. For example this would forcibly rebuild the www/apache22 package and no other package:

poudriere bulk -C -j release93amd64 www/apache22



10. Using the package repository with pkg(8)

The packages can be installed from the local packages directory with pkg add.

pkg add /usr/local/poudriere/data/packages/release93amd64-default/All/apache22.txz

A better way to use the packages is to use the pkg remote repository feature (see pkg-repository(5)) by creating a repository configuration file /usr/local/etc/pkg/repos/myrepo.conf and specifying the repository address as a file:// URL (to use the local file system) in this configuration file.

Code:
myrepo: {
    url             : "file:///usr/local/poudriere/data/packages/release93amd64-default",
    enabled         : yes,
    mirror_type     : NONE
}
Details of the syntax of this file can found in the pkg.conf(5) manual page.

Note that the repository URL does not have to end in /Latest like it used to be with the old pkg_* tools.

With that setting in place you can just do:

pkg install apache22

Updating the system when updated packages have been built is done simply by:

pkg upgrade

Note that pkg upgrade command updates the repository metadata automatically before any other operations, there is no need to run pkg update first.

Note that you probably want to disable the use of the official package repository by creating a /usr/local/etc/pkg/repos/FreeBSD.conf file with these contents:

Code:
FreeBSD: {
  enabled: no
}

Sharing the repository to other machines can be done with a web server, FTP server, NFS etc. The only requirement is that the repository files and directories are fetch(1)able.

For example this is how the repository would be shared using www/apache22. Rough guideline what to put in the httpd.conf file:

Code:
...
<Directory "/usr/local/poudriere/data/packages">
    Options Indexes FollowSymLinks

    AllowOverride None

    Order allow,deny
    Allow from all
</Directory>
...

<IfModule alias_module>
Alias /packages /usr/local/poudriere/data/packages
</IfModule>
With those settings a client machine can access the package repository with this repository configuration (save as for example /usr/local/etc/pkg/repos/myrepo.conf):

Code:
myrepo: {
  url: "http://webserver.mydomain.tld/packages/release93amd64-default"
  enabled : yes,
  mirror_type : NONE
}

11. RSA signing key for cryptographic verification of the repository. Optional.

This information is largely from Glen Barber's page https://glenbarber.us/2012/06/11/Maintaining-Your-Own-pkgng-Repository.html

mkdir -p /usr/local/etc/ssl/keys
chmod 600 /usr/local/etc/ssl/keys
mkdir -p /usr/local/etc/ssl/certs
openssl genrsa -out /usr/local/etc/ssl/keys/pkg.key 4096
openssl rsa -in /usr/local/etc/ssl/keys/pkg.key -pubout > /usr/local/etc/ssl/certs/pkg.cert

Add this line to /usr/local/etc/poudriere.conf:

Code:
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/pkg.key
On any system that uses the packages you'll have to copy the /usr/local/etc/ssl/certs/pkg.cert file over and put it in the same place (DO NOT COPY THE pkg.key FILE ANYWHERE!) and add these lines to /usr/local/etc/pkg/repos/myrepo.conf:

Code:
signature_type  : "PUBKEY",
pubkey: "/usr/local/etc/ssl/certs/pkg.cert",
Now the downloaded packages are verified that they have the correct cryptographic signatures before installation.

There is also a different method for signing the packages that uses an external signing command that can be used to keep the private key separate from the package build host. Refer to pkg-repo(8) for details.
 
Last edited:

rusty

Well-Known Member

Thanks: 43
Messages: 267

#3
Set this up today. I used http://www.allbsd.org for a specific version, ie
# poudriere jails -c -j 9-STABLE -f none -M /export/jails/9-STABLE -m allbsd -v r248915 -a amd64
I set things up with the
Code:
USE_TMPFS=all
option in /usr/local/etc/poudriere.conf. It really sped things up :e
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#4
Yes, it does, except the jail setup doesn't get much faster unfortunately, that's the penalty you pay for not using ZFS. I tested
Code:
USE_TMPFS=all
on a machine with 4 GB of RAM and it didn't seem to choke on editors/vim.
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#7
I have downgraded my only remaining FreeBSD system (a firewall) to an i386 machine with only 1 GB or RAM. I can report that I can still use ports-mgmt/poudriere-devel for building packages as I did on the amd64 system that had four times the memory and somewhat faster CPU.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,079
Messages: 26,940

#8
Works like a charm. Thank you very much.

I did hit one little snag when using PKGNG, it doesn't seem to create the needed repo.txz and you need to run # pkg repo /usr/local/poudriere/data/packages/release90amd64-default by hand. Can't this be automatically generated after the build run? Or did I miss something?
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#9
It should be generated automatically but here are some very recent changes to the repository format and there might be bugs in the intergration with poudriere. I think the repo.txz file is now deprecated and the repository metadata is now split into multiple files. I don't have my poudriere build system set up atm so I can't check.

Edit: I found this bit in the ports-mgmt/pkg commit log:

http://www.freshports.org/commit.php?message_id=201306240602.r5O62L91054147@svn.freebsd.org

Code:
- pkg repo produce a repo.txz in pkg 1.0 (legacy) fromat
So it looks like ports-mgmt/poudriere-devel has been changed to produce the newer type of repository metadata that does not include the repo.txz file anymore.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,079
Messages: 26,940

#10
kpa said:
It should be generated automatically but here are some very recent changes to the repository format and there might be bugs in the intergration with poudriere.
Ah, it's not really an issue to do it by hand but it would be nice to automate.

I think the repo.txz file is now deprecated and the repository metadata is now split into multiple files. I don't have my poudriere build system set up atm so I can't check.
Right! I noticed that too, but it seems to be a bit wonky at the moment. Sometimes it creates the 'old' repo.txz and at other times I get the new files. Maybe it's PKGNG that's not working correctly. I'm sure it'll be fixed soon, it's updated quite frequently.
 

spork

Member

Thanks: 9
Messages: 99

#12
Is anyone using the "sets" feature with poudriere, and if so, how are you dealing with maintaining repos across boxes that have different requirements? Let's take the example of three poudriere buildlists:

  • packages common to ALL hosts (common)
    • rsync
    • sshguard
    • vim
    • bash
    • apache22
    • php5
  • host group A (group-a)
    • perl 5.16
    • mysql51
    • postfix
  • host group B (group-b)
    • mysql50
    • perl 5.14
    • php53

Using the "sets" feature (the -z option to the bulk and options subcommands) would allow for each group specified above to have a custom set of options. The sets however also leave you with a package repo for each set, and a duplication of the common ports amongst all sets.

For example, building the three sets above:

Code:
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common -f /usr/local/etc/poudriere.d/buildlists/group-a
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common -f /usr/local/etc/poudriere.d/buildlists/group-b
And then you would have three repos:

/poudriere_data/packages/common
/poudriere_data/packages/group-a
/poudriere_data/packages/group-b

And all packages in the "common" buildlist would be built three times and the resulting packages would also exist in all three repos. You would also need to define a different repo in each group of hosts in pkg.conf. I'm fine with that, but I wonder if there's a more clever way to do this without using sets to avoid the duplicate builds (both saving the time it takes to build a package multiple times and saving disk space).

Perhaps sets aren't needed and this can all be done with only a supplemental buildlist for each group of hosts? I'm not sure what poudriere does if you try to build packages that conflict with each other in the same poudriere bulk run (in the example above, MySQL, PHP, and Perl have conflicting versions).

Anyone doing something similar?

TL;DR: What's the intended use-case for poudriere sets?
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#13
I haven't had time to look into sets yet and I don't really need them because at the moment I'm building the packages for just one system. Feel free to post your findings on this thread.
 

hedgehog

Active Member

Thanks: 20
Messages: 191

#15
@kpa, thank you for the great post!

I'm currently switching to ports-mgmt/poudriere for building packages for desktop, but I have a question. Can I use it for incremental updates of my packages without rebuilding them all every time? If yes, what should I do in case if there is a port that have libraries version bumped, so everything dependent on it should be rebuilt?
 
Last edited by a moderator:

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,079
Messages: 26,940

#16
hedgehog said:
Can I use it for incremental updates of my packages without rebuilding them all? If yes, what should I do in case if there is a port that have libraries version bumped, so everything dependent on it should be rebuilt?
Yes, that's possible. And you don't really need to do anything, poudriere already does it by default. If you want to make 100% sure your dependencies are correctly built you can always add the -c switch to the bulk command, that removes all previously built packages and starts fresh. If you don't use -c it will do an 'incremental', updating only the packages that are newer. It will also take care of any dependent packages.
 

hedgehog

Active Member

Thanks: 20
Messages: 191

#17
SirDice said:
Yes, that's possible. And you don't really need to do anything, poudriere already does it by default. If you want to make 100% sure your dependencies are correctly built you can always add the -c switch to the bulk command, that removes all previously built packages and starts fresh. If you don't use -c it will do an 'incremental', updating only the packages that are newer. It will also take care of any dependent packages.
Does that mean that, for instance, if the devel/gettext has it's version bumped, all the ports depending on devel/gettext will get rebuilt automatically, even if their version remained the same? Do you happen to know how is that achieved?
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#18
The dependency solver takes care of that and my experience with poudriere is that it works correctly in almost all cases. I yet have to see a case where a version bump failed to rebuild a dependency correctly. A package is also considered to be changed if it's options have changed and you have enabled the options checking in poudriere.conf.

The PKGNG pkg(8) client also tracks the shared library versions and considers a package changed if there are changes in the shared library dependencies. This should catch ports that didn't get their port versions bumped but have a dependency to a shared library that got a version bump.
 

hedgehog

Active Member

Thanks: 20
Messages: 191

#19
Thank you for answers.

Sorry for bombarding you with questions. Do I still need to manually request of reinstalling all packages that depends on devel/gettext in that case on the target system? Like that:
# pkg install -R gettext

Will the ports-mgmt/poudriere take care of dependencies that have they origins changed as well? For instance:
Code:
20121002:
  AFFECTS: users of devel/clanlib
  AUTHOR: jhale@FreeBSD.org

  devel/clanlib has been updated to 2.3.6 and the legacy version has been
  moved to devel/clanlib1 and updated to 1.0.0.  Although both versions
  may be installed at the same time, you should update your port origins
  first since many ports that depend on clanlib will not work with the
  new version.

  Update your port origins as follows:

  # portmaster -o devel/clanlib1 devel/clanlib
  or
  # portupgrade -fo devel/clanlib1 devel/clanlib
  or
  # pkg set -o devel/clanlib:devel/clanlib1
I use Kubuntu at work and the only repository advantage I see there is that you don't have to care about dependencies at all. You just issue apt-get upgrade command and it takes care of everything. I think it would be great if you could add some metadata for packages in repository that will, for example, ask a user if he wants to automatically reinstall all the packages that depends on the specific package while updating it to current version. Same with changing ports origins.
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#20
Changed port origins are not a problem as long as the ports that you want get actually built by a build run. If the changed port is a dependency of a port in your list of ports that pass to poudriere then there's no problem, the whole chain of dependencies gets rebuilt if needed. If the port is a "leaf port" (that no other port depends on) that is directly listed in your list then you have to rename the port in the list. (I have to check if MOVED is consulted in the latter case, it would be really great if it was).
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,079
Messages: 26,940

#21
hedgehog said:
Do I still need to manually request of reinstalling all packages that depends on devel/gettext in that case on the target system? Like that:
# pkg install -R gettext
Just run # pkg upgrade. It will correctly figure out which packages to update and/or replace. I haven't had any major issues doing this.
 

hedgehog

Active Member

Thanks: 20
Messages: 191

#22
Great, thank you guys. Looks like it always rebuilds all dependencies automatically and updates the repository, so # pkg upgrade will know that the packages were rebuilt.

Quite a smart solution to avoid conflicts and make it even more automated. The only drawback is the longer time required for incremental builds, but that's the cost of stability.
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#23
The non-development version of ports-mgmt/poudriere seems have support for the NO_ZFS option starting from version 3.0.1 or 3.0.2. I have tested building a few packages with ports-mgmt/poudriere-devel replaced with ports-mgmt/poudriere without changing anything in my configuration files and everything seems to work. I'll add a notice to his HOWTO as soon as I'm satisfied that the non-development version can be used without issues.

Edit: Changes committed. I have built enough packages now to be confident that the non-development version works as well as the development version.
 

hedgehog

Active Member

Thanks: 20
Messages: 191

#24
rusty said:
I set things up with the
Code:
USE_TMPFS=all
option in /usr/local/etc/poudriere.conf. It really sped things up :e
It gives insane speed boost because ports-mgmt/poudriere uses disk a lot when installing dependencies. Before having everything in tmpfs, I saw build-depends status all the time and it required more time than actually building ports.

Now I build everything in tmpfs and it's really fast. Also, I noticed that I usually have 7-10GB RAM free, out of 16GB, when using two builders, so probably it would be a good idea to increase builders number to 3.
 

kpa

Beastie's Twin

Thanks: 1,679
Messages: 6,084

#25
A new configuration option was added a while ago to automatically detect dependency changes.

This is the full description from poudriere.conf.sample:

Code:
# Automatic Dependency change detection
# When bulk building packages, compare the dependencies from kept packages to
# the current dependencies for every port. If they differ, the existing package
# will be deleted and the port will be rebuilt. This helps catch changes such
# as DEFAULT_RUBY_VERSION, PERL_VERSION, WITHOUT_X11 that change dependencies
# for many ports.
# Valid options: yes, no
#CHECK_CHANGED_DEPS=yes
 
Top