Pkg package repository using ports-mgmt/poudriere. With or without ZFS.

OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#26
A note on the -m url=file:///some/path method for creating a jail. This method is only supported by ports-mgmt/poudriere-devel at the moment (non-development versions is at version 3.0.10 at the time of writing this). I'd add this information to the HOWTO itself but the message is on the character limit and there's no more room.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#27
I switched to stable/10 aka 10-STABLE on my system yesterday and I'm now rebuilding everything using a stable/10 jail, very slowly though on a VIA C7 based system :OOO Everything has worked so far. I think we can call 10-STABLE a supported version of FreeBSD now.
 

hedgehog

Active Member

Thanks: 20
Messages: 191

#28
Is it possible to configure automounting when a jail starts using /etc/jail.conf and /etc/fstab.jailname? I am mounting system's ports options as nullfs while using ZFS, so they don't mount automatically using rc.conf.

By the way, can I just symlink /var/db/ports to poudriere data directory instead of nullfs mounting it?

(You can use late option to achieve this)
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#29
I'd also add this to the HOWTO itself but there's no room.

It's also possible to use SSH for fetching repository catalogues and packages. This is done by using an ssh:// URL as PACKAGESITE. For example:
Code:
PACKAGESITE: ssh://user@packagehost.mydomain.tld/usr/local/poudriere/data/packages/stable10i386-default
This saves you from setting up a web server on the package host if there's no other reason to have one.

This is more convenient when you have public key authentication with agent forwarding set up for user@packagehost so you don't have to type passwords on every pkg update operation.

Do not use the root user unnecessarily for logging into the package host, unprivileged user is fine as long as the user has the necessary access to the package repository directories.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#30
A little issue caught me off guard. It seems that the default options directory was changed at some point. It is now /usr/local/etc/poudriere.d/${jailname}-options. I still had a /usr/local/etc/poudriere.d/options directory and I was quite confused why my changed options were being ignored by the build runs. I deleted the /usr/local/etc/poudriere.d/options directory and re-did all the options. Everything is fine now.
 
Last edited by a moderator:

wszczep

Member

Thanks: 2
Messages: 31

#31
What version of ports-mgmt/poudriere do you have installes at the moment? I didn't notice such behaviour with version 3.0.11 - I am building packages for four different jails with one package list file and one options set.

Before starting each build batch I am checking options with poudriere options -f /home/admin/pkg_list.txt without specifying the jail name. Might be that you did run it for a named jail?

BTW. I have noticed that though the 'daemon' option is no longer specified in the man page, it still works.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#32
Yes you're right, I am indeed running the options with the -j jail option. I started doing that for some reason that I can no longer remember. I'll have to review my HOWTO if it's incorrect somehow in this regard.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#33
I shortened the poudriere.conf file part of the HOWTO. The DISTFILES_CACHE setting defaults to /usr/ports/distfiles and the default setting is completely fine now. Few of the settings I showed are the defaults anyway and don't have to be changed. I also added note about using the -j jail option in poudriere options.
 

nanotek

Active Member

Thanks: 1
Messages: 212

#35
Re: PKGNG package repository using ports-mgmt/poudriere. No

kpa said:
The packages can be installed from the local packages directory with pkg add.

pkg add /usr/local/poudriere/data/packages/release91amd64-default/Latest/apache22.txz
Thanks for the comprehensive guide, @kpa. It was a great help.

My $JAILNAME-default/Latest directory doesn't contain any packages:

ls /zbank/poudriere/data/packages/10x64-default/Latest/
Code:
pkg.txz
However, my $JAILNAME-default/All directory does:

ls /zbank/poudriere/data/packages/10x64-default/All/
Code:
GeoIP-1.4.8_3.txz               php5-hash-5.4.24.txz
apache24-2.4.6_1.txz            php5-iconv-5.4.24_1.txz
apr-1.4.8.1.5.3.txz             php5-json-5.4.24.txz
aspell-0.60.6.1_4.txz           php5-ldap-5.4.24.txz
autoconf-2.69.txz               php5-mbstring-5.4.24.txz
autoconf-wrapper-20131203.txz   php5-mcrypt-5.4.24.txz
bigreqsproto-1.1.2.txz          php5-mysql-5.4.24.txz
bison-2.7.1,1.txz               php5-mysqli-5.4.24.txz
ca_root_nss-3.15.4.txz          php5-openssl-5.4.24.txz
cclient-2007f,1.txz             php5-pdo-5.4.24.txz
cmake-2.8.12.1.txz              php5-pdo_mysql-5.4.24.txz
cmake-modules-2.8.12.1_1.txz    php5-pdo_pgsql-5.4.24.txz
curl-7.34.0.txz                 php5-pdo_sqlite-5.4.24.txz
cyrus-sasl-2.1.26_4.txz         php5-pgsql-5.4.24.txz
damageproto-1.2.1.txz           php5-pspell-5.4.24.txz
db5-5.3.28.txz                  php5-session-5.4.24.txz
dovecot2-2.2.10.txz             php5-simplexml-5.4.24.txz
dri2proto-2.8.txz               php5-sqlite3-5.4.24.txz
expat-2.1.0.txz                 php5-tokenizer-5.4.24.txz
fixesproto-5.0.txz              php5-wddx-5.4.24.txz
fontconfig-2.11.0_1,1.txz       php5-xml-5.4.24.txz
freeglut-2.8.1.txz              php5-xmlreader-5.4.24.txz
freetype2-2.5.2.txz             php5-xsl-5.4.24.txz
gdbm-1.11.txz                   php5-zip-5.4.24.txz
gettext-0.18.3.1.txz            php5-zlib-5.4.24.txz
glproto-1.4.17.txz              php55-5.5.8.txz
gmake-3.82_1.txz                php55-bz2-5.5.8.txz
google-perftools-2.1_1.txz      php55-ctype-5.5.8.txz
help2man-1.43.3_1.txz           php55-curl-5.5.8.txz
icu-52.1.txz                    php55-dom-5.5.8.txz
inputproto-2.3.txz              php55-exif-5.5.8.txz
jbigkit-1.6.txz                 php55-extensions-1.8.txz
jpeg-8_4.txz                    php55-fileinfo-5.5.8.txz
kbproto-1.0.6.txz               php55-filter-5.5.8.txz
libGL-7.6.1_4.txz               php55-ftp-5.5.8.txz
libGLU-9.0.0.txz                php55-gd-5.5.8.txz
libICE-1.0.8,1.txz              php55-hash-5.5.8.txz
libSM-1.2.2,1.txz               php55-iconv-5.5.8_1.txz
libX11-1.6.2,1.txz              php55-imap-5.5.8.txz
libXau-1.0.8.txz                php55-json-5.5.8.txz
libXaw-1.0.12,2.txz             php55-ldap-5.5.8.txz
libXdamage-1.1.4.txz            php55-mbstring-5.5.8.txz
libXdmcp-1.1.1.txz              php55-mcrypt-5.5.8.txz
libXext-1.3.2,1.txz             php55-mysql-5.5.8.txz
libXfixes-5.0.1.txz             php55-mysqli-5.5.8.txz
libXi-1.7.2,1.txz               php55-opcache-5.5.8.txz
libXmu-1.1.2,1.txz              php55-openssl-5.5.8.txz
libXp-1.0.2,1.txz               php55-pdo-5.5.8.txz
libXpm-3.5.11.txz               php55-pdo_mysql-5.5.8.txz
libXrandr-1.4.2.txz             php55-pdo_pgsql-5.5.8.txz
libXrender-0.9.8.txz            php55-pdo_sqlite-5.5.8.txz
libXt-1.1.4,1.txz               php55-pgsql-5.5.8.txz
libXxf86vm-1.1.3.txz            php55-phar-5.5.8.txz
libcheck-0.9.12.txz             php55-posix-5.5.8.txz
libdrm-2.4.17_1.txz             php55-pspell-5.5.8.txz
libevent2-2.0.21.txz            php55-session-5.5.8.txz
libexecinfo-1.1_3.txz           php55-simplexml-5.5.8.txz
libgcrypt-1.5.3.txz             php55-sockets-5.5.8.txz
libgd-2.1.0_1,1.txz             php55-sqlite3-5.5.8.txz
libgpg-error-1.12.txz           php55-tokenizer-5.5.8.txz
libiconv-1.14_1.txz             php55-xml-5.5.8.txz
libltdl-2.4.2_2.txz             php55-xmlreader-5.5.8.txz
libmcrypt-2.5.8.txz             php55-xmlwriter-5.5.8.txz
libpciaccess-0.13.2.txz         php55-zip-5.5.8.txz
libpthread-stubs-0.3_4.txz      php55-zlib-5.5.8.txz
libspf2-1.2.10_1.txz            phpMyAdmin-4.1.6.txz
libtool-2.4.2_2.txz             pkg-1.2.6.txz
libxcb-1.9.3.txz                pkgconf-0.9.4.txz
libxml2-2.8.0_3.txz             png-1.5.17.txz
libxslt-1.1.28_1.txz            postfix-2.11.0,1.txz
m4-1.4.17,1.txz                 postgresql93-client-9.3.2.txz
makedepend-1.0.5,1.txz          postgresql93-server-9.3.2.txz
mp3info-0.8.5.txz               printproto-1.0.5.txz
mysql56-client-5.6.16.txz       py27-fail2ban-0.8.11_1.txz
mysql56-server-5.6.16.txz       py27-libxml2-2.8.0.txz
nginx-1.4.4_3,1.txz             py27-setuptools-2.0.1.txz
oniguruma4-4.7.1.txz            python2-2_2.txz
openldap-client-2.4.38.txz      python27-2.7.6_2.txz
owncloud-6.0.1.txz              randrproto-1.4.0.txz
p5-Locale-gettext-1.05_3.txz    renderproto-0.11.1.txz
pciids-20131225.txz             roundcube-0.9.5,1.txz
pcre-8.34.txz                   sqlite3-3.8.2_1.txz
pdflib-7.0.5_2.txz              t1lib-5.1.2_2,1.txz
pecl-intl-3.0.0_1.txz           tiff-4.0.3.txz
pecl-pdflib-3.0.4.txz           tor-devel-0.2.4.20.txz
perl5-5.16.3_6.txz              wordpress-3.8.1,1.txz
php5-5.4.24.txz                 xcb-proto-1.9.txz
php5-bz2-5.4.24.txz             xcmiscproto-1.2.2.txz
php5-ctype-5.4.24.txz           xextproto-7.2.1.txz
php5-curl-5.4.24_1.txz          xf86bigfontproto-1.2.0.txz
php5-dom-5.4.24.txz             xf86vidmodeproto-2.3.1.txz
php5-exif-5.4.24.txz            xorg-macros-1.18.0.txz
php5-fileinfo-5.4.24.txz        xproto-7.0.25.txz
php5-filter-5.4.24.txz          xtrans-1.3.2.txz
php5-gd-5.4.24.txz
Did Poudriere change where packages are stored or has something gone awry in my configuration?
 
Last edited by a moderator:
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#36
Re: PKGNG package repository using ports-mgmt/poudriere. No

I believe I should re-write that part. The structure of the repository has changed a bit from the time I wrote the HOW-TO. What you're seeing is normal assuming you can install packages when the repository URL ends with packages/10x64-default.
 

nanotek

Active Member

Thanks: 1
Messages: 212

#37
Re: PKGNG package repository using ports-mgmt/poudriere. No

Thanks. That's good to know. My repo.conf:

cat /usr/local/etc/pkg/repos/repo.conf
Code:
repo: {
    url             : "file:///zbank/poudriere/data/packages/10x64-default",
    enabled         : yes,
    signature_type  : "PUBKEY",
    mirror_type     : NONE
    pubkey          : "/usr/local/etc/ssl/certs/repo.crt"
}
Works as expected:

pkg install tor-devel
Code:
Updating repository catalogue
The following 2 packages will be installed:

        Installing libevent2: 2.0.21 [repo]
        Installing tor-devel: 0.2.4.20 [repo]

The installation will require 6 MB more space

1 MB to be downloaded

Proceed with installing packages [y/N]: y
libevent2-2.0.21.txz                100%  269KB 268.8KB/s 268.8KB/s   00:00
tor-devel-0.2.4.20.txz              100% 1082KB   1.1MB/s   1.1MB/s   00:00
Checking integrity... done
[1/2] Installing libevent2-2.0.21... done
[2/2] Installing tor-devel-0.2.4.20...===> Creating users and/or groups.
Creating group '_tor' with gid '256'.
Creating user '_tor' with uid '256'.
 done
================================================================================
To enable the tor server, set tor_enable="YES" in your /etc/rc.conf
and edit /usr/local/etc/tor/torrc.  Note that the rc.subr script overrides
many torrc options and is tunable: see /usr/local/etc/rc.d/tor for details.
To use the torify script, install the net/torsocks port. After removing the
package, or updating to a new version, users may need to delete all files
in /var/db/tor.
================================================================================
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#38
Re: PKGNG package repository using ports-mgmt/poudriere. No

Small updates here and there. I changed all references to PACKAGESITE to refer to the repository configuration files under /usr/local/etc/pkg/repos. The optional RSA signing key part moved to the end since it's optional.
 

mecano

Active Member

Thanks: 10
Messages: 128

#39
Re: PKGNG package repository using ports-mgmt/poudriere. No

Thanks for this great tutorial @kpa,
at step 3 you enable devel/ccache in poudriere.conf :
Code:
# Use devel/ccache to speed up building, optional.
CCACHE_DIR=/var/cache/ccache
From /usr/local/share/doc/ccache/ccache-howto-freebsd.txt, I read
To use ccache for ports, just add WITH_CCACHE_BUILD=yes to
/etc/make.conf.
In your example make.conf for the ports-mgmt/poudriere jail I see no mention of it? Why? Is it enough to just define WITH_CCACHE_BUILD in /etc/make.conf?
 
Last edited by a moderator:
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#40
Re: PKGNG package repository using ports-mgmt/poudriere. No

I haven't used devel/ccache in a long time because I had some serious problems with it so I can't answer your question unfortunately. I left the reference to ccache in this HOWTO as an example, nothing more.
 

mecano

Active Member

Thanks: 10
Messages: 128

#41
Re: PKGNG package repository using ports-mgmt/poudriere. No

thanks will try with the WITH_CCACHE_BUILD in the jail make.conf to see.
[edit] I just realized that putting this option in the jail make.conf doesn't make sense because the devel/ccache cache folder will be out of the scope of the jail.
[edit] Also ports-mgmt/poudriere will take care of it: from poudriere.conf for CCACHE_DIR
It will be mounted into the jail and be shared among all jails.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#42
Re: PKGNG package repository using ports-mgmt/poudriere. No

Something to take note of. If you upgrade your jail, all packages built with that jail will be consider out of date and rebuilt. This is because the jails now have a timestamp associated with them and the packages built using that jail are actually located in a directory that uses the timestamp of the last built package in the name of the directory.

For example:

Code:
$ cat /usr/local/etc/poudriere.d/jails/release_10_0_amd64/timestamp 
1402191481

$ ls -l /usr/local/poudriere/data/packages/release_10_0_amd64-default/.latest    
lrwxr-xr-x  1 root  wheel  16 Jun  8 18:16 /usr/local/poudriere/data/packages/release_10_0_amd64-default/.latest -> .real_1402240565
In this case the timestamp of the jail is 1402191481 (seconds after epoch) and the last package building run was done at timestamp 1402240565. All good. Now if I upgraded the build jail so that the timestamp of the jail was greater that this 1402240565 all packages would have to be rebuilt.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#43
Re: PKGNG package repository using ports-mgmt/poudriere. No

There's a very neat new feature in the latest (3.0.99.20140619_1) version of ports-mgmt/poudriere-devel. It now allows the use of an existing jail as the build jail. This simplifies the updating of a jail greatly if the jail is of a stable branch and you're compiling everything from source yourself. This is how the jails are now created (this assumes you have finished make buildworld buildkernel on the host and /usr/obj contains the built files):

Code:
cd /usr/src
make installworld DESTDIR=/data/jails/build-stable-10-amd64 DB_FROM_SRC=1
make distrib-dirs DESTDIR=/data/jails/build-stable-10-amd64 DB_FROM_SRC=1
make distribution DESTDIR=/data/jails/build-stable-10-amd64 DB_FROM_SRC=1

poudriere jail -c -m null -M /data/jails/build-stable-10-amd64 -v 267841 -j build-stable-10-amd64
I used /data/jails/* here as the paths because that's what I have on my own system. The argument for the -v option needs to be an SVN revision number for some reason.

That's it. The same works for i386 jails. The neat thing is that the jails can be now shared between poudriere and other purposes like (for example) cross-compiling world and kernel for i386 only hosts. To update the jail you would first make sure the jail is not running and then just do the normal build, install, mergemaster(8), make delete-old delete-old-libs procedure on the jail. The other way to update the jail would be to run make installworld, mergemaster and make delete-old delete-old-libs in the jail itself in case you're also using the jail for building the world and kernel.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#44
With the help from moderators (thanks wblock@!) the HOWTO has been split into two separate posts to overcome the character limit for a single post that is lower (now 12500 characters) on this new forum software than it used to be and I can edit the HOWTO again.
 
OP
OP
K

kpa

Beastie's Twin

Thanks: 1,705
Messages: 6,123

#47
Removed some outdated information. For example WITH_PKGNG has been the default for a while for all supported versions of FreeBSD.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Thanks: 6,237
Messages: 27,228

#49
Apparently people are having a hard time finding this excellent howto. As it's an older thread it got snowed under with newer posts. This is just a bump to get it to float back to the top.
 
Top