jails Getting a handle on Poudriere jail management

astyle

astyle

I'd like to ask for some help in getting a handle on Poudriere jail management. After a LOT of reading (mostly off https://github.com/freebsd/poudriere/wiki/), but also the Handbook's sections on "Building Ports with Poudriere" and the chapter on jails, I was able to distill 3 commands that I'd need to use to compile KDE packages:
  1. # poudriere jail -c kde . This should create my build jail. I don't need arch or other options, the compiled output should match the host anyway.
  2. # poudriere ports -c -m null /usr/ports/poudriere_tree . This should set up the ports tree that I created at /usr/ports/poudriere_tree/, with distfiles still in original /usr/ports/distfiles.
  3. # poudriere bulk -f ~/poud_kf5_plasma_sorted.txt -j kde . This compiles my list of packages that I later find in /usr/local/poudriere/data/packages/kde/ directory on the host.
Now, based on above discussion, I have some questions:
--
Should command #2 be run inside the jail named kde? The reason I'm asking - I tried to figure out if that's even important. The Handbook's chapter on jails suggests that I need to have a running jail, and to connect to it, before running stuff inside a jail (similar to a VM). However, Handbook's section on "Building Ports with Poudriere" doesn't say anything about that. Neither does poudriere(8). This leads me to conclude that command #2 is basically like setting an environment variable (the null-mounted directory), because the jail name is not even mentioned (Shouldn't there be an option to set it in poudriere.conf, then?)
--
Just trying to connect the dots here... am I missing something jail-related between commands #1 and #2? Or is that something that Poudriere will take care of for me, and all I need to do is run those commands consecutively? Am I on the right track?
--
 
Yeah, but command #2 looks like an environment variable, so it would look at home in poudriere.conf as a setting that I should be able to turn on. Lack of a jail name in that line worries me.
 
PS , I like to keep things very simple. I have only one ports tree on pc for everything and i do:
Code: 
/usr/local/bin/poudriere ports -c -m null -M /usr/ports    -p $PORTS

With poudriere you :
1. create a jail with a name
2. create a ports tree with a name, totally unrelated to any jail
3. configure options for the bulk make(if needed), which combines the jail 1. & the portstree 2.
4. do a bulk make you have to specify the jail 1. & the portstree 2.
 
Unless you are willing to modify the ports tree for some reason, like working with ports, there is absolute no reason to use a separated ports three to poudriere. Otherwise, with you are looking to modify the ports tree then, that would be better to clone/setup it on your $HOME to avoid using root/sudo/doas to work on it.

The second command is just to tell poudriere what ports three it should use as default. You can have as many as you want, for instance a the regular upstream as default to build the ports you use, and another where you work. Same for jails etc.

You can also have a look on the Pourdriere WIKI.
 
rigoletto@ said:
The second command is just to tell poudriere what ports three it should use as default. You can have as many as you want, for instance an the regular upstream as default to build the ports you use, and another where you work. Same for jails etc.
Click to expand...
Thanks, this suggests to me that my post (#3) is on the right track, and Poudriere will keep track of that variable for me. Seems like command #2 should only there if I need to change the tree away from the default specified in poudriere.conf.

Well, I do plan to keep updating the ports tree - I spent some time distilling a list of KDE ports I want to keep updated. But I also want a clean backup to fall back to if I mess up somewhere. I think that qualifies for "modifying the ports tree". 😅
 
The poudriere-bulk(8) command usually has a -p myportstree option, if not specified it will use the default ports tree. This ports tree is nullfs(5) mounted inside each of the jails that are spawned. Depending on your system you can have 1 or more jobs running, each job will spawn a fresh new jail for each build. So if you have 8 cores you will see 8 jobs, each with its own jail (and the ports tree plus distfiles nullfs(5) mounted in them).
 
astyle said:
… that variable …
Click to expand...

Try to not think of it as a variable.

… want a clean backup …
Click to expand...

If clean means without all your modifications, then like this – although don't think of it as a backup (it's simply a separate tree):

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # poudriere ports -c
[00:00:01] Error: The ports tree, default, already exists
root@mowa219-gjp4-8570p-freebsd:~ # poudriere ports -c -p nonmodified
[00:00:00] Creating nonmodified fs at /usr/local/poudriere/ports/nonmodified... done
[00:00:01] Cloning the ports tree...^C[00:00:03] Error: Signal SIGINT caught, cleaning up and exiting
[00:00:03] Error while creating ports tree, cleaning up.
root@mowa219-gjp4-8570p-freebsd:~ #

Alain De Vos said:
… only one ports tree …

1. create a jail with a name
2. create a ports tree with a name, totally unrelated to any jail
3. configure options for the bulk make(if needed), which combines the jail 1. & the portstree 2.
4. do a bulk make you have to specify the jail 1. & the portstree 2.
Click to expand...

If you have only one tree – with its default name – then poudriere-bulk(8) should not require specification of a tree.

<https://github.com/freebsd/poudriere/wiki/poudriere-bulk.8-devel>

An example:

poudriere bulk -C -j main deskutils/recoll


In the transcript below, note the phrase Copying /var/db/ports from: /usr/local/etc/poudriere.d/options –

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # poudriere ports -u
Updating files: 100% (38/38), done.ult" with git+https...Updating files:  15% (6/38)
 done
root@mowa219-gjp4-8570p-freebsd:~ # poudriere bulk -C -j main deskutils/recoll
[00:00:00] Creating the reference jail... done
[00:00:27] Mounting system devices for main-default
[00:00:27] Stashing existing package repository
[00:00:27] Mounting ccache from: /var/cache/ccache
[00:00:27] Mounting ports from: /usr/local/poudriere/ports/default
[00:00:27] Mounting packages from: /usr/local/poudriere/data/packages/main-default
[00:00:27] Mounting distfiles from: /usr/ports/distfiles
[00:00:27] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
[00:00:27] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/main-default/ref/etc/resolv.conf
[00:00:27] Starting jail main-default
[00:00:27] Will build as nobody:nobody (65534:65534)
[00:00:28] Logs: /usr/local/poudriere/data/logs/bulk/main-default/2021-12-02_05h08m09s
[00:00:28] Loading MOVED for /usr/local/poudriere/data/.m/main-default/ref/usr/ports
[00:00:29] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:29] Inspecting ports tree for modifications to git checkout... yes
[00:00:46] Ports top-level git hash: 78f384660e (dirty)
[00:00:46] Gathering ports metadata
[00:00:51] Calculating ports order and dependencies
[00:00:51] -C specified, cleaning listed packages
[00:00:51] (-C) Will delete existing package: recoll-1.31.2.pkg
[00:00:51] (-C) Flushing package deletions
[00:00:52] Trimming IGNORED and blacklisted ports
[00:00:52] Package fetch: Looking for missing packages to fetch from pkg+http://pkg.FreeBSD.org/${ABI}/latest
Updating FreeBSD repository catalogue...
[main-default] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01  
[main-default] Fetching packagesite.pkg: 100%    6 MiB   6.5MB/s    00:01  
Processing entries: 100%
FreeBSD repository update completed. 30854 packages processed.
All repositories are up to date.
[00:01:13] Package fetch: Will fetch 2 packages from remote or local pkg cache
The following packages will be fetched:

New packages to be FETCHED:
        libxkbfile: 1.1.0 (105 KiB: 56.67% of the 186 KiB to download)
        xkbcomp: 1.4.5 (81 KiB: 43.33% of the 186 KiB to download)

Number of packages to be fetched: 2

186 KiB to be downloaded.
[main-default] Fetching libxkbfile-1.1.0.pkg: 100%  105 KiB 107.8kB/s    00:01  
[main-default] Fetching xkbcomp-1.4.5.pkg: 100%   81 KiB  82.4kB/s    00:01  
[00:01:13] Sanity checking the repository
[00:01:13] Checking packages for incremental rebuild needs
[00:01:14] Deleting llvm12-12.0.1_6.pkg: new version: 12.0.1_7
[00:01:15] Deleting vulkan-headers-1.2.200.pkg: new version: 1.2.201
[00:01:16] Deleting mesa-dri-21.1.8.pkg: missing dependency: llvm12-12.0.1_6
[00:01:16] Deleting qt5-gui-5.15.2_7.pkg: missing dependency: mesa-dri-21.1.8
[00:01:16] Deleting qt5-widgets-5.15.2_3.pkg: missing dependency: qt5-gui-5.15.2_7
[00:01:16] Deleting qt5-printsupport-5.15.2_1.pkg: missing dependency: qt5-gui-5.15.2_7
[00:01:17] Deleting stale symlinks... done
[00:01:17] Deleting empty directories... done
[00:01:17] Package fetch: Generating logs for fetched packages
[00:01:18] Unqueueing existing packages
[00:01:18] Unqueueing orphaned build dependencies
[00:01:18] Sanity checking build queue
[00:01:18] Processing PRIORITY_BOOST
[00:01:18] Balancing pool
[main-default] [2021-12-02_05h08m09s] [balancing_pool:] Queued: 9  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 2  Tobuild: 7   Time: 00:00:50
[00:01:18] Recording filesystem state for prepkg... done
[00:01:18] Building 7 packages using 2 builders
[00:01:18] Starting/Cloning builders
[00:01:19] Hit CTRL+t at any time to see build progress and stats
[00:01:19] [01] [00:00:00] Building devel/llvm12 | llvm12-12.0.1_7
[00:01:19] [02] [00:00:00] Building graphics/vulkan-headers | vulkan-headers-1.2.201
[00:01:43] [02] [00:00:24] Finished graphics/vulkan-headers | vulkan-headers-1.2.201: Success
^C[00:07:47] Error: Signal SIGINT caught, cleaning up and exiting
[main-default] [2021-12-02_05h08m09s] [sigint:] Queued: 9 Built: 1  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 2  Tobuild: 6   Time: 00:07:20
[00:07:47] Logs: /usr/local/poudriere/data/logs/bulk/main-default/2021-12-02_05h08m09s
[00:07:47] Cleaning up
main-default: removed
main-default-n: removed
[00:07:48] Unmounting file systems
root@mowa219-gjp4-8570p-freebsd:~ #

The phraseology (Copying /var/db/ports) might be questionable, but here's the essence:

Code: 
% ls -hl /usr/local/etc/poudriere.d/options
total 1
drwxr-xr-x  2 root  wheel     3B 28 Nov 10:48 deskutils_recoll
drwxr-xr-x  2 root  wheel     3B 10 Nov 03:01 emulators_wine
drwxr-xr-x  2 root  wheel     2B 20 Nov 04:03 www_falkon
drwxr-xr-x  2 root  wheel     3B 10 Nov 02:59 www_firefox
% cat /usr/local/etc/poudriere.d/options/deskutils_recoll/options
# This file is auto-generated by 'make config'.
# Options for recoll-1.31.2
_OPTIONS_READ=recoll-1.31.2
_FILE_COMPLETE_OPTIONS_LIST=ASPELL CHM DJVU IMAGE LATEX PDF PS QT5 SOUND X11MON
OPTIONS_FILE_SET+=ASPELL
OPTIONS_FILE_SET+=CHM
OPTIONS_FILE_SET+=DJVU
OPTIONS_FILE_SET+=IMAGE
OPTIONS_FILE_SET+=LATEX
OPTIONS_FILE_SET+=PDF
OPTIONS_FILE_SET+=PS
OPTIONS_FILE_SET+=QT5
OPTIONS_FILE_SET+=SOUND
OPTIONS_FILE_SET+=X11MON
%

Hint

I recently made the mistake of having an empty /usr/local/etc/poudriere.d/main-options directory (where main is the name of the jail), probable debris from an experiment whilst attempting to learn. I assumed that removing the content of the directory would be enough for the directory to have no effect.

<https://github.com/freebsd/poudriere/discussions/957#discussioncomment-1712493> I learnt that presence of the empty directory had an unwanted effect. After I removed the directory, the next run of poudriere got what I intended from my /usr/local/etc/poudriere.d/options/deskutils_recoll/options file.
 
You must log in or register to reply here.
Back
Top