How to use packages with Poudriere Image

Phishfry

Phishfry

I have Poudriere Image under my belt for a while.

Now I am trying to add packages to my images.

Let me show my commands.
Create my jail:
poudriere jail -c -j mbm-mfs -v 13.5-RELEASE -K GENERIC -b -m src=/usr/src -J4

Create Ports Tree:
poudriere ports -c -p mbm_ports -m null -M /usr/ports
This seem to give me troubles with null. So I did not use /usr/ports/. Instead I installed git and poudriere built its own ports tree.
poudriere ports -c -p mbm_ports

Build pkglist:
poudriere bulk -j mbm-mfs -p mbm_ports -f mbm-pkglist
This all works but I really want to download packages instead of building them.
So I add the -b flag and quarterly setting.
poudriere bulk -j mbm-mfs -p mbm_ports -f mbm-pkglist -b quarterly
This gives me some packages but still tries to build some ports. I do not understand this logic. It acts like it cannot find some packages.
I am aware of pkg blacklist and whitelist in poudriere.conf.

Can anyone give me advice? Do I need to alter anything in poudiere.conf?

I did change my port name to default.
Code: 
[mbm-mfs-default] Fetching xvinfo-1.1.5_1~6286b15567.pkg: 100%    7 KiB   7.2kB/s    00:01
[mbm-mfs-default] Fetching xorg-fonts-cyrillic-7.7~e5d1180597.pkg: 100%    1 KiB   1.1kB/s    00:01
[mbm-mfs-default] Fetching xlsclients-1.1.5_1~f22a195864.pkg: 100%    9 KiB   9.2kB/s    00:01
[mbm-mfs-default] Fetching xcmsdb-1.0.5_1~05a3402512.pkg: 100%   17 KiB  17.0kB/s    00:01
[mbm-mfs-default] Fetching xorg-fonts-truetype-7.7_1~8fb04bdd34.pkg: 100%    1 KiB   1.1kB/s    00:01
[mbm-mfs-default] Fetching autoconf-switch-20220527~a8a5bcd5ab.pkg: 100%    1 KiB   1.5kB/s    00:01
[mbm-mfs-default] Fetching xdriinfo-1.0.7~49180e07d8.pkg: 100%    5 KiB   5.6kB/s    00:01
[mbm-mfs-default] Fetching libGLU-9.0.3~d7327df97a.pkg: 100%  211 KiB 216.2kB/s    00:01
[mbm-mfs-default] Fetching lerc-4.0.0~8aa41412a1.pkg: 100%  164 KiB 167.7kB/s    00:01
[00:01:09] Package fetch: Using cached copy of aom-3.13.1
[00:01:09] Package fetch: Using cached copy of appres-1.0.7
[00:01:09] Package fetch: Using cached copy of autoconf-switch-20220527
[00:01:09] Package fetch: Using cached copy of bash-completion-zfs-2.3.3
[00:01:09] Package fetch: Using cached copy of bdftopcf-1.1_3
[00:01:09] Package fetch: Using cached copy of bitmap-1.1.1
[00:01:09] Package fetch: Using cached copy of dav1d-1.5.2
[00:01:09] Package fetch: Using cached copy of docbook-1.5
[00:01:09] Package fetch: Using cached copy of docbook-sgml-4.5_1
[00:01:09] Package fetch: Using cached copy of docbook-xml-5.0_3
So it acts like it is getting packages then starts building.
Here is right before it starts building ports:
Code: 
[00:01:09] Package fetch: Using cached copy of xxhash-0.8.3
[00:01:09] Checking packages for incremental rebuild needs
[00:01:10] Deleting libXaw-1.0.16,2.pkg: missing dependency: libXpm-3.5.17_1
[00:01:10] Deleting x11perf-1.6.1_1.pkg: missing dependency: libXft-2.3.8
[00:01:10] Deleting vulkan-loader-1.4.327.pkg: missing dependency: wayland-1.24.0_2
[00:01:10] Deleting xcalc-1.1.2_1.pkg: missing dependency: font-adobe-100dpi-1.0.3_4
[00:01:10] Deleting xclock-1.0.9_1.pkg: missing dependency: libXaw-1.0.16,2
Deleting its work "for incremental rebuild needs".

How can I get this thing on all packages?

Code: 
[00:01:10] Deleting stale symlinks... done
[00:01:10] Deleting empty directories... done
[00:01:10] Package fetch: Generating logs for fetched packages
[00:01:12] Unqueueing existing packages
[00:01:12] Unqueueing orphaned build dependencies
[00:01:12] Sanity checking build queue
[00:01:12] [mbm-mfs-default] [2025-12-18_17h45m56s] [pkgqueue_sanity_check] Time: 00:01:10
           Queued: 396 Inspected: 0 Ignored: 0 Built: 0 Failed: 0 Skipped: 0 Fetched: 156 Remaining: 240

poudriere image -t usb+mfs -s 1200m -j mbm-mfs -h mbm-mfs -n mbm-mfs -X excluded.files -c ./overlay/mbm-mfs -f mbm-pkglist
 
This is the crippler:
Code: 
[01:07:09] [01] [00:00:00] Building   devel/llvm19@default | llvm19-19.1.7_1
Spent 8 hours on it last night building on a NUC7 and it seemed to be the wrong version with llvm15@default

I think I have it figured out. The ports I built last night were using "-M /usr/ports/" and then I switched to git and its download was newer by grabbing quarterly. 13.5 Installer ports were is older.

So I did delete the ports tree with poudriere ports -d but it looks like the were some lingering items in the /data/cache directory.
That might be causing this problem.. I need delete old files better next time.
 
I wiped the box and reinstalled from scratch. I wanted to start new.

One thing that I have trouble understanding is why if I am building an image with poudriere and I want all packages. Why do I need this step:
poudriere ports -c -p NAME
The reason it exists is to create a ports tree. I don't want that. I want packages. It seems you should be able to skip ports all-together with poudriere bulk and just download packages.

So hack in a variable name for poudriere bulk? The '-b BRANCH' seems to require the ports tree setting '-p NAME'. That is a problem.
There should be a way to just download everything with a pkglist.
-b option needs fixing I believe. Thoughts? Settings I am missing? Do you use the -b option with bulk?
poudriere bulk -j mbm -p mbm_ports -f mbm-pkglist -b quarterly
I am telling it to use a ports tree and packages. No wonder it is confused.

mbm-pkglist
Code: 
misc/ytree
x11/xorg
x11-wm/openbox
11-fm/xfe
x11/tint
astro/gpsd
astro/gpxloggerd

I realize my usage of "packages for everything" goes against the whole concept of poudriere.
But for building MFS images, ports//packages//applicatons need to be pre-installed. I need no port options set. Default packages work for me.

So you can see from my pkglist that llvm19 and rust are needed as dependencies.
I have uncommented them in the poudriere whitelist for trial..

#PACKAGE_FETCH_WHITELIST="gcc* rust llvm*"
Should I just add my 300 packages to this list? It seems excessive and probably won't work....
I have experimented with it and it does not handle dependent ports. So all would be required.
 
Here my two scripts i'm using,

poudriere_nice :
Code: 
nice -n 31 idprio 31 ./poudriere_run
pkg update -f
pkg upgrade

poudriere_run:
Code: 
#poudriere jail  -d          -j pjail2
poudriere jail  -c          -j pjail2 -v 15.0-RELEASE
poudriere ports -c                    -p default2 -m git+https -B 2025Q4
poudriere ports -u                    -p default2
poudriere options           -j pjail2 -p default2 -f ./packagelist.txt
poudriere bulk -b quarterly -j pjail2 -p default2 -f ./packagelist.txt
 
Whats that saying: The definition of insanity is doing the same thing over and over and expecting different results????

Poudriere Bulk is making me insane. I have wasted so many hours trying to figure out the logic. Its maddening.
My newest attempt was use packages for all big toolkits that are a time suck..

perl5
python311
bin-utils
cmake-core
rust
llvm19
libclc-llvm19
spirv*
glslang
libjxl
mesa-dri

Utterly failed and only used one from the WHITELIST. One of the smallest ports-perl5..
Code: 
[10:24:53] Fetched ports: lang/perl5.42
[10:24:53] [mbm-default] [2025-12-20_05h26m31s] [committing] Time: 10:24:49
           Queued: 169 Inspected: 0 Ignored: 0 Built: 168 Failed: 0 Skipped: 0 Fetched: 1 Remaining: 0
[10:24:53] Logs: /poudriere/data/logs/bulk/mbm-default/2025-12-20_05h26m31s
[10:24:53] Cleaning up
[10:24:53] Stopping up to 4 builders
[10:24:53] Unmounting file systems

I only used a smaller subset with the above result.
/usr/local/etc/poudriere.conf Relevant line:
PACKAGE_FETCH_WHITELIST="gcc* rust llvm* spirv* bin-utils cmake-core perl5 python311 libclc-llvm19"
Click to expand...
 
I have currently 2500 packages installed binary.
And a 10 of them are from source, poudriere,
lang/php84
lang/php84-extensions
net-mgmt/zabbix74-server
net-mgmt/zabbix74-agent
net-mgmt/zabbix74-frontend
misc/py-pytorch
lang/harec
devel/hare
ports-mgmt/pkg

For me everything works perfectly fine, meaning somewhere you must be doing something really wrong.

This is my /usr/local/etc/pkg/repos # cat poudriere.conf
poudriere: {
url: "file:///usr/local/poudriere/data/packages/pjail2-default2",
mirror_type: "none",
signature_type: "none",
enabled: yes,
priority: 10
}
 
Phishfry said:
This all works but I really want to download packages instead of building them.
So I add the -b flag and quarterly setting.
poudriere bulk -j mbm-mfs -p mbm_ports -f mbm-pkglist -b quarterly
This gives me some packages but still tries to build some ports. I do not understand this logic. It acts like it cannot find some packages.
Click to expand...
A long known problem with Poudriere. See a useless PR 286954 as the maintainer is bugzilla ignorant. ports-mgmt/poudriere-dsh2dsh seams to work better.
You might get better results using option -b latest as quarterly is lagging by design.
 
Thank You for that PR.
I am not insane after all.

I worried about "-b latest" because it adds the time factor. Always a newer version of something when there is hundreds of ports involved.
 
That was a humbling PR read. The complainer makes good points but rules are made to be broken?
NanoBSD falls under the source tree maybe poudriere should too.

I had long wondered why simple things like a few variables for ARM w/bootsector builds are not included.
Being a script makes it so easy to modify. But should you...
I thought of making a "feature request" patch with my work for poudriere images for ARM but it seems so simple I dunno why it's not there..
I kinda figured the authors were gatekeeping.

NanoBSD fell off a similar cliff with phk drifting away and imp doing some work for ARM images. It kinda feels like abandonedware.
No EFI support made poudriere-image a nice upgrade. But these build times are very poor.

Previously I used "-t usb" images and I could add packages later. Now I need them baked in for MFS and RO filesystem
 
I am not sure, what time factor is being mentioned here. The Poudriere compiler machine would most likely run a wrapper script, that downloads the latest branch of the ports tree and then bulk builds the thousands of ports from one or more sets and build lists. I hardly have a single failed port and such are almost always related to the huge Electron framework. The script is called weekly or so from CRON - and maintains package repository, that can by used by PKG on all machines in the network.
 
My comment about time factor was about this.
You start a build of 400 packages with 'latest' and it takes 2 days. By the time you are done you may already have stale packages.
Things change that quickly.
With Quarterly (depending on date) you can start a build and 4 days later the source is the same.
Time factor of "Latest" means it could be stale before it is done compiling. Circular looping for newness.
Quarterly negates that. (unless on cusp of Quarterly change)
 
Well I tried /ports-mgmt/poudriere-dsh2dsh and As Stated it does better but now 4 packages fetched:
Code: 
FreeBSD is up to date.
[00:00:31] Package fetch: Will fetch 4 packages from remote or local pkg cache
Updating database digests format: 100%
The following packages will be fetched:

New packages to be FETCHED:
        cmake-core: 3.31.9 (13 MiB: 6.22% of the 215 MiB to download)
        perl5: 5.42.0_1 (15 MiB: 7.06% of the 215 MiB to download)
        python311: 3.11.13_1 (25 MiB: 11.69% of the 215 MiB to download)
        rust: 1.89.0_1 (161 MiB: 75.04% of the 215 MiB to download)

Number of packages to be fetched: 4

The process will require 215 MiB more space.
215 MiB to be downloaded.
[mbm-mbm_ports] Fetching rust-1.89.0_1~aa648dcd06.pkg: 100%  161 MiB   8.9MB/s    00:19
[mbm-mbm_ports] Fetching perl5-5.42.0_1~e999ef0678.pkg: 100%   15 MiB   5.3MB/s    00:03
[mbm-mbm_ports] Fetching cmake-core-3.31.9~fa796d2511.pkg: 100%   13 MiB   3.5MB/s    00:04
[mbm-mbm_ports] Fetching python311-3.11.13_1~82ca08750f.pkg: 100%   25 MiB   6.6MB/s    00:04
[00:01:04] Package fetch: Using cached copy of cmake-core-3.31.9
[00:01:04] Package fetch: Using cached copy of perl5-5.42.0_1
[00:01:04] Package fetch: Using cached copy of python311-3.11.13_1
[00:01:04] Package fetch: Using cached copy of rust-1.89.0_1
[00:01:04] Deleting stale symlinks... done
[00:01:04] Deleting empty directories... done
[00:01:04] Package fetch: Generating logs for fetched packages
[00:01:04] Unqueueing existing packages
[00:01:04] Unqueueing orphaned build dependencies
[00:01:04] Sanity checking build queue
[00:01:04] [mbm-mbm_ports] [2025-12-20_22h10m20s] [pkgqueue_sanity_check] Time: 00:00:59
           Queued: 195 Inspected: 0 Ignored: 0 Built: 0 Failed: 0 Skipped: 0 Fetched: 4 Remaining: 191
This is on fresh install of 13.5-RELEASE with source tree installed from memstick installer. 2025Q4 ports tree downloaded via git.
git clone -b 2025Q4 https://git.freebsd.org/ports.git /usr/ports
Same pkglist as above. (minus gps stuff)
PACKAGE_FETCH_WHITELIST="perl5 cmake-core python311 rust llvm19 libclc-llvm19 glslang libjxl mesa-dri libheif"
 
I just had a thought.
PACKAGE_FETCH_WHITELIST=
Does not resolve pkg dependencies from my testing. It forces port build for the dependencies..

So maybe some of these build tools have build dependencies and it is failing.
I will chase down that idea.
What does llvm19 depend on.

This new fork I am trying has some different settings I need to twiddle.
 
You must log in or register to reply here.
Back
Top