Problems with local pkg repo catalog

So I reran poudriere pkgclean with the "-vv" option and captured the voluminous output. I can attach
the file if desired but even compressed, it's over 400 kB.

Lots of lines of the form

Code:
[00:00:56] Found incorrect format file: /var/poudriere/data/packages/r130amd64full-default/.latest/All/zziplib-0.13.72_1.txz

seem suspicious.

I'm working on grokking pkgclean.sh where the file suffix is being checked but maybe this will ring a bell
with someone.
 
OK, after spending some time spelunking through the pkgclean logic, I think I understand what's
supposed to happen. The only files that are getting called out to be fed into the shredder are *.txz files.
In the new scheme, these are symlinks to *.pkg files and it looks like the *.pkg files
are safe.

Had I gone ahead with what poudriere pkgclean wanted to do, everything probably would have
been fine.

Still a bit afraid to try it though! :)
 
poudriere-devel-3.3.99.20210521.

… week rebuilding everything …

It should not take so long with binaries.

Code:
     -b name  Specify the name of the binary package branch to use to prefetch
              packages.  Should be "latest", "quarterly", "release_*", or url.
              With this option poudriere will first try to fetch from the
              binary package repository specified the binary package prior to
              do the sanity check if the package does not already exist.  When
              -t is used along with -C, or -c, then listed packages will not
              be fetched.

              See PACKAGE_FETCH_BRANCH, and PACKAGE_FETCH_URL, in
              poudriere.conf.sample.
 
… Had I gone ahead with what poudriere pkgclean wanted to do, …

Here, yesterday:

Code:
root@mowa219-gjp4-zbook-freebsd:~ # pkg clean -n
Nothing to do.
root@mowa219-gjp4-zbook-freebsd:~ # pkg clean -a -y
The following package files will be deleted:
        /var/cache/pkg/kmix-21.04.3.pkg
        /var/cache/pkg/kmix-21.04.3~0adcfd78b4.pkg
        /var/cache/pkg/volumeicon-0.5.1_1~dd3f2e06cc.pkg
        /var/cache/pkg/volumeicon-0.5.1_1.pkg
The cleanup will free 1 MiB
Deleting files: 100%
All done
root@mowa219-gjp4-zbook-freebsd:~ # poudriere pkgclean
poudriere pkgclean [options] [-f file|cat/port ...]

Parameters:
    -A          -- Remove all packages
    -a          -- Keep all known ports
    -f file     -- Get the list of ports to keep from a file
    [ports...]  -- List of ports to keep on the command line

Options:
    -j jail     -- Which jail to use for packages
    -J n        -- Run n jobs in parallel (Defaults to the number of
                   CPUs times 1.25)
    -n          -- Do not actually remove anything, just show what would be
                   removed
    -N          -- Do not build the package repository when clean completed
    -p tree     -- Which ports tree to use for packages
    -R          -- Clean RESTRICTED packages after building
    -v          -- Be verbose; show more information. Use twice to enable
                   debug output
    -y          -- Assume yes when deleting and do not confirm
    -z set      -- Specify which SET to use for packages
root@mowa219-gjp4-zbook-freebsd:~ # poudriere pkgclean -a -j 13-0-release
[00:00:00] Gathering all expected packages
[00:00:00] Creating the reference jail... done
[00:00:57] Mounting system devices for 13-0-release-default
[00:00:57] Mounting ccache from: /var/cache/ccache
[00:00:57] Mounting ports from: /usr/local/poudriere/ports/default
[00:00:57] Mounting packages from:
[00:00:57] Mounting distfiles from: /usr/ports/distfiles
[00:00:57] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
[00:00:57] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/13-0-release-default/ref/etc/resolv.conf
[00:00:57] Starting jail 13-0-release-default
[00:00:58] Loading MOVED for /usr/local/poudriere/data/.m/13-0-release-default/ref/usr/ports
[00:00:58] Ports supports: FLAVORS SELECTED_OPTIONS
[00:01:23] Gathering ports metadata
[00:05:42] Calculating ports order and dependencies
[00:06:08] Sanity checking the repository
[00:06:08] Unqueueing existing packages
[00:06:10] Sanity checking build queue
[00:06:12] Looking for unneeded packages
[00:06:12] Calculating size for found files... done
[00:06:12] These stale packages will be deleted:
/usr/local/poudriere/data/packages/13-0-release-default/.latest/All/bsdisks-0.26.txz
/usr/local/poudriere/data/packages/13-0-release-default/.latest/All/lsblk-3.6.txz
/usr/local/poudriere/data/packages/13-0-release-default/.latest/All/webcamd-5.13.2.6.txz
[00:06:12] Removing these 3 stale packages will free: 1.50 KiB
[00:06:12] Proceed? [y/N] y
[00:17:54] Removing files... done
[00:17:54] Deleting stale symlinks... done
[00:17:54] Deleting empty directories... done
[00:17:54] Creating pkg repository
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:18:01] Cleaning up
13-0-release-default: removed
13-0-release-default-n: removed
[00:18:01] Unmounting file systems
root@mowa219-gjp4-zbook-freebsd:~ #

Packages remain:

Code:
% du -hs /usr/local/poudriere/data/packages/13-0-release-default/All/
130M    /usr/local/poudriere/data/packages/13-0-release-default/All/
%
 
It should not take so long with binaries.

Code:
     -b name  Specify the name of the binary package branch to use to prefetch
              packages.  Should be "latest", "quarterly", "release_*", or url.
              With this option poudriere will first try to fetch from the
              binary package repository specified the binary package prior to
              do the sanity check if the package does not already exist.  When
              -t is used along with -C, or -c, then listed packages will not
              be fetched.

              See PACKAGE_FETCH_BRANCH, and PACKAGE_FETCH_URL, in
              poudriere.conf.sample.
I'm having some difficulty finding info on exactly what the "-b" option does. I find the man page
description a bit on the ambiguous side. Is there a doc somewhere that goes into more detail?

Suppose my list of ports includes www/firefox and I have non-standard options set for it
(which is, of course, my main reason for using poudriere in the first place). If I run
poudriere bulk -f <list of ports> -j <jail name> -b latest, will poudriere pull down binary
packages for all dependencies but then build firefox locally according to my options?

If the answer to the above is "yes". then if I have
Code:
OPTIONS_UNSET+=<stuff I don't want>
in /usr/local/etc/poudriere.d/make.conf, is the system smart enough to only pull
down packages that don't conflict with that?
 
Lazily:

poudriere bulk -b latest -j r130amd64full -Ct www/firefox

Downside of the laziness:

Code:
     -t       Add some testing to the specified ports.  Add -r to recursively
              test all port dependencies as well.  Currently, the behavior is
              to uninstall a port, and to disable parallel jobs for make(1).
              When used with -a then -rk are implied.

If you want parallel jobs, edit a .conf accordingly then (instead):

poudriere bulk -b latest -j r130amd64full -C www/firefox

Postscript: see below …
 
I have just pushed this change to Poudriere git (not ports yet) to clarify what -b will fetch. It's smart within reason.

-b name Specify the name of the binary package branch to use to prefetch packages. Should be "latest", "quarterly", "release_*", or url. With this option poudriere will first try to fetch from the binary
package repository specified the binary package prior to do the sanity check if the package does not already exist.

poudriere will only use packages that:
• come from a repository having the same or older version of pkg.
• do not have a locally fetched package already.
• are not IGNORED.
• match the expected local version.
• match the expected ABI.
• match the expected runtime and library dependencies.
• match the expected OPTIONS when CHECK_CHANGED_OPTIONS is enabled (default: on).
• The package is NOT listed in PACKAGE_FETCH_BLACKLIST
• The package is NOT listed with -C, or -c, when -t is used.
The -v flag can be used to show these decisions during build. Specifing twice will show more details on why some are skipped.

!!

poudriere has no way of determining differences outside of the above list. That is, if the local ports framework, or port, has custom patches or special WITH_FOO knobs (not OPTIONS) then it is
required to add its name into PACKAGE_FETCH_BLACKLIST. Otherwise a package may be fetched and used that lacks the custom patch or knob.

!!

See PACKAGE_FETCH_BRANCH, PACKAGE_FETCH_URL, PACKAGE_FETCH_BLACKLIST, and PACKAGE_FETCH_WHITELIST in poudriere.conf.sample. The entries in the lists will be matched against package names without
versions.

During the migration of pkg from .txz to .pkg it may fetch some packages that it then does not use. This isn't worth documenting or coding for as it is a temporary period.

About the - issue I've also blocked creating jails and ports trees with '-' and warn for set names. It doesn't necessarily cause build problems unless the various names have overlap, but it can break poudriere status parsing.

As for this...
-t Add some testing to the specified ports. Add -r to recursively
test all port dependencies as well. Currently, the behavior is
to uninstall a port, and to disable parallel jobs for make(1).
When used with -a then -rk are implied.
It is not correct. -t has no direct impact on whether or not make jobs are used. bulk does that normally unless a package is in ALLOW_MAKE_JOBS_PACKAGES. testport on the other hand *enables* make jobs for the port being tested.
I've added this blurb to poudriere-bulk(8):

poudriere will disable make jobs for packages not listed in ALLOW_MAKE_JOBS_PACKAGES, unless ALLOW_MAKE_JOBS is set in poudriere.conf. The number of jobs can be controlled in make.conf with the ports(7)
flags MAKE_JOBS_NUMBER and MAKE_JOBS_NUMBER_LIMIT. Beware that there is no global job control so each builder created from -J (or the default PARALLEL_JOBS) will spawn as many jobs as the ports framework
allows.
And this to poudriere-testport(8):
poudriere will allow make jobs for the port being tested. See poudriere-bulk(8) for more details on make jobs.
 
Back
Top