Solved poudriere-bulk(8) prefetching binary packages

grahamperrin

Daemon

Reaction score: 345
Messages: 1,165

From the manual page, after installing ports-mgmt/poudriere-devel 3.3.99.20210521:

Code:
-b name  Specify the name of the binary package branch to use to prefetch
         packages.  Should be "latest", "quarterly", "release_*", or url.
         With this option poudriere will first try to fetch from the
         binary package repository specified the binary package prior to
         do the sanity check if the package does not already exist.  When
         -t is used along with -C, or -c, then listed packages will not
         be fetched.

         See PACKAGE_FETCH_BRANCH, and PACKAGE_FETCH_URL, in
         poudriere.conf.sample.

Below: for things such as llvm10-10.0.1_5, why are some binary packages not prefetched?

Code:
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # pkg rquery -r FreeBSD '%o %v %R' llvm10
devel/llvm10 10.0.1_5 FreeBSD
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # poudriere ports -u
[00:00:00] Updating portstree "default" with git+https... done
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # cat /usr/local/poudriere/ports/default/print/pdfarranger/Makefile
# Created by: Alexander Vereeken <Alexander88207@protonmail.com>

PORTNAME=       pdfarranger
DISTVERSION=    1.7.1
CATEGORIES=     print

MAINTAINER=     Alexander88207@protonmail.com
COMMENT=        Helps merge or split PDF documents

LICENSE=        GPLv3
LICENSE_FILE=   ${WRKSRC}/COPYING

BUILD_DEPENDS=  ${PYTHON_PKGNAMEPREFIX}python-distutils-extra>0:devel/py-python-distutils-extra@${PY_FLAVOR}
RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}dateutil>0:devel/py-dateutil@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}pikepdf>0:textproc/py-pikepdf@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}gobject3>0:devel/py-gobject3@${PY_FLAVOR} \
                poppler-glib>0:graphics/poppler-glib

USES=           python gnome gettext
USE_PYTHON=     distutils
USE_GNOME=      intltool:build gtk30
USE_GITHUB=     yes
GH_ACCOUNT=     pdfarranger

INSTALLS_ICONS= yes

.include <bsd.port.mk>

root@mowa219-gjp4-vm-freebsd-13-zfs:~ # poudriere bulk -b latest -J 3 -j 13 print/pdfarranger
[00:00:01] Creating the reference jail... done
[00:00:01] Mounting system devices for 13-default
[00:00:01] Using packages from previously failed build: /usr/local/poudriere/data/packages/13-default/.building
[00:00:01] Mounting ports from: /usr/local/poudriere/ports/default
[00:00:01] Mounting packages from: /usr/local/poudriere/data/packages/13-default
[00:00:01] Mounting distfiles from: /usr/ports/distfiles
[00:00:01] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
/etc/resolv.conf -> /usr/local/poudriere/data/.m/13-default/ref/etc/resolv.conf
[00:00:01] Starting jail 13-default
[00:00:02] Logs: /usr/local/poudriere/data/logs/bulk/13-default/2021-05-28_00h04m59s
[00:00:02] Loading MOVED for /usr/local/poudriere/data/.m/13-default/ref/usr/ports
[00:00:02] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:02] Gathering ports metadata
[00:00:06] Calculating ports order and dependencies
[00:00:07] Trimming IGNORED and blacklisted ports
[00:00:07] Prefetching missing packages from pkg+http://pkg.FreeBSD.org/${ABI}/latest
Updating FreeBSD repository catalogue...
[13-default] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01   
[13-default] Fetching packagesite.txz: 100%    6 MiB   6.5MB/s    00:01   
Processing entries: 100%
FreeBSD repository update completed. 30505 packages processed.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
        cups: 2.3.3op2 (1 MiB: 28.50% of the 5 MiB to download)
        gnutls: 3.6.15 (2 MiB: 47.42% of the 5 MiB to download)
        libepoxy: 1.5.7 (280 KiB: 5.39% of the 5 MiB to download)
        libraqm: 0.7.1 (28 KiB: 0.54% of the 5 MiB to download)
        py38-pikepdf: 2.12.0 (366 KiB: 7.05% of the 5 MiB to download)
        py38-pillow: 8.2.0 (576 KiB: 11.10% of the 5 MiB to download)

Number of packages to be fetched: 7

The process will require 5 MiB more space.
5 MiB to be downloaded.
[13-default] Fetching libepoxy-1.5.7.txz: 100%  280 KiB 286.3kB/s    00:01   
[13-default] Fetching cups-2.3.3op2.txz: 100%    1 MiB   1.5MB/s    00:01   
[13-default] Fetching gnutls-3.6.15.txz: 100%    2 MiB   2.5MB/s    00:01   
[13-default] Fetching py38-pikepdf-2.12.0.txz: 100%  366 KiB 375.0kB/s    00:01   
[13-default] Fetching py38-pillow-8.2.0.txz: 100%  576 KiB 590.1kB/s    00:01   
[13-default] Fetching libraqm-0.7.1.txz: 100%   28 KiB  28.7kB/s    00:01   
[00:00:24] Sanity checking the repository
[00:00:24] Checking packages for incremental rebuild needs
[00:00:26] Deleting cups-2.3.3op2.txz: missing dependency: avahi-app-0.8
[00:00:26] Deleting gnutls-3.6.15.txz: missing dependency: p11-kit-0.23.22_1
[00:00:26] Deleting libepoxy-1.5.7.txz: missing dependency: mesa-libs-20.2.3
[00:00:26] Deleting libraqm-0.7.1.txz: missing dependency: harfbuzz-2.8.1
[00:00:27] Deleting py38-pillow-8.2.0.txz: missing dependency: libraqm-0.7.1
[00:00:27] Deleting py38-pikepdf-2.12.0.txz: missing dependency: py38-pillow-8.2.0
[00:00:29] Deleting stale symlinks... done
[00:00:29] Deleting empty directories... done
[00:00:29] Cleaning the build queue
[00:00:29] Sanity checking build queue
[00:00:29] Processing PRIORITY_BOOST
[00:00:29] Balancing pool
[13-default] [2021-05-28_00h04m59s] [balancing_pool:] Queued: 35 Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 35  Time: 00:00:28
[00:00:30] Recording filesystem state for prepkg... done
[00:00:30] Building 35 packages using 3 builders
[00:00:30] Starting/Cloning builders
[00:00:31] Hit CTRL+t at any time to see build progress and stats
[00:00:31] [01] [00:00:00] Building devel/llvm10 | llvm10-10.0.1_5
[00:00:31] [02] [00:00:00] Building textproc/yelp-xsl | yelp-xsl-40.0
[00:00:51] [02] [00:00:20] Finished textproc/yelp-xsl | yelp-xsl-40.0: Failed: fetch
…
 

Alain De Vos

Daemon

Reaction score: 541
Messages: 1,880

My 5-cent.
Or you use 100% binary.
Or you use 100% from source.
All the rest is asking for trouble.
At my PC everything is compiled from source using poudriere including every gcc,llvm.
 

Alain De Vos

Daemon

Reaction score: 541
Messages: 1,880

If you have CPU-time to compile applications, you have CPU-time to compile the compiler.
On the other hand, there is also room for improvement in poudriere.
I can live by keep promises or remove them.

Example what if my zpool does gzip-9 compression by default ?
Poudriere say's, I don't care, I don't inherit.
 

mtu

Active Member

Reaction score: 101
Messages: 156

The question:
  • for things such as llvm10-10.0.1_5, why are some binary packages not prefetched?
That's a known limitation/bug which is being worked on:

Example what if my zpool does gzip-9 compression by default ?
Poudriere say's, I don't care, I don't inherit.
That's a known limitation/bug which is being worked on:
 
OP
grahamperrin

grahamperrin

Daemon

Reaction score: 345
Messages: 1,165

Top