Solved Poudriere compiles too many packages when I update

Grell

Grell

Hello, I am having an issue where when I update my packages using Poudriere, it winds up updating almost every package on my system, or at least a fair number of them, every time I update using Poudriere. For instance, there may only be a handful of packages that actually need updating, say maybe like 10-20, but Poudriere will go about recompiling almost everything on my system. The list that I have given Poudriere to compile from has been given using the command portmaster --list-origins so it fed a list of all "main" packages on my system (i.e. all the packages listed, but not all their dependencies). That may be an issue? Anyone have any ideas as it is not so optimal to compile 700-800 ports when only a handful have been updated.
Thanks.
 
By default, when poudriere needs to rebuild something, it also recursively deletes anything that depends on it (so, all these ports will be rebuilt as well). This is the only way to really ensure the packages work correctly, although it should not be necessary when using all default options (but then, why not use binary packages?) and assuming nobody ever forgot a necessary bump of a PORTREVISION.

Nevertheless, you can disable this behavior (which I wouldn't recommend), see poudriere-bulk(8):
Code: 
  -S    Do not recursively rebuild packages affected by    other packages
        requiring incremental rebuild.     This  may  result  in    broken
        packages  if the ones they depend on are updated, are not ABI-
        compatible, and    were not properly PORTREVISION bumped.

Another way to avoid unnecessary building is to use the package pre-fetching option, which will use "perfectly matching" official packages, for that feature you still need ports-mgmt/poudriere-devel.
 
Ok, thanks. So if that is the case then it would make sense to update packages a lot less frequently. When using portmaster I would update around once per day, some times more often. If it is going to pull in so many packages into the compilation process with each update it would make sense to update less. How often do most people update using Poudriere?
 
Grell said:
Anyone have any ideas as it is not so optimal to compile 700-800 ports when only a handful have been updated.
Click to expand...
True. Having loved to compile all with Poudriere, this turned to negative experiences due to some ports with huge dependencies. I do use Poudriere now only for those ports where I need to change the default port options. I stopped using large bulk lists and compile only those ports where there is a need to build a new version i.E. when there was bug fixing, not so for new features. This is of course not convenient, because you need to keep an eye on what ports changed and for what.
 
zirias@ said:
Another way to avoid unnecessary building is to use the package pre-fetching option, which will use "perfectly matching" official packages
Click to expand...
But this does not work well. Here it loads packages
Updating FreeBSD repository catalogue...
[pkg.FreeBSD.org] Fetching meta.conf: 100% 163 B 0.2kB/s 00:01
[pkg.FreeBSD.org] Fetching packagesite.pkg: 100% 7 MiB 2.5MB/s 00:03
Processing entries: 100%
FreeBSD repository update completed. 34259 packages processed.
All repositories are up to date.
[00:00:40] Package fetch: Will fetch 3 packages from remote or local pkg cache
The following packages will be fetched:

New packages to be FETCHED:
py39-hatch-vcs: 0.4.0 (10 KiB: 0.00% of the 1 GiB to download)
tex-formats: 20230313 (17 MiB: 1.48% of the 1 GiB to download)
texlive-texmf: 20230313 (1 GiB: 98.52% of the 1 GiB to download)

Number of packages to be fetched: 3

The process will require 1 GiB more space.
1 GiB to be downloaded.
Click to expand...

Even if Poudriere does not build a single port from the bulk list. texlive-texmf gets fetched every time again and again. This is more than only annoying. It does fetch it again if you just imediately restart Poudriere. What a waste of resources!
 
poudriere has a local package cache and there's nothing special about texlive-texmf (except for being a really huge package, but that's unrelated). If it's deleted (for the reasons I described above), it must be either built or fetched. And yes, this happens somewhat frequently here as well. If it's already there (in your local repo or the package cache) and nothing in the dependency chain changed, there's also no reason to fetch it, and I certainly can't confirm this behavior:
getopt said:
It does fetch it again if you just imediately restart Poudriere.
Click to expand...

So, if you're sure nothing else happened (like updating your ports tree and/or poudriere executing the "recursively delete ..." step), this would be about bug hunting.
 
zirias@ said:
So, if you're sure nothing else happened
Click to expand...
Yes, this happens when nothing changed (few seconds between Poudriere stopped and run again).

Wondering where fetched packages get stored/cached if so.
texlive-texmf seems to be a dependency for build only, not for run. Maybe this is the reason it does not get cached?
 
The Poudriere strategy that I use is "selectively update the ports tree (which is listed by poudriere ports -l) using the same list that I feed to poudriere-bulk(8) later on"...

This does result in deps not needing to be re-fetched. Re-compiled - probably, but the tarballs are already in the jail's /usr/ports/distfiles/ :P

The downside is that if my list of stuff has newer dependencies than what already exists in my tree, my compilation will fail, and I'd need to look through logs to see if some deps need to be updated, as well.
 
zirias@
Having watched what happens to the 1GB heavy fetched package:

/poudriere/data/packages/13amd64-default/.pkg-cache # ll texl*
-rw-r--r-- 1 root wheel 642M 231214_1730 texlive-texmf-20230313~8a890f52ff.pkg

so it gets downloaded there and this shows the state in the middle of the download. But after the poudriere bulk run has been completed, it is gone and no symlink has been created.
 
getopt if I'm not mistaken, it "works as intended" that this cache is gone after a successfully completed run, because it's in your repository then. So, the mystery is another one then: Why does poudriere think it must be rebuilt (or re-fetched) when indeed nothing changed?

Really sounds like there is some bug somewhere.
 
zirias@ said:
because it's in your repository then.
Click to expand...
It's not:
Code: 
# find / -name "texlive*"
/poudriere/ports/default/print/texlive-texmf-source
/poudriere/ports/default/print/texlive-tlmgr
/poudriere/ports/default/print/texlive-docs
/poudriere/ports/default/print/texlive-full
/poudriere/ports/default/print/texlive-texmf
/poudriere/ports/default/print/texlive-base
/poudriere/data/logs/bulk/latest-per-pkg/texlive-tlmgr
/poudriere/data/logs/bulk/latest-per-pkg/texlive-texmf
/poudriere/data/logs/bulk/13amd64-default/2023-11-15_12h40m42s/logs/texlive-texmf-20230313.log
/poudriere/data/logs/bulk/13amd64-default/2023-11-15_12h40m42s/logs/texlive-tlmgr-20230313.log
/poudriere/data/logs/bulk/13amd64-default/latest-per-pkg/texlive-tlmgr-20230313.log
/poudriere/data/logs/bulk/13amd64-default/latest-per-pkg/texlive-texmf-20230313.log
 
Grell said:
it winds up updating almost every package on my system, or at least a fair number of them, every time I update using Poudriere.
Click to expand...
To prevent it I use next script:

Code: 
#!/bin/sh

JAIL="132amd64"
PKGLIST="/usr/local/etc/poudriere.d/pkglist"
PKGLIST_ARGS="-j $JAIL -f $PKGLIST"

gitup ports > /dev/null

nbfailed=0
for i in `seq 10`; do
    poudriere bulk -S $PKGLIST_ARGS && break
    _rc=$?
    echo "pourdiere failed $i times with $nbfailed -> $_rc"
    [ "$nbfailed" -gt 0 -a "$_rc" -ge "$nbfailed" ] && break
    nbfailed=$_rc
done

poudriere pkgclean -y $PKGLIST_ARGS
poudriere distclean -y -f "$PKGLIST"

This script calls poudriere with "-S" in the loop, while number of builded ports decreases, because it isn't able to build all changed ports at once, because of broken dependencies. Also, if I remember it right, you need

Code: 
TRIM_ORPHANED_BUILD_DEPS=no

in the poudriere.conf.
 
Denis Shaposhnikov said:
To prevent it I use next script:

Code: 
#!/bin/sh

JAIL="132amd64"
PKGLIST="/usr/local/etc/poudriere.d/pkglist"
PKGLIST_ARGS="-j $JAIL -f $PKGLIST"

gitup ports > /dev/null

nbfailed=0
for i in `seq 10`; do
    poudriere bulk -S $PKGLIST_ARGS && break
    _rc=$?
    echo "pourdiere failed $i times with $nbfailed -> $_rc"
    [ "$nbfailed" -gt 0 -a "$_rc" -ge "$nbfailed" ] && break
    nbfailed=$_rc
done

poudriere pkgclean -y $PKGLIST_ARGS
poudriere distclean -y -f "$PKGLIST"

This script calls poudriere with "-S" in the loop, while number of builded ports decreases, because it isn't able to build all changed ports at once, because of broken dependencies. Also, if I remember it right, you need

Code: 
TRIM_ORPHANED_BUILD_DEPS=no

in the poudriere.conf.
Click to expand...
Now, this doesn't make much sense, because if deps are broken, all you gotta do is look at logs, fix port options, and restart poudriere-bulk(8) with the same list.
 
You must log in or register to reply here.
Back
Top