poudriere seems to like to rebuild everything

Alain De Vos

Alain De Vos

I would like to rebuild only what is necessary.
In a file pkglist.txt i have 400 ports listed
This is how i run poudriere.
Code: 
pkglist="./pkglist.txt"
jail="poudbjail"
port="local"
time nice -n 15 poudriere bulk    -j $jail -p $port -f $pkglist

This command currently want to build 700 packages. I want poudriere to stop rebuilding everything when it is not needed. Maybe something in poudriere.conf ?

When i put only one port in pkglist.txt poudriere seems to be clever enough to know a rebuild is not needed. With 400 ports it just rebuilds everything.
 
poudriere never rebuilds "everything". But it's "playing it safe" with dependencies -- whenever a port needs to be rebuilt, all dependent packages are deleted as well, so they will be rebuilt, too. Often, this wouldn't be really necessary (e.g. with RUN_DEPENDS, or LIB_DEPENDS if the library ABI doesn't change), but poudriere currently has no way to figure that out. The only thing you can do is to use ccache; at least with C and C++ software, this will avoid a lot of unnecessary recompiles.
 
Zirias said:
poudriere ... is "playing it safe" with dependencies -- whenever a port needs to be rebuilt, all dependent packages are deleted as well, so they will be rebuilt, too. Often, this wouldn't be really necessary ... but poudriere currently has no way to figure that out.
Click to expand...
Humans don't necessarily have a way to figure that out either. For several years I used Archlinux and, at least at that time and probably still, the developers did not "play it safe" and rebuild dependencies when a package was updated. The new package was rebuilt and the binary tossed into the pile for everyone to use. Usually they caught the need to rebuild depending packages but sometimes not, and then there would be bugs, messages in the forum, and a dev would realize that "oops, we need to rebuild X for the new Y". Mysterious crashes, strange bugs, frequent "fixes" were the price of avoiding "massive rebuilds". I could live with that, but I know that it's not the FreeBSD way. The only way to guarantee that packages that depend on one another play nice together is to build them together. FreeBSD has the infrastructure for that and keeps "perfect" binary repositories.

The poudriere and synth approach guarantees a "clean" repository. pkg does not necessarily keep the installation that clean. To keep a perfectly consistent installation you either use an even more advanced package manager such as Guix or do a complete re-install for each upgrade (remove all packages, then re-install from the clean repository). When I do build my own repository with poudriere or synth I do a complete re-install for each upgrade. Since the packages are already on the local disk a complete re-install of all packages does not take long, just a couple of minutes.

If you aren't willing to build a clean repository then don't bother with poudriere.
 
Rebuilding 700 packages takes around two days, not minutes
In the winter I can use the CPU-heat, but in the summer it is lost energy.
I have 8G or ram and wired + active memory is 4G leaving at max 4G for builds.
What ccache size would you advice to speed up ?
How do i make sure the cache is used by all building jails.
 
Use ccache. If you still think there's too much time wasted, abandon poudriere, use something simpler (or just manually use the ports tree) and conciously opt for the occassional breakage you will experience by *not* having rebuilt dependent ports ;)
 
Just noticed your edit ...
Alain De Vos said:
What ccache size would you advice to speed up ?
Click to expand...
ccache is on disk, so "ideal" size doesn't depend on your available RAM. IMHO, the ideal size is such that the results of a complete build of your whole package set fits. My package list contains a few huge desktop ports, so I use max_size = 20G
Alain De Vos said:
How do i make sure the cache is used by all building jails.
Click to expand...
You don't have to worry about that, poudriere integrates ccache pretty well. Just set a CCACHE_DIR (containing a ccache.conf) in /usr/local/etc/poudriere.conf and poudriere will use it. Of course, there are ports that can't make use of ccache, either because they use something other than C or C++, or because their build-system is "broken".
 
You can also use swap file given that you don't have so much RAM. It plays well with Poudrière. Ccache is important given your situation. Your buildsystem can take advantage of already built dependencies in cache. A Big swap file md99 will help with swap error owing to lack of ram.
 
The problem with "swapping"' even on SSD, is that at that moment real-time stuff like playing a simple youtube music video stops functioning.
On the danger of nagging, I do not mind compiling "bash" each day, but do not want to compile "rust" or "llvm" each day.
 
It is invaluable for such a small RAM even if you need prevent building rust, llvm, gcc, chromium, firefox, etc. You can also make some reasonable changes to sysctl.conf e.g. hw.page.size.
 
garry said:
The only way to guarantee that packages that depend on one another play nice together is to build them together. FreeBSD has the infrastructure for that and keeps "perfect" binary repositories.
Click to expand...

Alain De Vos said:
On the danger of nagging, I do not mind compiling "bash" each day, but do not want to compile "rust" or "llvm" each day.
Click to expand...
There are a couple possibilities in poudriere.conf... CHECK_CHANGED_DEPS and BAD_PKGNAME_DEPS_ARE_FATAL... in my case, I want to update KDE ports only, and build them against what's already installed. I have ~400 such ports for my whole tree, and need to build ~200 of those for a functional desktop. So I give Poudriere the list with 200 ports, it wants to rebuild 100 more - even though they're already installed, they're the stuff I want to build against!
 
Alain De Vos said:
Rebuilding 700 packages takes around two days, not minutes
In the winter I can use the CPU-heat, but in the summer it is lost energy.
Click to expand...
That's why I decided to rent a VPS for this use. This VPS will also host my HTTP, Mail, and backup servers for my personal use. I found a bigger VPS than the one which currently host my HTTP and Mail servers for a great price ;-)
The annual cost of this VPS will be lower than the electricity bill of my own poudriere build machine at home.
 
Please elaborate , can you specify exactly which VPS at which price.
And then after a repository is build we should be able to transfer it easily to the desktop PC you are running ...
 
You must log in or register to reply here.
Back
Top