Simultaneous compilation of browsers without regard to PRIORITY_BOOST settings

I am compiling Firefox and Chromium browsers using poudriere:
Code:
poudriere bulk -j $jail -p $port -z $s -f pkglist/mypackages
where 'mypackages' contains:
Code:
www/chromium
www/firefox
The problem is that browsers, which are heavy packages to compile, compile at the same time. This happens regardless of how the PRIORITY_BOOST option is configured:

var1 - chromium + firefox:
Code:
PRIORITY_BOOST="chromium firefox emacs-devel gcc* ghc llvm* rust"
or var2 - firefox only:
Code:
PRIORITY_BOOST="firefox emacs-devel gcc* ghc llvm* rust"
As a result, compilation of browsers occurs simultaneously. I can clearly see this in the logs in the poudriere's web interface. The machine has an E5-2420 cpu (6 cores, 12 threads) and 24Gb of RAM, and this load is too high. As a result of simultaneous compilation, this build fails.

poudriere.conf (please critique this config)
Code:
ALLOW_MAKE_JOBS=yes
BASEFS=/build
BUILDER_HOSTNAME=pkg.example.com
CCACHE_DIR=/build/ccache
CHECK_CHANGED_DEPS=yes
CHECK_CHANGED_OPTIONS=verbose
DISTFILES_CACHE=/build/distfiles
FREEBSD_HOST=https://download.freebsd.org
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=4
NOLINUX=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
POUDRIERE_DATA=${BASEFS}/data
PRIORITY_BOOST="chromium firefox emacs-devel gcc* ghc llvm* rust"
RESOLV_CONF=/etc/resolv.conf
USE_COLORS=yes
USE_PORTLINT=no
USE_TMPFS=yes
ZPOOL=mypool
ZROOTFS=/poudriere

Does anyone have any ideas how to make it so that 1 of any browser is compiled first, and then the second, in turn?
 
Last edited:
Priority boost would hardly do anything in this case because those packages are leaves, they are at the end of the line. A way to workaround it is building the browsers separated --- after the other ports, and so you would need to remove them from your the list:

poudriere bulk -r -J 1:12 -j $jail www/chromium www/firefox

This command will build one port per time while using 12 threads. I use this method to build compilers (specially because of Rust) first.

Btw, this is not the load that make the build to fail but you run out of memory, including SWAP --- likely due to Chromimum.
 
Number of 'CPU-cores' and number of 'GB free memory' set a maximum for 'the number of parallel jobs' you should use.
For instance i have 8GB free memory and 8 cores but the number of parallel jobs i use is only 5.
Note, compilation of a port can take upto 2GB free memory and you don't want too much swapping.
Offcourse when all swap is used things get killed.
 
  • Thanks
Reactions: dnb
A way to workaround it is building the browsers separated --- after the other ports
Very interesting! Thank you! First, I use script not only with poudriere bulk, but also the poudriere pkgclean operation to clean packages after poudriere bulk. So right now (before fixes) it looks like this:
Code:
poudriere bulk -j $jail -p $port -z $s -f all_pkg_WITH_browsers
poudriere pkgclean -y -j $jail -p $port -z $s -f all_pkg_WITH_browsers
Step 1. After intial fixes (using files and the -f flag):
Code:
poudriere bulk -r -J 1:12 -j $jail -p $port -z $s -f my_pkg_WITHOUT_browsers
poudriere bulk -r -J 1:12 -j $jail -p $port -z $s -f firefox_only -f chromium_only
poudriere pkgclean -y -j $jail -p $port -z $s -f my_pkg_without_browsers -f firefox_only -f chromium_only
Here I'm using files because I'm afraid of losing the packages with the built browsers during the poudriere pkgclean operation. As a result, PRIORITY_BOOST becomes like this:
Code:
PRIORITY_BOOST="emacs-devel gcc* ghc llvm* rust"
I use this method to build compilers (specially because of Rust) first.
Step 2. Following your idea of building compilers before all other builds (if I understand correctly), I should put the script in this form (please critique):
Code:
# build compilers first
poudriere bulk -r -J 1:12 -j $jail -p $port -z $s -f compilers_only

# build various small packages
poudriere bulk -r -J 1:12 -j $jail -p $port -z $s -f my_pkg_WITHOUT_browsers

# do the same with Emacs, since it will require GCC (build with libgccjit), this is a relatively heavy build.
poudriere bulk -r -J 1:12 -j $jail -p $port -z $s -f emacs_with_gcc

# compile chromium and firefox strictly in turn
poudriere bulk -r -J 1:12 -j $jail -p $port -z $s -f firefox_only -f chromium_only

# clean up after ourselves, but do not remove compiled packages
poudriere pkgclean -y -j $jail -p $port -z $s \
        -f compilers_only \
        -f my_pkg_WITHOUT_browsers \
        -f emacs_with_gcc \
        -f firefox_only \
        -f chromium_only

And PRIORITY_BOOST becomes like this:
Code:
PRIORITY_BOOST="ghc"
Btw, this is not the load that make the build to fail but you run out of memory, including SWAP --- likely due to Chromimum.
I should have said this right away, but I don't have SWAP. The fact is that this server has zfs pool, which is configured as a 3-way mirror of 3 SAS disks (7200 rpm). The server is designed in such a way that it is problematic to add any other devices suitable for SWAP (like SSD, NVME, etc.) there.

This pool is loaded using zfsboot, that is, I don’t even have a partition table, and I have an empty fstab. Accordingly, I could use SWAP on zvol, but as far as I know, this is not recommended (this is a debatable issue).
 
My 5-cent. If builder jails get killed just reduce the number of parallel builds. (for all your builds)
Priority-boosts is not needed or good for nothing.
 
  • Thanks
Reactions: dnb
My 5-cent. If builder jails get killed just reduce the number of parallel builds. (for all your builds)
Priority-boosts is not needed or good for nothing.
Perhaps this is the most prudent decision. Let's say I just remove ALLOW_MAKE_JOBS=yes and PRIORITY_BOOST from the config in the first post. And I add PARALLEL_JOBS=6 there (instead of 12 jobs). This removes problems with lack of memory.

On the other hand, these fixes do not preclude fine-grained control over the order in which packages are installed (compilers first, browsers last, etc.) as rigoletto suggested above.
 
Back
Top