HOWTO: Speeding up poudriere build times

Using ports-mgmt/poudriere with devel/ccache, tmpfs and parallel jobs can speed up your package build times. How much depends entirely on your hardware. Keep in mind there are quite a few pages out on the Internet about this so when I did my set up, I pulled from several. My aim for this HOWTO was to pull these together in one place so other folks who use ports-mgmt/poudriere to build ports can benefit from this.

From the Man page (ccache):

"ccache is a compiler cache. It speeds up recompilation by caching the result of previous compilations and detecting when the same compilation is being done again. Supported languages are C, C++, Objective-C and Objective-C++."

The implication here is that the FIRST time you run a ports-mgmt/poudriere build you won't notice any difference in speed using ccache. Subsequent builds will be faster.

Settings for ccache:

Assumptions/Procedure:

  1. Have a running ports-mgmt/poudriere installation (see thread Thread 38859 to set up ports-mgmt/poudriere)
  2. Install devel/ccache on the ports-mgmt/poudriere build machine
    1. pkg install ccache
  3. Configure devel/ccache
    1. Add
      Code:
      CCACHE_DIR=/var/cache/ccache
      to /usr/local/etc/poudriere.conf
Make sure the directory you point this config entry to exists.

Ccache automatically creates its configuration as <ccachedir>/ccache.conf. The only modification I made to mine was to increase the cache size from max_size = 5.0G to
max_size = 10.0G, mainly because my build machine is not used for anything else and I have a lot of disk space.

I also added the following 2 options to /usr/local/etc/poudriere.d/make.conf although I am not sure the devel/ccache directory reference is really needed since it is also in /usr/local/etc/poudriere.conf.

CCACHE_DIR=/var/cache/ccache
WITH_CCACHE_BUILD=yes


That's it. On to the tmps configuration.

Settings for tmpfs:

  1. Configure tmpfs per the FreeBSD Wiki except the /etc/fstab should look like
Code:
tmpfs           /tmp            tmpfs   rw,mode=1777 0 0


Set up ports-mgmt/poudriere to use tmpfs. The setting you choose will depend on how much ram your build system has. The "all" setting uses more ram because the entire build is done in memory. I have 96GB of ram so chose "all".

In /usr/local/etc/poudriere.conf:

Code:
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=all

Other specific /usr/local/etc/poudriere.conf settings:

ccache:
Code:
# ccache support. Supply the path to your ccache cache directory.
# It will be mounted into the jail and be shared among all jails.
# It is recommended that extra ccache configuration be done with
# ccache -o rather than from the environment.
CCACHE_DIR=/var/cache/ccache

Parallel build settings will depend on your hardware: number of CPU cores and amount of ram. The settings below are for 2 6-core Xeons (24 virtual cores) and 96GB of ram so adjust accordingly based on your hardware.

Parallel builds:
Code:
# By default poudriere uses hw.ncpu to determine the number of builders.
# You can override this default by changing PARALLEL_JOBS here, or
# by specifying the -J flag to bulk/testport.
#
# Example to define PARALLEL_JOBS to one single job
# PARALLEL_JOBS=1
PARALLEL_JOBS=6
# How many jobs should be used for preparing the build? These tend to
# be more IO bound and may be worth tweaking. Default: PARALLEL_JOBS * 1.25
# PREPARE_PARALLEL_JOBS=1
PREPARE_PARALLEL_JOBS=36

These settings are for my specific port requirements so adjust accordingly. The "*" by some ports is because the version is concatenated to the port name, e.g., llvm60.

Priority boost:
Code:
# Define pkgname globs to boost priority for
# Default: none
PRIORITY_BOOST="llvm* rust chromium"

My build times with none of these configurations and building ~55 ports, which ripples out to ~700 ports including dependencies, went from around 11 hours to around 2-3 hours, even after running poudriere pkgclean -A -j JAILNAME.

I would like to thank vermaden for inspiring me to write this HOWTO and rigoletto@ and SirDice for some of the setting suggestions. I need to also thank kpa for the original ports-mgmt/poudriere HOWTO.

If anyone sees any mistakes or omissions in this HOWTO, please bring them to my attention so I can correct them.
 
Last edited:
I compile my ports in bhyve machine with 6Gb memory and 6 X5680 cores. Any attempt to use tmpfs results OOM for big projects and I don't want to waste more than 6Gb just to occasionally build things. So, usually I build my 50 ports in 17 hours.
 
You didn't mention the ALLOW_MAKE_JOBS setting. If this setting is not enabled, only one job will be used to build each port.
 
You didn't mention the ALLOW_MAKE_JOBS setting. If this setting is not enabled, only one job will be used to build each port.
Which is what you want most of the time, as poudriere already builds [ncpu] packages in parallel, so there's nothing to gain from having a lot of tasks waiting and competing for CPU time.

There are exceptions with huge ports that have a lot of dependent ports, so it can happen that a bulk run ends up building only a single port on a single core while the queue must wait for that to finish. If you experience this problem, it might be a good idea to put this specific huge port in ALLOW_MAKE_JOBS_PACKAGES, similar to the already mentioned PRIORITY_BOOST.
 
There are exceptions with huge ports that have a lot of dependent ports, so it can happen that a bulk run ends up building only a single port on a single core while the queue must wait for that to finish...
This happened to me with llvm enough times that I enabled ALLOW_MAKE_JOBS. I haven't noticed any downsides yet.
 
This happened to me with llvm enough times that I enabled ALLOW_MAKE_JOBS. I haven't noticed any downsides yet.
You will notice if you want the machine to do anything other than building at the same time, see also my recent thread asking about idprio ;) in a nutshell: idprio is the way to go here, but as in-kernel tasks are still scheduled with higher priority, there's still a chance to impair other tasks on the machine (to the point when they're reacting slow as hell). In my experience, having as many build tasks as there are cores/threads (the default with poudriere) will, when they're set to idprio, not have any noticeable effect, the machine can be used normally for other things.

Also, at least in theory, performance with more tasks is a little bit lower, because more tasks competing for the available CPUs means more frequent context switches. Of course you won't be able to even measure that.

As a compromise, ALLOW_MAKE_JOBS_PACKAGES is a way to selectively work around the queue waiting for *one* huge port, while still limiting the number of concurrent jobs to the number of available cores/threads *most* of the time.
 
it might be a good idea to put this specific huge port in ALLOW_MAKE_JOBS_PACKAGES
I have this:
Code:
ALLOW_MAKE_JOBS_PACKAGES="pkg llvm* gcc* node* *webengine rust* firefox* mame mess"
But I may need to revise that list. My build server has spontaneously rebooted during a large build run, this happened a couple of times now. I suspect building one of these sucks up too much resources when there are still other jobs running and the whole thing just blows up. It could also be due to hardware errors, this board, CPU and memory is a couple of years old now. Last summer's heat killed off a few hard disks too. I really need to run a memory test to make sure it's not the cause of the reboots.
 
Unlike other ports that I build with poudriere, firefox refuses to use ccache. I think it relies on the --without-ccache option that I see in the poudriere log file for firefox.
 
Interesting - wonder why it wouldn't use this. I was thinking it might be toggled in the port options but does not appear to be available.
Code:
===> The following configuration options are available for firefox-76.0_1,1:
     CANBERRA=off: Sound theme alerts
     DBUS=on: D-Bus IPC system support
     DEBUG=off: Build with debugging support
     FFMPEG=on: FFmpeg support (WMA, AIFF, AC3, APE...)
     GCONF=on: GConf configuration backend support
     LIBPROXY=off: Proxy support via libproxy
     OPTIMIZED_CFLAGS=on: Use extra compiler optimizations
     PROFILE=on: Build with profiling support
     TEST=off: Build and/or run tests
====> Options available for the group AUDIO
     ALSA=on: ALSA audio architecture support
     JACK=on: JACK audio server support
     PULSEAUDIO=on: PulseAudio sound server support
     SNDIO=on: Sndio audio support
===> Use 'make config' to modify these settings
 
In the log file for firefox, I see this part:
WITH_CCACHE_BUILD=yes
CCACHE_DIR=/root/.ccache

I also see this part:
===> firefox-76.0_1,1 depends on file: /usr/local/bin/ccache - not found
===> Installing existing package /packages/All/ccache-3.7.1_1.txz

I see this part:
===> firefox-76.0_1,1 depends on file: /usr/local/bin/ccache - found

However, then I see this part:
js/src> running /wrkdirs/usr/ports/www/firefox/work/firefox-76.0/configure.py --enable-project=js --enable-gconf --disable-install-strip --disable-libproxy --enable-official-branding --enable-startup-notification --disable-strip --enable-system-pixman --disable-updater --prefix=/usr/local --with-intl-api --with-system-bz2 --with-system-icu --with-system-libevent --with-system-nss --with-system-png=/usr/local --with-system-zlib --host=x86_64-unknown-freebsd13.0 --target=x86_64-unknown-freebsd13.0 --disable-tests --disable-debug --disable-rust-debug --enable-release --enable-optimize --without-ccache

As you can see, the last part has this --without-ccache thing. When I run ccache -s, I see that nothing gets updated.
 
How much cache is required nowadays? In the past I have played around with devel/ccache without any problems. But with things as devel/llvm getting huge I found that compiling such a port caused filling and cleanup of the cache multiple times. See ccache(1) and the query options. And caching makes only sense if there is some old content to re-use.

Currently I am happy using packages and continue to do so. But I am still interested in building ports and the required effort because it is good to be able to build from ports, just in case. devel/poudriere just worked fine for me beside considering the required computing resources.
 
I don't know how much is required - the synth docs suggested 15G and I have plenty of space so set mine to that. I'll could check the build time again tonight to see if faster but I would really need to remove everything and do a full build over again to see if there is a decrease in time. Not willing to do that because every time I try, I break something 🤣. Incremental updates work fine.

I like ports because I use audio/sndio and I uncheck the "sub shell" support in misc/mc. Other than that, I don't really need to, just sort of fun for me to do.
 
Dear Sevendogsbsd,
Thank you for the reply! I think the output of ccache -s before and after a new build of the "big ports" would be interesting. If it is about just to have a cache of 15G - this is nothing to talk about nowadays. If I remember correctly I have had a cache of 2G or 5G. But with compiling devel/llvm or similar the cache has been cleaned up multiple times during compilation. Then the cache should not be useful anymore but an additional load.
 
I did get a huge decrease in build time using poudriere when I had a build server - I got a suggestion from rigoletto(?) to clean things out and do the build again after implementing ccache to see the difference. It went down from like 11 hours to about 3. This was on an older Xeon though with SAS drives. My i7 quad with SSD drives seems to do pretty well actually. It's a newer architecture, faster RAM, etc so that is to be expected I guess.
 
Just some quick thoughts about firefox:
  • To use ccache, one aspect is "wrapping" compiler calls -- this can be done with most build systems by setting CC and/or CXX environment variables appropriately. Without having a deeper look here, if firefox provides a --with-ccache knob on configure, this is probably meant for users compiling it manually to use ccache here. As poudriere offers a "global" ccache usage, it makes sense to disable this knob, so it doesn't interfere.
  • firefox uses quite some rust code, which won't profit from ccache. If rust will be "trending", ccache should support it sooner or later.
 
The build time decreases when it is about rebuilding small ports or just one big port presumed everything fits in the cache. But if devel/llvm or more consume each the complete the cache each time things get time consuming. There is no re-use anymore. May be I should just give it a try wih a larger cache, just to see what happens.
 
If a single poudriere run for your complete package-list "overflows" the cache, yes, then it's time to increase its size. For my list of packages, covering all my desktop and server needs, it currently looks like this:
Code:
builder# CCACHE_DIR=/usr/local/poudriere/data/ccache ccache -s 
cache directory                     /usr/local/poudriere/data/ccache
primary config                      /usr/local/poudriere/data/ccache/ccache.conf
secondary config      (readonly)    /usr/local/etc/ccache.conf
stats updated                       Fri May  1 09:11:16 2020
cache hit (direct)               4312942
cache hit (preprocessed)          528479
cache miss                       1708920
cache hit rate                     73.91 %
called for link                   748370
called for preprocessing          278415
multiple source files                362
compiler produced stdout             621
compiler produced no output           27
compiler produced empty output       353
compile failed                     69989
ccache internal error                895
preprocessor error                 55176
can't use precompiled header       16177
cache file missing                 14548
bad compiler arguments             17710
unsupported source language          337
autoconf compile/link             334619
unsupported compiler option         7046
unsupported code directive           179
no input file                     118996
cleanups performed                  1933
files in cache                    725942
cache size                          18.2 GB
max cache size                      20.0 GB
 
If one is using devel/ccache with ports-mgmt/poudriere, I found it helpful to set hash_dir = false in ccache.conf. The default for this configuration variable changed with version 3.3 to hash_dir = true. This caused unexpectedly poor cache hit performance for me. Since poudriere creates jails with paths with a different index for each cpu core, each recompilation has a 1/ncpu chance of finding a hit. I believe you would also experience cache inflation as well if the paths are hashed.

According to the ccache(1) documentation the downside of setting hash_dir = false is in debugging.

After making this configuration change my cache hit rate increased from an historical 10% +/- 0.5 to greater than 31% in just two runs without resetting the statistics. I imagine resetting the stats will result in a higher hit rate for future runs.

Use ccache -p to check your current config.
 
If memory serves me, I think I left the ccache alone for my Poudriere setup... I have a rig with 32 GB RAM (and a Ryzen 5 1400). Yeah, compilation can go slow, but I plan for that by doing overnight jobs.

I do like the fact that Poudriere can pick the job right back up even after a reboot, without missing a beat.

I'd probably want to avoid changing too many details in my Poudriere setup if I were to verify that it was in fact ccache and not something else responsible for the dramatic decrease in compilation times. Like same hardware, same sources, same number of jobs/threads, etc. Cleaning out the resulting packages with poudriere pkgclean -A -j JAILNAME is not quite enough, you also need something similar to ( make clean that you run in /usr/ports/). If you don't run make clean, there won't be any re-compilation really done... Yeah, the processor will again work hard to re-run the compilation after make clean, because all the intermediate files in the work/ subdirectory will be gone. But that's kind of what it takes to see if you really have a dramatic decrease in compilation times.
 
Back
Top