HOWTO: Speeding up poudriere build times

Sevendogsbsd

Daemon

Reaction score: 699
Messages: 1,147

Using ports-mgmt/poudriere with devel/ccache, tmpfs and parallel jobs can speed up your package build times. How much depends entirely on your hardware. Keep in mind there are quite a few pages out on the Internet about this so when I did my set up, I pulled from several. My aim for this HOWTO was to pull these together in one place so other folks who use ports-mgmt/poudriere to build ports can benefit from this.

From the Man page (ccache):

"ccache is a compiler cache. It speeds up recompilation by caching the result of previous compilations and detecting when the same compilation is being done again. Supported languages are C, C++, Objective-C and Objective-C++."

The implication here is that the FIRST time you run a ports-mgmt/poudriere build you won't notice any difference in speed using ccache. Subsequent builds will be faster.

Settings for ccache:

Assumptions/Procedure:

  1. Have a running ports-mgmt/poudriere installation (see thread Thread 38859 to set up ports-mgmt/poudriere)
  2. Install devel/ccache on the ports-mgmt/poudriere build machine
    1. pkg install ccache
  3. Configure devel/ccache
    1. Add
      Code:
      CCACHE_DIR=/var/cache/ccache
      to /usr/local/etc/poudriere.conf
Make sure the directory you point this config entry to exists.

Ccache automatically creates its configuration as <ccachedir>/ccache.conf. The only modification I made to mine was to increase the cache size from max_size = 5.0G to
max_size = 10.0G, mainly because my build machine is not used for anything else and I have a lot of disk space.

I also added the following 2 options to /usr/local/etc/poudriere.d/make.conf although I am not sure the devel/ccache directory reference is really needed since it is also in /usr/local/etc/poudriere.conf.

CCACHE_DIR=/var/cache/ccache
WITH_CCACHE_BUILD=yes


That's it. On to the tmps configuration.

Settings for tmpfs:

  1. Configure tmpfs per the FreeBSD Wiki except the /etc/fstab should look like
Code:
tmpfs           /tmp            tmpfs   rw,mode=1777 0 0


Set up ports-mgmt/poudriere to use tmpfs. The setting you choose will depend on how much ram your build system has. The "all" setting uses more ram because the entire build is done in memory. I have 96GB of ram so chose "all".

In /usr/local/etc/poudriere.conf:

Code:
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=all

Other specific /usr/local/etc/poudriere.conf settings:

ccache:
Code:
# ccache support. Supply the path to your ccache cache directory.
# It will be mounted into the jail and be shared among all jails.
# It is recommended that extra ccache configuration be done with
# ccache -o rather than from the environment.
CCACHE_DIR=/var/cache/ccache

Parallel build settings will depend on your hardware: number of CPU cores and amount of ram. The settings below are for 2 6-core Xeons (24 virtual cores) and 96GB of ram so adjust accordingly based on your hardware.

Parallel builds:
Code:
# By default poudriere uses hw.ncpu to determine the number of builders.
# You can override this default by changing PARALLEL_JOBS here, or
# by specifying the -J flag to bulk/testport.
#
# Example to define PARALLEL_JOBS to one single job
# PARALLEL_JOBS=1
PARALLEL_JOBS=6
# How many jobs should be used for preparing the build? These tend to
# be more IO bound and may be worth tweaking. Default: PARALLEL_JOBS * 1.25
# PREPARE_PARALLEL_JOBS=1
PREPARE_PARALLEL_JOBS=36

These settings are for my specific port requirements so adjust accordingly. The "*" by some ports is because the version is concatenated to the port name, e.g., llvm60.

Priority boost:
Code:
# Define pkgname globs to boost priority for
# Default: none
PRIORITY_BOOST="llvm* rust chromium"

My build times with none of these configurations and building ~55 ports, which ripples out to ~700 ports including dependencies, went from around 11 hours to around 2-3 hours, even after running poudriere pkgclean -A -j JAILNAME.

I would like to thank vermaden for inspiring me to write this HOWTO and rigoletto@ and SirDice for some of the setting suggestions. I need to also thank kpa for the original ports-mgmt/poudriere HOWTO.

If anyone sees any mistakes or omissions in this HOWTO, please bring them to my attention so I can correct them.
 
Last edited:

abishai

Aspiring Daemon

Reaction score: 196
Messages: 782

I compile my ports in bhyve machine with 6Gb memory and 6 X5680 cores. Any attempt to use tmpfs results OOM for big projects and I don't want to waste more than 6Gb just to occasionally build things. So, usually I build my 50 ports in 17 hours.
 

Oleg_NYC

Member

Reaction score: 5
Messages: 85

You didn't mention the ALLOW_MAKE_JOBS setting. If this setting is not enabled, only one job will be used to build each port.
 

Zirias

Son of Beastie

Reaction score: 1,790
Messages: 3,073

You didn't mention the ALLOW_MAKE_JOBS setting. If this setting is not enabled, only one job will be used to build each port.
Which is what you want most of the time, as poudriere already builds [ncpu] packages in parallel, so there's nothing to gain from having a lot of tasks waiting and competing for CPU time.

There are exceptions with huge ports that have a lot of dependent ports, so it can happen that a bulk run ends up building only a single port on a single core while the queue must wait for that to finish. If you experience this problem, it might be a good idea to put this specific huge port in ALLOW_MAKE_JOBS_PACKAGES, similar to the already mentioned PRIORITY_BOOST.
 

Jose

Daemon

Reaction score: 1,169
Messages: 1,389

There are exceptions with huge ports that have a lot of dependent ports, so it can happen that a bulk run ends up building only a single port on a single core while the queue must wait for that to finish...
This happened to me with llvm enough times that I enabled ALLOW_MAKE_JOBS. I haven't noticed any downsides yet.
 

Zirias

Son of Beastie

Reaction score: 1,790
Messages: 3,073

This happened to me with llvm enough times that I enabled ALLOW_MAKE_JOBS. I haven't noticed any downsides yet.
You will notice if you want the machine to do anything other than building at the same time, see also my recent thread asking about idprio ;) in a nutshell: idprio is the way to go here, but as in-kernel tasks are still scheduled with higher priority, there's still a chance to impair other tasks on the machine (to the point when they're reacting slow as hell). In my experience, having as many build tasks as there are cores/threads (the default with poudriere) will, when they're set to idprio, not have any noticeable effect, the machine can be used normally for other things.

Also, at least in theory, performance with more tasks is a little bit lower, because more tasks competing for the available CPUs means more frequent context switches. Of course you won't be able to even measure that.

As a compromise, ALLOW_MAKE_JOBS_PACKAGES is a way to selectively work around the queue waiting for *one* huge port, while still limiting the number of concurrent jobs to the number of available cores/threads *most* of the time.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 13,156
Messages: 39,777

it might be a good idea to put this specific huge port in ALLOW_MAKE_JOBS_PACKAGES
I have this:
Code:
ALLOW_MAKE_JOBS_PACKAGES="pkg llvm* gcc* node* *webengine rust* firefox* mame mess"
But I may need to revise that list. My build server has spontaneously rebooted during a large build run, this happened a couple of times now. I suspect building one of these sucks up too much resources when there are still other jobs running and the whole thing just blows up. It could also be due to hardware errors, this board, CPU and memory is a couple of years old now. Last summer's heat killed off a few hard disks too. I really need to run a memory test to make sure it's not the cause of the reboots.
 

Oleg_NYC

Member

Reaction score: 5
Messages: 85

Unlike other ports that I build with poudriere, firefox refuses to use ccache. I think it relies on the --without-ccache option that I see in the poudriere log file for firefox.
 
OP
Sevendogsbsd

Sevendogsbsd

Daemon

Reaction score: 699
Messages: 1,147

Interesting - wonder why it wouldn't use this. I was thinking it might be toggled in the port options but does not appear to be available.
Code:
===> The following configuration options are available for firefox-76.0_1,1:
     CANBERRA=off: Sound theme alerts
     DBUS=on: D-Bus IPC system support
     DEBUG=off: Build with debugging support
     FFMPEG=on: FFmpeg support (WMA, AIFF, AC3, APE...)
     GCONF=on: GConf configuration backend support
     LIBPROXY=off: Proxy support via libproxy
     OPTIMIZED_CFLAGS=on: Use extra compiler optimizations
     PROFILE=on: Build with profiling support
     TEST=off: Build and/or run tests
====> Options available for the group AUDIO
     ALSA=on: ALSA audio architecture support
     JACK=on: JACK audio server support
     PULSEAUDIO=on: PulseAudio sound server support
     SNDIO=on: Sndio audio support
===> Use 'make config' to modify these settings
 

Oleg_NYC

Member

Reaction score: 5
Messages: 85

In the log file for firefox, I see this part:
WITH_CCACHE_BUILD=yes
CCACHE_DIR=/root/.ccache

I also see this part:
===> firefox-76.0_1,1 depends on file: /usr/local/bin/ccache - not found
===> Installing existing package /packages/All/ccache-3.7.1_1.txz

I see this part:
===> firefox-76.0_1,1 depends on file: /usr/local/bin/ccache - found

However, then I see this part:
js/src> running /wrkdirs/usr/ports/www/firefox/work/firefox-76.0/configure.py --enable-project=js --enable-gconf --disable-install-strip --disable-libproxy --enable-official-branding --enable-startup-notification --disable-strip --enable-system-pixman --disable-updater --prefix=/usr/local --with-intl-api --with-system-bz2 --with-system-icu --with-system-libevent --with-system-nss --with-system-png=/usr/local --with-system-zlib --host=x86_64-unknown-freebsd13.0 --target=x86_64-unknown-freebsd13.0 --disable-tests --disable-debug --disable-rust-debug --enable-release --enable-optimize --without-ccache

As you can see, the last part has this --without-ccache thing. When I run ccache -s, I see that nothing gets updated.
 
OP
Sevendogsbsd

Sevendogsbsd

Daemon

Reaction score: 699
Messages: 1,147

Interesting. I have a Firefox update to do tonight as well. I use ports-mgmt/synth though but I should be able to see logs for the build. I'll check to see if I notice the same thing.
 

chrbr

Aspiring Daemon

Reaction score: 382
Messages: 912

How much cache is required nowadays? In the past I have played around with devel/ccache without any problems. But with things as devel/llvm getting huge I found that compiling such a port caused filling and cleanup of the cache multiple times. See ccache(1) and the query options. And caching makes only sense if there is some old content to re-use.

Currently I am happy using packages and continue to do so. But I am still interested in building ports and the required effort because it is good to be able to build from ports, just in case. devel/poudriere just worked fine for me beside considering the required computing resources.
 
OP
Sevendogsbsd

Sevendogsbsd

Daemon

Reaction score: 699
Messages: 1,147

I don't know how much is required - the synth docs suggested 15G and I have plenty of space so set mine to that. I'll could check the build time again tonight to see if faster but I would really need to remove everything and do a full build over again to see if there is a decrease in time. Not willing to do that because every time I try, I break something 🤣. Incremental updates work fine.

I like ports because I use audio/sndio and I uncheck the "sub shell" support in misc/mc. Other than that, I don't really need to, just sort of fun for me to do.
 

chrbr

Aspiring Daemon

Reaction score: 382
Messages: 912

Dear Sevendogsbsd,
Thank you for the reply! I think the output of ccache -s before and after a new build of the "big ports" would be interesting. If it is about just to have a cache of 15G - this is nothing to talk about nowadays. If I remember correctly I have had a cache of 2G or 5G. But with compiling devel/llvm or similar the cache has been cleaned up multiple times during compilation. Then the cache should not be useful anymore but an additional load.
 
OP
Sevendogsbsd

Sevendogsbsd

Daemon

Reaction score: 699
Messages: 1,147

I did get a huge decrease in build time using poudriere when I had a build server - I got a suggestion from rigoletto(?) to clean things out and do the build again after implementing ccache to see the difference. It went down from like 11 hours to about 3. This was on an older Xeon though with SAS drives. My i7 quad with SSD drives seems to do pretty well actually. It's a newer architecture, faster RAM, etc so that is to be expected I guess.
 

Zirias

Son of Beastie

Reaction score: 1,790
Messages: 3,073

Just some quick thoughts about firefox:
  • To use ccache, one aspect is "wrapping" compiler calls -- this can be done with most build systems by setting CC and/or CXX environment variables appropriately. Without having a deeper look here, if firefox provides a --with-ccache knob on configure, this is probably meant for users compiling it manually to use ccache here. As poudriere offers a "global" ccache usage, it makes sense to disable this knob, so it doesn't interfere.
  • firefox uses quite some rust code, which won't profit from ccache. If rust will be "trending", ccache should support it sooner or later.
 

chrbr

Aspiring Daemon

Reaction score: 382
Messages: 912

The build time decreases when it is about rebuilding small ports or just one big port presumed everything fits in the cache. But if devel/llvm or more consume each the complete the cache each time things get time consuming. There is no re-use anymore. May be I should just give it a try wih a larger cache, just to see what happens.
 

Zirias

Son of Beastie

Reaction score: 1,790
Messages: 3,073

If a single poudriere run for your complete package-list "overflows" the cache, yes, then it's time to increase its size. For my list of packages, covering all my desktop and server needs, it currently looks like this:
Code:
builder# CCACHE_DIR=/usr/local/poudriere/data/ccache ccache -s 
cache directory                     /usr/local/poudriere/data/ccache
primary config                      /usr/local/poudriere/data/ccache/ccache.conf
secondary config      (readonly)    /usr/local/etc/ccache.conf
stats updated                       Fri May  1 09:11:16 2020
cache hit (direct)               4312942
cache hit (preprocessed)          528479
cache miss                       1708920
cache hit rate                     73.91 %
called for link                   748370
called for preprocessing          278415
multiple source files                362
compiler produced stdout             621
compiler produced no output           27
compiler produced empty output       353
compile failed                     69989
ccache internal error                895
preprocessor error                 55176
can't use precompiled header       16177
cache file missing                 14548
bad compiler arguments             17710
unsupported source language          337
autoconf compile/link             334619
unsupported compiler option         7046
unsupported code directive           179
no input file                     118996
cleanups performed                  1933
files in cache                    725942
cache size                          18.2 GB
max cache size                      20.0 GB
 
Top