Poudriere conf

Alain De Vos

Alain De Vos

Configuring poudriere is a bit of a dark art , because you are always limited by
1. Compilation time
2. Free Memory (& tune ARC cache)
3. Swap space (& No dying , being to busy swapping, or running out of it)
4. Number of cores

My current config on 16GB PC with 12 cores.
Note :
- I will later I increase TMPFS_LIMIT as some extractions fail.
- I have USE_TMPF="wrkdir" , meaning log are written to disk.,as the builder jails. But .o files are kept in memory.
- The limited number of builders I use. So few builders , but build in memory ,ie use CPU & ALLOW_MAKE_JOBS=yes

My workflow. Compile with TMPFS_LIMIT=3 ; repeat 4 ; repeat 5 until all done.

Code: 
ZPOOL=SSD
ZROOTFS=/usr/local/poudriere
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS="wrkdir"
DISTFILES_CACHE=/usr/ports/distfiles
BUILDS=2
BUILDERS=2
PARALLEL_JOBS=2
PREPARE_PARALLEL_JOBS=2
MAKE_JOBS_NUMBER=7
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
TIMESTAMP_LOGS=yes
BUILDER_HOSTNAME=myfreebsd
USE_COLORS=yes
URL_BASE=http://127.0.0.1/poudriere/
PACKAGE_FETCH_BRANCH=quarterly
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
#File Storage
TMPFS_LIMIT=3
#CC usage
MAX_MEMORY=4

#ALLOW_MAKE_JOBS_PACKAGES="llvm* gcc* rust* waterfox firefox*"
#MAX_FILES=8192

Feel free to elaborate on your experience. Alot depends on your hardware.
 
I'm spoiled. My current build server is a dual Xeon E5-2667 with 192 GB of memory. USE_TMPFS=all, PARALLEL_JOBS=6 and ALLOW_MAKE_JOBS=yes. This thing is slicing through my package list like a hot knife through butter. Especially compared to my previous build "server" (Core i5-3470; 16GB).
 
On the other end, keeping a lower end machine is an intellectual interesting thing. It learns me alot about poudriere conf. Because , not everything is for free. There is even fun in it. Also note some use poudriere even on Raspberry.
Until now i was able to compile everything except "zed" which need more than 16GB according to maintainer.
90% of time i use only neovim-qt & firefox. So in background , i let it take its time.
 
SirDice i found an interesting test, let it build then do,

Code: 
cat ./ssd.test 
fio --name=ssd_true_speed \
    --directory=/usr/local/poudriere/data \
    --size=50G \
    --rw=write \
    --bs=1M \
    --direct=1 \
    --ioengine=posixaio \
    --refill_buffers \
    --buffer_compress_percentage=0 \
    --runtime=300 \
    --group_reporting
 
output,
Code: 
root@myfreebsd:/SSD/home/x # ./ssd.test
ssd_true_speed: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=1
fio-3.40
Starting 1 process
ssd_true_speed: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=112MiB/s][w=112 IOPS][eta 00m:00s]
ssd_true_speed: (groupid=0, jobs=1): err= 0: pid=30418: Tue Mar 31 11:27:42 2026
  write: IOPS=141, BW=142MiB/s (148MB/s)(41.5GiB/300007msec); 0 zone resets
    slat (nsec): min=658, max=6543.6k, avg=5687.03, stdev=56963.00
    clat (nsec): min=651, max=109121k, avg=6961824.38, stdev=3635534.07
     lat (usec): min=69, max=109124, avg=6967.51, stdev=3635.22
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[ 5211], 10.00th=[ 5473], 20.00th=[ 5800],
     | 30.00th=[ 5997], 40.00th=[ 6194], 50.00th=[ 6456], 60.00th=[ 6652],
     | 70.00th=[ 6849], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[11076],
     | 99.00th=[23725], 99.50th=[26346], 99.90th=[47449], 99.95th=[51643],
     | 99.99th=[69731]
   bw (  KiB/s): min=16384, max=2195456, per=100.00%, avg=145028.49, stdev=91652.90, samples=599
   iops        : min=   16, max= 2144, avg=141.63, stdev=89.50, samples=599
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 100=0.55%, 250=0.27%, 500=1.35%
  lat (usec)   : 750=0.09%, 1000=0.04%
  lat (msec)   : 2=0.34%, 4=0.65%, 10=90.11%, 20=5.11%, 50=1.41%
  lat (msec)   : 100=0.05%, 250=0.01%
  cpu          : usr=1.47%, sys=0.08%, ctx=43719, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,42474,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=142MiB/s (148MB/s), 142MiB/s-142MiB/s (148MB/s
Only 140MB/s ,rest is marketing cache,one sees in speed tests.

Intenso SSD , technical details,
Sequential writing: Up to 500 MB/s*
 
My experience:

I made some tests end of last year to check if 64GB were enough due the actual indecent price of the ram.

Hardware: amd ryzen 16 cores / 48GB memory (ddr4) / 16GB swap / ssd
built all from port (about 1300 packages) and had always some failures and restarted again...

I did not wanted to restrict poudriere (too much) so I first tincreased the ram to 64GB
mem_64_swap_full.png

I still had some failures!!! (swap full)

So, I increased the swap to 48GB and no failure after this. Maybe 24GB or 32GB would be enouhg, I just wanted to be sure.
My config:
Code: 
ZPOOL=zroot
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=yes
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/var/cache/ccache
PARALLEL_JOBS=12
PREPARE_PARALLEL_JOBS=15
ALLOW_MAKE_JOBS=yes
ALLOW_MAKE_JOBS_PACKAGES="pkg llvm* gcc* node* *webengine firefox* rust* ccache py*"
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=3
PRESERVE_TIMESTAMP=yes
DETERMINE_BUILD_FAILURE_REASON=yes
PACKAGE_FETCH_URL=file:///usr/local/poudriere/data/packages/amd64-current-current
 
Building is usually a tradeoff between memory, CPU and storage.
CPU because the CPU does the actual work
Memory because stuff has to be in memory for the CPU to do the work
Storage, because storage speed/bandwidth gates how quickly artifacts get stored.

I don't have any specifics because I don't do poudiere builds, but building in general:
Max out your memory. The more you have the better the results.
Storage BW: faster storage lets the process save artifacts and get onto the next thing quicker.
CPU: More cores, better clock speed means "more things can happen at the same time" BUT builds have dependencies and kicking off more things just to have them wait on other completions, typically blocks downstream.

For all my systems, from build machines to user desktops, I've always settled on max memory (at highest spec mobo supports) and a CPU one generation back from "latest and greatest". Storage as fast as feasible given BW and size tradeoffs. I'm fine with taking a little longer vs builds breaking.

Builds, especially things like "rebuild entire ports tree" is one of the best stress tests on hardware you can have.

Above is purely my opinion, not related to anything other than my experience and needs.
If I were "the foundation" and had unlimited resources, I would spec build systems with maximum amount of the fastest RAM, the fastest CPU with the most cores for parallel stuff and the highest BW to the fastest storage devices.
But that costs $$$$$
 
My build machine is a VM with 24092MB RAM allocated on an 8-core AMD Ryzen. I build in RAM for speed and to save wear on the SSD.

My poudriere.conf:
Code: 
NO_ZFS=yes
FREEBSD_HOST=ftp://ftp.uk.FreeBSD.org/
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS=all
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=yes
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/var/cache/ccache
NOLINUX=yes

Everything else is default.
 
You must log in or register to reply here.
Back
Top