Solved tmpfs for building ports

cortexmancer

cortexmancer

Hello! After a long time having painful periods of waiting long hours to build big ports I had discovered ccache, and now I have discovered tmpfs.

I do have 32GB of ram, and at maximum I am able to use 15/16GB when building a port, and it keeps at 8-10GB when using normally. I think this big usage is due to zfs, but when I checked the arc consumption it is about to 4GB so I do not really know where it all goes, but surelly firefox consumes up to 3GB when running with 2-3 tabs.

But my point is, I have some spare ram. I think I could use 10-12GB of it as tmpfs for building ports, but yet, I got a question: would this amount be enough to build big ports such FF?

I would have probably to take care while building alot of ports maybe?

Instead of just updating everyone, would be safest to build one each time and them clear tmpfs?

Would this give a significant performance over build time?
 
Yes, you have enough RAM. I have less and I could build firefox, libreoffice, qtcreator, etc. And yes, some ports may take several hours to build, but with poudriere you can check the progress pressing CTL+T or interrupt the building with CTL+C. Poudriere will keep the progress and starts from there, not all over from the start.

My recommendations is you definitely need to use Poudriere. In fact, you want to install poudriere-dsh2dsh, which is an optimized poudriere: https://github.com/dsh2dsh/poudriere/
It works just like poudriere with the same config file.

pkg install poudriere-dsh2dsh

In general, check the Handbook for poudriere use, configure ccache and tmpfs.
From there you want to make a list of ports in a text file,

# bulk_list.txt
editors/libreoffice
editors/vim@gtk3
graphics/inkscape
...
Click to expand...

and then you run as root something like this:

poudriere bulk -j 14intel64 -p default -f /usr/local/etc/poudriere.d/bulk_list.txt

Here I send part of my poudriere.config as example. It works for me. Notice the whitelists and blacklists. Notice you can use wildcards like qt*. They are very helpful for big ports.
Edit yours to fit your needs.

...

# Use portlint to check ports sanity
USE_PORTLINT=no

# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir - Use tmpfs(5) for port building WRKDIRPREFIX
# data - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all - Run the entire build in memory, including builder jails.
# yes - Enables tmpfs(5) for wrkdir and data
# no - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=yes

# How much memory to limit tmpfs size to for *each builder* in GiB
# (default: none)
#TMPFS_LIMIT=8

# List of package globs that are not allowed to use tmpfs for their WRKDIR
# Note that you *must* set TMPFS_BLACKLIST_TMPDIR
# EXAMPLE: TMPFS_BLACKLIST="rust"
TMPFS_BLACKLIST="vscode* rust* qt* web* llvm* firefox *chromium* libre* apache* electron* virtual* wget* *cad*"

# The host path where tmpfs-blacklisted packages can be built in.
# A temporary directory will be generated here and be null-mounted as the
# WRKDIR for any packages listed in TMPFS_BLACKLIST.
# EXAMPLE: TMPFS_BLACKLIST_TMPDIR=${BASEFS}/data/cache/tmp
TMPFS_BLACKLIST_TMPDIR=${BASEFS}/data/cache/tmp

# How much memory to limit jail processes to for *each builder*
# in GiB (default: none)
# This can also be set per PKGBASE, such as MAX_MEMORY_rust=20.
# Package names with hyphens (-) should be replaced with underscores (_).
# MAX_MEMORY=8

# How many file descriptors to limit each jail process to (default: 8192)
# This can also be set per PKGBASE, such as MAX_FILES_RStudio=2048.
# Package names with hyphens (-) should be replaced with underscores (_).
# MAX_FILES=8192

# If set the given directory will be used for the distfiles
# This allows to share the distfiles between jails and ports tree
# If this is "no", poudriere must be supplied a ports tree that already has
# the required distfiles.
DISTFILES_CACHE=/usr/ports/distfiles

....

# ccache support. Supply the path to your ccache cache directory.
# It will be mounted into the jail and be shared among all jails.
# It is recommended that extra ccache configuration be done with
# ccache -o rather than from the environment.
#CCACHE_DIR=/var/cache/ccache
CCACHE_DIR=/var/cache/ccache

# Static ccache support from host. This uses the existing
# ccache from the host in the build jail. This is useful for
# using ccache+memcached which cannot easily be bootstrapped
# otherwise. The path to the PREFIX where ccache was installed
# must be used here, and ccache must have been built statically.
# Note also that ccache+memcached will require network access
# which is normally disabled. Separately setting RESTRICT_NETWORKING=no
# may be required for non-localhost memcached servers.
#CCACHE_STATIC_PREFIX=/usr/local

...

# parallel build support.
#
# By default poudriere uses hw.ncpu to determine the number of builders.
# You can override this default by changing PARALLEL_JOBS here, or
# by specifying the -J flag to bulk/testport.
#
# Example to define PARALLEL_JOBS to one single job
# PARALLEL_JOBS=1
PARALLEL_JOBS=3

# How many jobs should be used for preparing the build? These tend to
# be more IO bound and may be worth tweaking. Default: PARALLEL_JOBS * 1.25
# PREPARE_PARALLEL_JOBS=2
# PREPARE_PARALLEL_JOBS=5
PREPARE_PARALLEL_JOBS=10
...

# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
# ALLOW_MAKE_JOBS=yes

# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
#ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"
ALLOW_MAKE_JOBS_PACKAGES="llvm* pkg ccache py* rust qt* libre* *chromium* vscode electron*"

# Timestamp every line of build logs
# Default: no
#TIMESTAMP_LOGS=no

# This defines the max time (in seconds) that a command may run for a build
# before it is killed for taking too long. Default: 86400
#MAX_EXECUTION_TIME=86400

# This defines the time (in seconds) before a command is considered to
# be in a runaway state for having no output on stdout. Default: 7200
#NOHANG_TIME=7200
...

# Keep older package repositories. This can be used to rollback a system
# or to bisect issues by changing the repository to one of the older
# versions and reinstalling everything with `pkg upgrade -f`
# ATOMIC_PACKAGE_REPOSITORY is required for this.
# Default: no
# KEEP_OLD_PACKAGES=no
KEEP_OLD_PACKAGES=yes

# How many old package repositories to keep with KEEP_OLD_PACKAGES
# Default: 5
KEEP_OLD_PACKAGES_COUNT=5

....

# Define pkgname globs to boost priority for
# Default: none
#PRIORITY_BOOST="pypy openoffice*"
PRIORITY_BOOST="llvm* py* open* libre* qt* rust llvm firefox *chromium* apache* gimp virtual* *gtk* *web* wget* *cad* vscode electron*"

....
Click to expand...

Good luck!
 
cortexmancer said:
I think I could use 10-12GB of it as tmpfs for building ports, but yet, I got a question: would this amount be enough to build big ports such FF?
Click to expand...
Firefox is not that big of a build, it takes a lot of time yes, but it doesn't use that much resources. The thing to watch out for is when your poudriere might start building GCC, LLVM and rust concurrently. That's what typically killed my old build server (16GB memory, 16GB swap). MUTUALLY_EXCLUSIVE_BUILD_PACKAGES="llvm* rust* gcc*" can help prevent that from happening.

My current build server has 192GB memory, that's plenty to be able to set USE_TMPS=all, and I have a fair number of cores (12 cores, 24 threads) for ALLOW_MAKE_JOBS=yes, but I have limited PARALLEL_JOBS=6 to give each job more 'spare' cores to use with ALLOW_MAKE_JOBS.

Code: 
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=all
{...}
# parallel build support.
#
# By default poudriere uses hw.ncpu to determine the number of builders.
# You can override this default by changing PARALLEL_JOBS here, or
# by specifying the -J flag to bulk/testport.
#
# Example to define PARALLEL_JOBS to one single job
PARALLEL_JOBS=6
{...}
# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
ALLOW_MAKE_JOBS=yes
{...}
 
32GB of RAM is insufficient at least for me not to swap on poudriere builds with 6 builders and with ALLOW_MAKE_JOBS.
Even though I've blocked some too huge ones from tmpfs, I observe around 25GB of swap at maximum until building everything (as of OSVERSION bump or BRANCH bump).

Note that I'm building llvm things first, then rust, and gcc things prior to others and blocklisting too huge ones like www/chromium to be built later using my script here (actual list differs, as it is the example list).

So I consider 64GB would be the minimum to avoid swaps for 6 builders.
More builders could require more, though.
 
On my VirtualBox FreeBSD VM (8G RAM, 8 CPU, 4GB swap) I can build firefox, thunderbird & rust while in XFCE doing things (MAKE_JOBS_NUMBER=4). For llvm19 and llvm20 I have to stay in vt, they take 3/4 hours each one (llvm). In XFCE compiling llvm ports fail (out of mem and swap). I do not use tmpfs
 
cortexmancer said:
But my point is, I have some spare ram. I think I could use 10-12GB of it as tmpfs for building ports, but yet, I got a question: would this amount be enough to build big ports such FF?
Click to expand...

That question makes no sense without stating how many cores you build on and whether you want to use hyperthreading.

What is your CPU(s)?
 
malvarco said:
Yes, you have enough RAM. I have less and I could build firefox, libreoffice, qtcreator, etc. And yes, some ports may take several hours to build, but with poudriere you can check the progress pressing CTL+T or interrupt the building with CTL+C. Poudriere will keep the progress and starts from there, not all over from the start.

My recommendations is you definitely need to use Poudriere. In fact, you want to install poudriere-dsh2dsh, which is an optimized poudriere: https://github.com/dsh2dsh/poudriere/
It works just like poudriere with the same config file.
...
Click to expand...
I have been using only portmaster to update ports, and to build the ones when a simple make didnt went well.

While doing some research I saw alot of old posts talking about poudriere along with tmpfs. But I was hoping to not need to change the manager that I am currently using. But I'll sure read about poudriere.
SirDice said:
My current build server has 192GB memory, that's plenty to be able to set USE_TMPS=all, and I have a fair number of cores (12 cores, 24 threads) for ALLOW_MAKE_JOBS=yes, but I have limited PARALLEL_JOBS=6 to give each job more 'spare' cores to use with ALLOW_MAKE_JOBS.
Click to expand...

Neat.
cracauer@ said:
That question makes no sense without stating how many cores you build on and whether you want to use hyperthreading.

What is your CPU(s)?
Click to expand...

11th Gen Intel i7-1165G7 (8) @ 2.803GHz

I have readed about parallel jobs as well. And I have never tried it as well, but I also do not know how much would my computer supports. It is a thing, beucase when building ports all cores/threads are allways at 100% so I do not know if even 2 parallel jobs would be a good idea.

I also have readed about distcc, and I have a steamdeck that I could use for computation when not gaming. But I havent made any research to know if is possible to use distcc with two differents OS'es.

I also didnt havent setup much swap (only 2GB), I think it was the default. In other times I did had setuped more swap, but since I never used it, this last time I just leaved it as it was.
 
cortexmancer said:
You mean by that that I should/could run 8 parallel jobs?
Click to expand...
Of course, you can. I run parallel jobs with my modest CPU Model: Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz.
But give Poudriere a big try, as I said above. For me it's a must for building on FreeBSD. Once you understand it, it's not even hard.
The key lines on poudriere.conf here would be:

TMPFS_BLACKLIST="rust* qt* llvm* firefox libreoffice" # In general, all the hungry-resource ports. It depends also on your tmpfs config.
ALLOW_MAKE_JOBS=no # This is the default by the way
ALLOW_MAKE_JOBS_PACKAGES="rust* qt* llvm* firefox libreoffice" # Again the hungry ports. This is to allow an exception to the line above.
Click to expand...
 
I have a CPU with 20 thread and 64GB of RAM and ran several tests involving tmpfs, full bulks, ccache, and make jobs.
I must admit that in my tests, using tmpfs didn’t have much impact on compilation time — so I don’t use tmpfs.
What really makes a difference in build time is the combination of parallel jobs with make jobs, and especially the use of ccache.

What do I usually do?
I define some heavy packages (llvm, rust, gcc, node22...) as MUTUALLY_EXCLUSIVE_BUILD_PACKAGES.
Before each bulk, I always do a dry-run to check if any of those heavy packages are going to be built. If so, I run a bulk just for those heavy packages. Since they are mutually exclusive, make jobs will fully apply to each of them, without parallel jobs.
After they are built, I run the bulk for all the others — and since they are not mutually exclusive, parallel jobs do an excellent job.
Combined with ccache, the build times for rust and llvm are very fast.
The last llvm19 build took just 11 minutes.
I usually do this process in two stages (first all mutually exclusive packages, and second the others) , and I’ve had excellent results.
But I don’t use tmpfs, because in a series of tests, it didn’t produce any significant difference on my hardware.

Of course, this will depend on your hardware, and you should run tests and measure resource usage versus build time until you find the combination that makes the most sense for you.
 
Based on all info that I received in this thread I've setuped poudriere.
My config at the moment, taking parts of everyone who sent things:


Code: 
diff --git a/poudriere/poudriere.conf b/poudriere/poudriere.conf
index fa24392..1cbe6c9 100644
--- a/poudriere/poudriere.conf
+++ b/poudriere/poudriere.conf
@@ -9,14 +9,14 @@
 # You need at least 7GB of free space in this pool to have a working
 # poudriere.
 #
-#ZPOOL=zroot
+ZPOOL=zroot
 
 ### NO ZFS
 # To not use ZFS, define NO_ZFS=yes
 #NO_ZFS=yes
 
 # root of the poudriere zfs filesystem, by default /poudriere
-# ZROOTFS=/poudriere
+ ZROOTFS=/poudriere
 
 # the host where to download sets for the jails setup
 # You can specify here a host or an IP
@@ -27,7 +27,7 @@
 # Also note that all protocols supported by fetch(1) are supported here, even
 # file:///
 # Suggested: https://download.FreeBSD.org
-FREEBSD_HOST=_PROTO_://_CHANGE_THIS_
+FREEBSD_HOST=https://download.FreeBSD.org
 
 # By default the jails have no /etc/resolv.conf, you will need to set
 # RESOLV_CONF to a file on your hosts system that will be copied to
@@ -66,11 +66,13 @@ USE_TMPFS=yes
 # List of package globs that are not allowed to use tmpfs for their WRKDIR
 # Note that you *must* set TMPFS_BLACKLIST_TMPDIR
 # EXAMPLE: TMPFS_BLACKLIST="rust"
+TMPFS_BLACKLIST="rust* qt* llvm* firefox*"
 
 # The host path where tmpfs-blacklisted packages can be built in.
 # A temporary directory will be generated here and be null-mounted as the
 # WRKDIR for any packages listed in TMPFS_BLACKLIST.
 # EXAMPLE: TMPFS_BLACKLIST_TMPDIR=${BASEFS}/data/cache/tmp
+TMPFS_BLACKLIST_TMPDIR=${BASEFS}/data/cache/tmp
 
 # How much memory to limit jail processes to for *each builder*
 # in GiB (default: none)
@@ -157,7 +159,7 @@ DISTFILES_CACHE=/usr/ports/distfiles
 # It will be mounted into the jail and be shared among all jails.
 # It is recommended that extra ccache configuration be done with
 # ccache -o rather than from the environment.
-#CCACHE_DIR=/var/cache/ccache
+CCACHE_DIR=/var/cache/ccache
 
 # Static ccache support from host.  This uses the existing
 # ccache from the host in the build jail.  This is useful for
@@ -191,10 +193,12 @@ DISTFILES_CACHE=/usr/ports/distfiles
 #
 # Example to define PARALLEL_JOBS to one single job
 # PARALLEL_JOBS=1
+PARALLEL_JOBS=2
 
 # How many jobs should be used for preparing the build? These tend to
 # be more IO bound and may be worth tweaking. Default: PARALLEL_JOBS * 1.25
 # PREPARE_PARALLEL_JOBS=1
+PREPARE_PARALLEL_JOBS=10
 
 
 # If set, failed builds will save the WRKDIR to ${POUDRIERE_DATA}/wrkdirs
@@ -229,11 +233,11 @@ DISTFILES_CACHE=/usr/ports/distfiles
 # List of package name globs that will always be allowed to use MAKE_JOBS
 # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
 # which holdup the rest of the queue to build more quickly.
-#ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"
+ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py* llvm* rust* qt*"
 
 # List of package name globs that should never build at the same time.
 # Default: none
-#MUTUALLY_EXCLUSIVE_BUILD_PACKAGES="llvm* rust* gcc*"
+MUTUALLY_EXCLUSIVE_BUILD_PACKAGES="llvm* rust* gcc* node*"
 
 # Timestamp every line of build logs
 # Default: no
@@ -270,10 +274,12 @@ DISTFILES_CACHE=/usr/ports/distfiles
 # ATOMIC_PACKAGE_REPOSITORY is required for this.
 # Default: no
 #KEEP_OLD_PACKAGES=no
+KEEP_OLD_PACKAGES=yes
 
 # How many old package repositories to keep with KEEP_OLD_PACKAGES
 # Default: 5
 #KEEP_OLD_PACKAGES_COUNT=5
+KEEP_OLD_PACKAGES_COUNT=5
 
 # Make testing errors fatal.
 # If set to 'no', ports with test failure will be marked as failed but still
@@ -311,6 +317,7 @@ DISTFILES_CACHE=/usr/ports/distfiles
 # Define pkgname globs to boost priority for
 # Default: none
 #PRIORITY_BOOST="pypy openoffice*"
+#PRIORITY_BOOST="pypy openoffice*"
 
 # Define format for buildnames
 # Default: %Y-%m-%d_%Hh%Mm%Ss

The funny part is that this config is currently using less resources than normal make/portmaster on system ports:
 

Attachments

  • test.png
    test.png
    31.9 KB · Views: 27
It takes a bit of experimenting with the various options, jobs, make, tmpfs, etc. in order to get the most effective use out of your build system. It's not always advantageous to have all cores going at 100% and end up bottlenecking on I/O for example. Or running out of memory halfway through a build run. Sometimes it's more efficient to use less concurrent jobs and prevent an I/O bottleneck or using too much memory.

I'd set up the web interface for poudriere too. You can find some example configurations in /usr/local/share/examples/poudriere/. The web UI makes it a lot easier to keep track of what it's doing, what it's working on, how long it took, etc. It also keeps track of previous build runs, so you can easily compare how a build run performs after making some changes.
 
SirDice said:
It takes a bit of experimenting with the various options, jobs, make, tmpfs, etc. in order to get the most effective use out of your build system. It's not always advantageous to have all cores going at 100% and end up bottlenecking on I/O for example. Or running out of memory halfway through a build run. Sometimes it's more efficient to use less concurrent jobs and prevent an I/O bottleneck or using too much memory.

I'd set up the web interface for poudriere too. You can find some example configurations in /usr/local/share/examples/poudriere/. The web UI makes it a lot easier to keep track of what it's doing, what it's working on, how long it took, etc. It also keeps track of previous build runs, so you can easily compare how a build run performs after making some changes.
Click to expand...
Thank you! I'll sure look into that.
I noticed that poudriere produced a log dir after the build, and that contains some index.html, jail.html, but I render nothing besides the header/footer.
I also noticed that my conf let the most usage for the big boys (rust, llvm, etc). When it reached those it was the point where all cpus went full working.
But, as you said, is good to have a little for I/O and I could keeping using the machine.
 
You must log in or register to reply here.
Back
Top