# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
ALLOW_MAKE_JOBS=YES
Does not seem to work.
# Use the following to allow it anyway
ALLOW_MAKE_JOBS=YES
Does not seem to work.
MAKE_JOBS_UNSAFE
set. This is only set for good reasons (the upstream build system is broken and can't correctly handle parallel make). When it's set, this port will never use parallel make.MAKE_JOBS_NUMBER
there, that's what's passed to make -j
, so, in theory, yes, although it's extremely unlikely make
will ever find 1000 jobs it can execute in parallel. There are dependencies during builds!poudriere
would attempt to create 1000 builder jails and use them in parallel (and each of them trying to use up to 1000 jobs).That's the default configuration. SettingMy experience is that with my 8=4x2-core CPU poudriere allows me to build 8 different ports in parallel.
But not one port using the 8-cores.
ALLOW_MAKE_JOBS=yes
will use 8 jobs for every port where that's possible in addition (ending up with up to 8*8=64 jobs when all 8 builders are in the build phase). If that doesn't happen on your machine, check all your configs. You will for example see everything poudriere puts in make.conf in the build log of a port.MAKE_JOBS_NUMBER=
in /usr/local/etc/poudriere.d/make.conf.These two have nothing to do with each other[*]. poudriere.conf is for configuring poudriere, it defines how the application itself operates. make.conf(5) is for setting generic build time options.and the question of what poudriere.conf does in relation to /usr/local/etc/poudriere.d/make.conf was one of the points that annoyed me.
llvm*
for example. If you have a lot of jobs waiting on a single core build of llvm10 for example and all the other jobs are idling.It's a feature. You can have for example jail- and ports-tree-specific make.conf entries with poudriere(8) (it's all documented in the manpage).I remember being confused as hell about poudriere's configuration, and the question of what poudriere.conf does in relation to /usr/local/etc/poudriere.d/make.conf was one of the points that annoyed me.
Sure, butThese two have nothing to do with each other. poudriere.conf is for configuring poudriere, it defines how the application itself operates. make.conf(5) is for setting generic build time options.
MAKE_JOBS_NUMBER
is a make.conf configuration. I see no reason to change it though… make
does when buildings ports in poudriere's jails, except in the case of MAKE_JOBS_NUMBER=
, which is ignored, unless ALLOW_MAKE_JOBS=yes
is also set in /usr/local/etc/poudriere.conf, or the package being built is matched in ALLOW_MAKE_JOBS_PACKAGES=""
in /usr/local/etc/poudriere.conf.And nobody cares.but I call that a mess.
MAKE_JOBS_NUMBER
isn't documented in any manpage. Of course it is documented in /usr/ports/Mk/bsd.port.mk, but that's not the most obvious place. OTOH, it's pretty obvious that overriding the number of jobs is an entirely different thing than enabling/disabling parallel builds.)Some ports ignore this setting completely and will only use one core regardless if you enabled ALLOW_MAKE_JOBS_PACKAGES or ALLOW_MAKE_JOBS.
MAKE_JOBS_NUMBER
) to the port build. There are two typical issues with parallel building of software: MAKE_JOBS_UNSAFE
exists, which is meant to be put in a port's Makefile when the software is known to fail building in parallel ... it will force using only a single job no matter what for that port.-j
flag to specify the number of parallel jobs – the names of the ports variables directly derive from that. Now more and more software uses different ("modern") build systems. FreeBSD's ports framework has support for several of them that are widely used (e.g. cmake, meson, ...) and knows how to tell them to use multiple jobs. Some other software uses "fully custom" build systems, so the port itself must figure out how to make them build in parallel. This seems to be the case for rust, see below. With all the varieties how to configure a build system for parallel jobs, problems are to be expected, either by issues with the tools used or errors in using the build system.do-build:
@cd ${WRKSRC} && \
${SETENV} ${MAKE_ENV} ${PYTHON_CMD} x.py dist --jobs=${MAKE_JOBS_NUMBER}
# The build system reads the environment variable $NINJA_PATH to decide whether
# to boostrap ninja or not (and also to invoke it afterwards). CC and CXX are
# read by some Chromium code to determine which compiler to invoke when running
# some configuration tests.
CONFIGURE_ENV+= NINJAFLAGS="-j${MAKE_JOBS_NUMBER}" \
NINJA_PATH="${LOCALBASE}/bin/ninja" \
PATH=${CONFIGURE_WRKSRC}/bin:${LOCALBASE}/bin:${PATH}
MAKE_ENV+= CC="${CC}" CXX="${CXX}" \
C_INCLUDE_PATH=${LOCALBASE}/include \
CPLUS_INCLUDE_PATH=${LOCALBASE}/include \
${CONFIGURE_ENV}
# Avoid running multiple make(1) jobs, but only those. Otherwise the build
# fails intermittently due race conditions if multiple ninja instances are
# running at the same time (mostly for the targets "WebEngineCore" and
# "convert_dict").
#
# MAKE_JOBS_UNSAFE is too invasive because it also affects the number of jobs
# for ninja(1) and would slow everything down which we don't want. We pass the
# real number of make jobs via MAKE_JOBS_NUMBER to ninja(1) to CONFIGURE_ENV.
DO_MAKE_BUILD= ${SETENV} ${MAKE_ENV} ${MAKE_CMD} ${MAKE_FLAGS} ${MAKEFILE} -j1 ${MAKE_ARGS:N${DESTDIRNAME}=*}
ninja
. For some reason, make must be forced to only use a single job, while ninja should use the value from NINJAFLAGS
. If this suddenly doesn't work any more, upstream must have introduced some breaking changes to their build.