Utilize all cores during ports compilation

[Grell]

Hello, I am trying to compile ports using portmaster but am having problems in that whenever I do so only one CPU core is being used. I have a 12 core CPU so as you can imagine my hardware is not being taken advantage of. It takes quite long to compile ports, especially those that are quite large like chromium. For some reason I have not had this issue before and have only noticed it recently, I cannot trace back to where it first started, however, or anything I may have done necessarily to cause this to happen. In my make.conf file in /etc I have set both MAKE_JOBS_NUMBER=13 as well as MAKE_JOBS_UNSAFE=yes. What can I do to fix this problem? Thank you for your help.

 

[T-Aoki]

Remove MAKE_JOBS_UNSAFE=yes from /etc/make.conf.
It prohibits pararell make.

But in some unfortunate cases, it would be needed (but not set in ports Makefile by the maintainer yet). For those unfortunate case, if the specific port is foo/bar only, you can specify it in /etc/make.conf like below. (Assuming your ports tree is at the default place.)

.if ${.CURDIR:M/usr/ports/foo/bar}
MAKE_JOBS_UNSAFE=yes
.endif

But if you build any port using lang/rust, the rust parts of the port would easily eats up all cores available by rustc, regardless the settings with my experience. I think rustc itself is designed as multithreaded program.

 

[zirias@]

Does your machine have a decent amount of RAM? Do you use ZFS? Then, better use poudriere instead, giving you lots of advantages:

• Your own pkg repo (which e.g. means no need to ever pollute your live system with build dependencies, no complicated procedures to follow updates of e.g. DEFAULT_VERSIONS, etc...)
• Clean build environments, guarding you from (sometimes weird and hard to track) issues caused by something from your live system being picked up by a build
• The possibility to build several ports in parallel (while you are still free to also use MAKE_JOBS)
• Automatic usage of tmpfs(5), e.g. for the work dirs, speeding up the builds
• Archived build logs, a common cache for distfiles, and easy to use "housekeeping" commands for these
• In case of "poudriere-devel", a clean and reliable way to integrate pre-built packages
• ... probably quite a few more I can't think of right now ...

I don't mean to say "portmaster" wouldn't work, it's just quite limited by design, so I wonder why it's still widely used (well, short of poudriere not being an option for very limited system resources)

 

[T-Aoki]

Yes, ports-mgmt/poudriere-devel would be a nice tool for *-Release.
Currently, no reason to choose ports-mgmt/poudriere, as ports-mgmt/poudriere-devel works better for now.

But even with ports-mgmt/poudriere-devel, I cannot recommend it for users of main (-Current) branch. Because poudriere[-devel] attemps to rebuild ALL ports specified (regardless updated or not) when OSVERSION is bumped.
If ports-mgmt/poudriere-devel ignores last 3 digits of OSVERSION by default (or at worst, option to do so is added), it can be more preferrable.

stable/* branch is intermediate, as it gets less bumps than main branch.

If switching to poudriere-devel,

• Consider using -S option for bulk builds.
• Initial setups would be complexed for casual users who want to build ports for the computer itself only, as it has too rich features for cross building etc.
• poudriere[-devel] defaults only one process per cpu for single port. You must manually configure to allow it if you want.

The last one means that poudriere[-devel] builds multiple ports at once but allows only one job for each port by default. See poudriere.conf.sample for details. The link points to the one for poudriere-devel.

 

[T-Aoki]

Forgot to mention.
There are several other ports builders (CI tools), such as ports-mgmt/jenkins[lts] and ports-mgmt/synth. I cannot comment for details as I haven't tried them yet.
If I recall correctly, official pkg builder uses jenkins.
And reading its pkg-descr, synth seems to fit better for casual users like me, but it is written in Ada. This means it is easily lost if current maintainer steps out. Actually, it could be already removed from ports if current maintainer didn't popped in at that time.

 

[blind0ne]

Forget about it, this problem is unsolvable, i've already spend few month on this few years ago. use precompiled binaries if you don't need to modify your program for some reason.

 

[zirias@]

But even with ports-mgmt/poudriere-devel, I cannot recommend it for users of main (-Current) branch. Because poudriere[-devel] attemps to rebuild ALL ports specified (regardless updated or not) when OSVERSION is bumped.
If ports-mgmt/poudriere-devel ignores last 3 digits of OSVERSION by default (or at worst, option to do so is added), it can be more preferrable.

First of all, this is dangerous. OSVERSION is bumped for a reason, some part of the ABI changed. There's no way to (automatically) know which ports exactly MUST be rebuilt, but a package using exactly the part of the ABI that changed will be broken for sure, therefore rebuilding everything is the only safe thing to do.

And second, if you don't want to do that (and live with the risk), it's as simple as: Just don't update your poudriere building jail.

Consider using -S option for bulk builds.

This is likely to break from time to time as well. If build times are a concern, I would rather recommend to use ccache. It will "only" reuse previously compiled object files (instead of not recompiling a whole port at all), but it's safe.

Forget about it, this problem is unsolvable,

Sorry, but ... what? I always did parallel builds, no matter whether using poudriere or not, and it's dead simple. OP's only immediate problem is setting MAKE_JOBS_UNSAFE, which is meant to be put into port Makefiles for cases when parallel building actually breaks, therefore it unconditionally disables it.

 

[T-Aoki]

First of all, this is dangerous. OSVERSION is bumped for a reason, some part of the ABI changed. There's no way to (automatically) know which ports exactly MUST be rebuilt, but a package using exactly the part of the ABI that changed will be broken for sure, therefore rebuilding everything is the only safe thing to do.

I 100% agree with it for main (-Current) branch, but not for stable/* and releng/* branches. Breaking changes are basically disallowed for stable/* and releng/*. So A[B|P]/K[B|P]I changes to stable/* and releng/* should be just adding something new using spare entries. And ports affected by such a changes should adapt them by itself. This is why I require the feature to ignore last 3 digits of OSVERSION.
But yes, I understand such a feature is NOT AT ALL for main (-Current) branch. Breaking changes are always allowed there unless src tree is feature-frozen for upcoming *.0-Release (before branching new stable/* for it). This is why I don't use poudriere[-devel] for main branch, taking risk.

This is likely to break from time to time as well. If build times are a concern, I would rather recommend to use ccache. It will "only" reuse previously compiled object files (instead of not recompiling a whole port at all), but it's safe.

Yes, but it is a trade-off. With -S option, sometimes dependency-race happens and some ports fails to build with missing dependencies (basically indirect dependencies). This goes OK by running poudriere multiple times. But it's much faster than without -S option.
And thanks for clarifying ccache. I suspected it breaks cleanliness of poudriere. I'll try without -S option with ccache when I can take enough time. Full build of all pkgs installed forces me over 24hours and meanwhile my computer becomes unusable when poudriere builds any of ports written in rust.

 

[astyle]

Hello, I am trying to compile ports using portmaster but am having problems in that whenever I do so only one CPU core is being used. I have a 12 core CPU so as you can imagine my hardware is not being taken advantage of. It takes quite long to compile ports, especially those that are quite large like chromium. For some reason I have not had this issue before and have only noticed it recently, I cannot trace back to where it first started, however, or anything I may have done necessarily to cause this to happen. In my make.conf file in /etc I have set both MAKE_JOBS_NUMBER=13 as well as MAKE_JOBS_UNSAFE=yes. What can I do to fix this problem? Thank you for your help.

So, you have a 12-core CPU, but 13 for MAKE_JOBS_NUMBER... Have you tried make -j12 ? That did the trick for me. A note from my profile post:

https://forums.freebsd.org/profile-posts/3201/

 

[zirias@]

Breaking changes are basically disallowed for stable/* and releng/*.

This reasoning makes sense, it should be safe on STABLE, but then:

This is why I require the feature to ignore last 3 digits of OSVERSION.

Why not just NOT update your poudriere jail instead?

meanwhile my computer becomes unusable when poudriere builds any of ports written in rust.

Running the build through idprio(1) might help with that.

 

[T-Aoki]

Why not just NOT update your poudriere jail instead?

At past, some IGNORE'ed or BROKEN ports were unblocked with additional feature. (Upstream developers hesitate to support FreeBSD on their new versions anymore without the features. Most notably, x11/nvidia-driver case.)
So yes, basically I can DELAY updating poudriere jail, but once such a case happenes on ports that I myself need (including direct/indirect dependencies) again, I should be forced to update poudriere jail.
Additionally, keeping poudriere jail up-to-date allows me smoking out ports which are accidentally broken by the update and report back to maintainers of affected ports. In most cases, these are worked around for main with OSVERSION checks, and need additional checks for MFC'ed OSVERSIONs.
With freature I'm requiring, I would be able to notice the breakage when affected ports are actually updated, without forced whole rebuild.

Running the build through idprio(1) might help with that.

Thanks! I'll look into man page and try, maybe on next rust or firefox update. These are most significantly affected.

 

[zirias@]

At past, some IGNORE'ed or BROKEN ports were unblocked with additional feature. (Upstream developers hesitate to support FreeBSD on their new versions anymore without the features. Most notably, x11/nvidia-driver case.)
So yes, basically I can DELAY updating poudriere jail, but once such a case happenes on ports that I myself need (including direct/indirect dependencies) again, I should be forced to update poudriere jail.

Well, sure, you can't have your cake and eat it ?

If you want to be sure to avoid breakage caused by ABI changes, it means rebuild everything. As soon as you follow a "moving target" ("main" or a "stable" branch), this will happen quite often.

Also note as you wrote yourself, although the ABI is kept stable on "stable" (with only features added), it doesn't mean that it will never break a build. It should never break a binary, but I wouldn't even guarantee this never happens... still, you could of course suggest this feature on poudriere's upstream repo (on github) if you want.

In general, when building yourself, the best you can do is try to optimize it:

• Use a machine with decent resources
• Configure poudriere to use tmpfs for as much as you can afford based on your available RAM
• Set ALLOW_MAKE_JOBS, or set ALLOW_MAKE_JOBS_PACKAGES for all the "larger" ports you build (to avoid wasting CPU while waiting for I/O)
• Configure poudriere to use ccache, maybe also use the sccache-overlay (providing similar caching for e.g. rust)
• If available (only on some architectures and only for supported releases and -CURRENT), use package-prefetching (only supported in poudriere-devel so far)

 

[Grell]

Thank you everyone for your replies.

Remove MAKE_JOBS_UNSAFE=yes from /etc/make.conf.
It prohibits pararell make.

But in some unfortunate cases, it would be needed (but not set in ports Makefile by the maintainer yet). For those unfortunate case, if the specific port is foo/bar only, you can specify it in /etc/make.conf like below. (Assuming your ports tree is at the default place.)

.if ${.CURDIR:M/usr/ports/foo/bar}
MAKE_JOBS_UNSAFE=yes
.endif

But if you build any port using lang/rust, the rust parts of the port would easily eats up all cores available by rustc, regardless the settings with my experience. I think rustc itself is designed as multithreaded program.

Removing MAKE_JOBS_UNSAFE=yes did it. I am looking into poudriere right now. Currently getting it set up. But my original problem is solved.