make with few threads or wher is the '-j' option

Every single port that I'm trying to compile takes ages, on linux there is '-j' option to specify the number of threads to compile with. I think that already questioned this issue, but really can't find this thread anymore (probably it was my dream =) ), so here is the example that i'm bumping in in every single port compilation attempt whenever need to run something from sources.
Code:
# make -j 24
--- config-conditional ---
--- do-config ---
*** [do-config] Stopped -- signal 22

So my question is - give me a hint where I can read 'handbook' on 'mastering' ports, or another links on it? How did you learn this skill in advance?
 
Maybe make -j24
Code:
root@ihor:/usr/ports/games/openarena # make -j24
--- stage ---
--- /usr/ports/games/openarena/work/.build_done.openarena._usr_local ---
--- stage-message ---
===>  Staging for openarena-0.8.8.s1910_17,2
--- stage-dir ---
--- run-depends ---
===>   openarena-0.8.8.s1910_17,2 depends on file: /usr/local/share/openarena/baseoa/pak0.pk3 - not found
--- check-deprecated ---
--- fake-pkg ---
--- /usr/ports/games/openarena-data/work/.PLIST.mktmp ---
===>   NOTICE:

The data port currently does not have a maintainer. As a result, it is
more likely to have unresolved issues, not be up-to-date, or even be removed in
the future. To volunteer to maintain this port, please create an issue at:
--- check-deprecated ---
https://bugs.freebsd.org/bugzilla
More information about port maintainership is available at:
https://docs.freebsd.org/en/articles/contributing/#ports-contributing
--- check-license ---
===>   Registering installation for openarena-data-0.8.8 as automatic
===>  License GPLv2 accepted by the user
--- pkg-depends ---
--- fake-pkg ---
pkg-static: open(/usr/ports/games/openarena-data/work/.metadir): No such file or directory
*** [fake-pkg] Error code 1

make[2]: stopped in /usr/ports/games/openarena-data
--- pkg-depends ---
===>   openarena-data-0.8.8 depends on file: /usr/local/sbin/pkg - found
--- /usr/ports/games/openarena-data/work/.PLIST.mktmp ---
*** [/usr/ports/games/openarena-data/work/.PLIST.mktmp] Error code 6

make[2]: stopped in /usr/ports/games/openarena-data
2 errors

make[2]: stopped in /usr/ports/games/openarena-data
*** [run-depends] Error code 2

make[1]: stopped in /usr/ports/games/openarena
1 error

make[1]: stopped in /usr/ports/games/openarena
*** [stage] Error code 2

make: stopped in /usr/ports/games/openarena
1 error

make: stopped in /usr/ports/games/openarena
root@ihor:/usr/ports/games/openarena #
 
Last edited by a moderator:
root@ihor:/usr/ports/games/openarena # make -j24 --- stage --- --- /usr/ports/games/openarena/work/.build_done.openarena._usr_local --- --- stage-message --- ===> Staging for openarena-0.8.8.s1910_17,2 --- stage-dir --- --- run-depends --- ===> openarena-0.8.8.s1910_17,2 depends on file: /usr/local/share/openarena/baseoa/pak0.pk3 - not found --- check-deprecated --- --- fake-pkg --- --- /usr/ports/games/openarena-data/work/.PLIST.mktmp --- ===> NOTICE: The data port currently does not have a maintainer. As a result, it is more likely to have unresolved issues, not be up-to-date, or even be removed in the future. To volunteer to maintain this port, please create an issue at: --- check-deprecated --- https://bugs.freebsd.org/bugzilla More information about port maintainership is available at: https://docs.freebsd.org/en/articles/contributing/#ports-contributing --- check-license --- ===> Registering installation for openarena-data-0.8.8 as automatic ===> License GPLv2 accepted by the user --- pkg-depends --- --- fake-pkg --- pkg-static: open(/usr/ports/games/openarena-data/work/.metadir): No such file or directory *** [fake-pkg] Error code 1 make[2]: stopped in /usr/ports/games/openarena-data --- pkg-depends --- ===> openarena-data-0.8.8 depends on file: /usr/local/sbin/pkg - found --- /usr/ports/games/openarena-data/work/.PLIST.mktmp --- *** [/usr/ports/games/openarena-data/work/.PLIST.mktmp] Error code 6 make[2]: stopped in /usr/ports/games/openarena-data 2 errors make[2]: stopped in /usr/ports/games/openarena-data *** [run-depends] Error code 2 make[1]: stopped in /usr/ports/games/openarena 1 error make[1]: stopped in /usr/ports/games/openarena *** [stage] Error code 2 make: stopped in /usr/ports/games/openarena 1 error make: stopped in /usr/ports/games/openarena root@ihor:/usr/ports/games/openarena #
An if I type just make it start's fetching! So what steps do I need to perform before typing make -j24?
 
Going by memory, "make" in a port directory does:
make fetch
make config
make <<<<<< This is the point where you want to add the -j

To actually install a port,
make install

there may be other targets for make if you want create a package, clean directories, etc.
 
There's a fundamental misunderstanding here. Of course, FreeBSD's make supports the -j option. It just won't do anything useful for building ports.

The reason is that a Port is kind of a wrapper around all steps necessary to build some software (fetching the source, extracting it, building, installing to some "staging" dir, building a binary package from the results), and this process is automated using make. For the actual "build" step, the ports framework calls the tool necessary for it (which might be make as well, but could be anything else). You want your -j option in that invocation!

Fortunately, the ports framework already detects the number of your CPUs and passes appropriate flags (e.g. -j for make) to the build. Yes, the default is to have as many make jobs as you have CPUs (cores, threads, ...).

This should already give best results, except for ports with a broken build system that would just fail with parallelization (in that case, MAKE_JOBS_UNSAFE is set in the port Makefile and the build always only uses a single job).

Still, you can override the number of jobs using MAKE_JOBS_NUMBER, e.g. in make.conf(5), see also bsd.port.mk.
 
Zirias : I once tried to compile devel/llvm10 with and without the -j flag. Without the flag, it took my processor 2.5 hours to compile that port. With the flag (-j4), it was 50 minutes for the same thing. But other than that, I have to agree with your assessment - there is a fundamental mis-understanding about processor threads and compilation jobs here. 😩
 
-j option somehow affects the dependencies installation. After -j2 at make step it gives error 22 or something like that. And complaining about dependencies
 
-j option somehow affects the dependencies installation. After -j2 at make step it gives error 22 or something like that. And complaining about dependencies
That's because you introduce race conditions.
  1. One thread compiles lib.so.1,
  2. another thread compiles lib.so.2, and finishes earlier than lib.so.1.
    1. Thread finishes, tries to start the linker as the next task, and fails - why?
    2. lib.so.1 is not done compiling, so it's not available to the linker yet!
Make is not smart enough to manage race conditions like that. Don't use the -j option for ports. Only use it when ALL the deps have been satisfied.
 
Back
Top