How to speed up compilation on a multi-core machine?

Hi all,

Is $subj possible at all? It's painful to see 25% CPU utilisation on a quad core PC.

I tried adding
Code:
MAKEOPTS="-j4"
to /etc/make.conf and it kind of works except:

  • It works only for `make` but many ports use `gmake`.
  • It seems `g?make install` requires -j1, at least I had a couple of broken port installations with -j4.

Any idea how to make -jX work with portmaster/portupgrade? Are there other ways to speed up compilation?

Thanks,
Alex
 
oliverh said:
First of all use ccache.
Thanks for directing me to ccache, it looks like a very useful tool.

oliverh said:
When building world you can use some jobs a la "make -jx". Regarding ports, there is no need to do so, ports will use - if possible - several jobs to compile.

If you see this line in the makefile "MAKE_JOBS_SAFE= yes", then the port will use multiple cores. See the makefile of gcc46 for example: http://www.freebsd.org/cgi/cvsweb.cgi/ports/lang/gcc46/Makefile?rev=1.451;content-type=text/plain
So it's up to maintainers to enable parallel builds for their ports? That makes sense, thanks!
 
alexk said:
Hi all,

Is $subj possible at all? It's painful to see 25% CPU utilisation on a quad core PC.

I tried adding `MAKEOPTS="-j4"` to /etc/make.conf and it kind of works except:

  • It works only for `make` but many ports use `gmake`.
  • It seems `g?make install` requires -j1, at least I had a couple of broken port installations with -j4.

Any idea how to make -jX work with portmaster/portupgrade? Are there other ways to speed up compilation?

Thanks,
Alex

Related: http://forums.freebsd.org/showthread.php?t=19172
MAKEOPTS is not supported by FreeBSD
 
alexk said:
So it's up to maintainers to enable parallel builds for their ports? That makes sense, thanks!

Not exactly.

By default, the parallel build only occurs if the port makefile explicitly states that MAKE_JOBS_SAFE=yes.

But the user can force parallel builds by adding the appropriate lines to make.conf, and the port makefiles will all respect that -- unless the maintainer has explicitly disabled it by specifying that MAKE_JOBS_UNSAFE=yes.

Users can also override this forcing of parallel make jobs on a port-by-port basis by adding appropriate entries in make.conf. For example:

Code:
FORCE_MAKE_JOBS=yes
MAKE_JOBS_NUMBER=4

.if ${.CURDIR:M*/audio/amarok}
.undef FORCE_MAKE_JOBS
.endif

Not all ports build successfully with multiple make jobs, however, and not all of the ones that fail have been marked MAKE_JOBS_UNSAFE. So if you use FORCE_MAKE_JOBS and a build fails, try overriding it for that specific port before you report it as a bug.
 
its not always desireable to have 100% used by the compiler as things running in the background need to maintain response time whilst compiling.
 
chrcol said:
its not always desireable to have 100% used by the compiler as things running in the background need to maintain response time whilst compiling.

I think we can credit the FreeBSD scheduler with being smart enough to avoid the kind of problem you're describing.

I've been using parallel make jobs to build ports for at least the last year and a half and I've never had any problems getting CPU cycles for the web server and other background processes running on the same machine. And this is with a relatively-underpowered Intel Atom microprocessor.

My experience, however, is only with home networks used only by myself and a few members of my family. I can't speak to the situation on production servers that need to handle much heavier loads. But if the load is so heavy that compiling on the same machine became a problem, I think I'd move that task off the production servers and onto an offline machine reserved for that purpose.
 
Hello all,
I know it's an old topic, I hope someone reads it. In the contrast of the topic's title, I'd like "slow down" the compilation. :)
While I'm compiling, the GUI becomes slow, unresponsive, and the audio stops for milliseconds, what is very annoying. Is there any way, I could configure the scheduler, to make the compilation less important, therefore rather slow down the compilation, then slow down any other process?
Thank you very much in advance.
I've already set the following in the /etc/sysctl.conf file:
Code:
kern.sched.preempt_thresh=224
 
ccache doesn't help much on newer machines, only on older.
(By the way a warning. Never had problems with ccache on i386, but on amd64 it messed up my system, and not only mine).
 
Back
Top