using "make -jx" with ports

P

ph0enix

I have a quad-core system so I've tried using "make -j5 install distclean" with a few ports but most of them fail to compile that way. They compile fine with just "make install distclean". Is there a reason for this?

Thanks! :D

J.
 
You could try make -j4 && make install clean. I think the make install part panics when some jobs are finished before others. So use the extra cpu power for compiling only, and leave the installing part to a 'serial process'. I'm not entirely sure how this pans out when the ports starts pulling in dependencies, so test.
 
ph0enix said:
I have a quad-core system so I've tried using "make -j5 install distclean" with a few ports but most of them fail to compile that way. They compile fine with just "make install distclean". Is there a reason for this?
Click to expand...
Yes - the reason is: parallell make (make -j) is not supported for ports.

Think about it - there are well over 19000 (almost 20000 now) ports in the ports collection, and they use many different methods for building and installing. Even if we (FreeBSD users and developers) hda all the manpower we wanted, getting the ports to support parallel building and install would be almost impossible.

I suggest you do something more productive or fun the next time you compile a port; play a game, read a book or surf the net.
 
tingo said:
Yes - the reason is: parallell make (make -j) is not supported for ports.

Think about it - there are well over 19000 (almost 20000 now) ports in the ports collection, and they use many different methods for building and installing. Even if we (FreeBSD users and developers) hda all the manpower we wanted, getting the ports to support parallel building and install would be almost impossible.

I suggest you do something more productive or fun the next time you compile a port; play a game, read a book or surf the net.
Click to expand...

Thanks for the great suggestions but I already played 3 games, read 7 books and surfed every single page on the internet. The code is still compiling ;)
 
You can set MAKE_ARGS to the arguments you want to use. The port makefile itself won't use them, but the actual make to build will. The problems arise when you try to use -jn with the port itself, or during install. So here is what I added to the bottom of my /etc/make.conf:

Code: 
.if !(make(*install) || make(package))
MAKE_ARGS+=-j4
.endif

So far this was worked great for me. Just remember though that some apps will still bomb with parallel builds.
 
Should I use -j4 or -j5 for a quad-core system? I always thought that the formula was:
number of CPUs +1

Is this not right?
 
ph0enix said:
Should I use -j4 or -j5 for a quad-core system? I always thought that the formula was:
number of CPUs +1

Is this not right?
Click to expand...

Depends, you need to measure it, I would set it to 2 x CORES count, I remember I measured once time buildworld with -j and I got best results (fastest) with 4-5 jobs at dual core system, so I will set 8-9 at quad core.
 
vermaden said:
Depends, you need to measure it, I would set it to 2 x CORES count, I remember I measured once time buildworld with -j and I got best results (fastest) with 4-5 jobs at dual core system, so I will set 8-9 at quad core.
Click to expand...

Can I test with buildkernel instead of buildworld? It'll be "a little" faster.
 
yup indeed. the issue is that only a few ports support it currently :)
 
vermaden said:
Yes, of course.
Click to expand...

I'd say the test results are inconclusive with buildkernel. I did two runs with each setting. It looks like -j3 is out but that's about it. I'll try buildworld:

-j8:
real 5m45.118s
user 10m8.636s
sys 1m18.063s

real 5m29.173s
user 10m8.689s
sys 1m17.048s

-j6:
real 5m24.774s
user 10m8.773s
sys 1m17.537s

real 5m26.855s
user 10m8.329s
sys 1m16.085s

-j5:
real 5m27.501s
user 10m9.188s
sys 1m16.038s

real 5m21.730s
user 10m7.112s
sys 1m16.250s

-j4:
real 5m38.467s
user 10m5.442s
sys 1m14.540s

real 5m20.220s
user 10m5.708s
sys 1m14.321s

-j3:
real 5m45.420s
user 10m0.737s
sys 1m10.541s

real 5m51.748s
user 10m0.168s
sys 1m11.131s

-j1 is about 11 minutes.
 
Official MAKE_JOBS hit ports tree

pav@ committed initial support for parallel builds. See this PR for details.

At present it works as opt-in on a per port basis, meaning the port maintainer has to set MAKE_JOBS_SAFE in the Makefile to enable parallel builds. When set, the number of jobs is set to the number of cores in the machine, specifically the output of sysctl -n kern.smp.cpus.

Usage of MAKE_ARGS is now discouraged. Adventurous users can set FORCE_MAKE_JOBS to enable parallel builds globally or on a per port basis, using Makefile.local or constructs in make.conf like so:
Code: 
.if !empty(.CURDIR:M*/x11/kdebase3)
FORCE_MAKE_JOBS=yes
.endif

An opt-out approach can be done like so:
Code: 
FORCE_MAKE_JOBS=yes
# These I know to be unsafe
UNSAFE_PORTS =audio/libsndfile
UNSAFE_PORTS+=audio/nas
UNSAFE_PORTS+=audio/xfce4-mixer
UNSAFE_PORTS+=devel/doxygen
UNSAFE_PORTS+=devel/gperf
UNSAFE_PORTS+=devel/libthai
UNSAFE_PORTS+=devel/nasm
UNSAFE_PORTS+=devel/pth
UNSAFE_PORTS+=devel/dbus-qt4
UNSAFE_PORTS+=editors/vim
UNSAFE_PORTS+=games/falconseye
UNSAFE_PORTS+=graphics/libart_lgpl
UNSAFE_PORTS+=lang/lua50
UNSAFE_PORTS+=lang/ocaml
UNSAFE_PORTS+=misc/e2fsprogs-libuuid
UNSAFE_PORTS+=net-mgmt/net-snmp
UNSAFE_PORTS+=print/ghostscript8
UNSAFE_PORTS+=security/libgpg-error
UNSAFE_PORTS+=security/nss
UNSAFE_PORTS+=textproc/xerces-c2
UNSAFE_PORTS+=www/lynx
UNSAFE_PORTS+=x11-clocks/kdetoys3
# Apply disabling make jobs
.for port in ${UNSAFE_PORTS}
.if !empty(.CURDIR:M*/${port})
DISABLE_MAKE_JOBS=yes
.endif
.endfor

Hope this helps.
 
Mel_Flynn said:
pav@ committed initial support for parallel builds. See this PR for details.

At present it works as opt-in on a per port basis, meaning the port maintainer has to set MAKE_JOBS_SAFE in the Makefile to enable parallel builds. When set, the number of jobs is set to the number of cores in the machine, specifically the output of sysctl -n kern.smp.cpus....
Click to expand...

To translate this...
It is implemented in a way that we, the retards, don't even have to think about it, as if port is capable of parallel build, then it will automatically build with -j option.
And not only that, it will also choose N (-jN) number.., again on it's own, so once again, we don't even have to think about it :stud
 
I realize this thread is a few years old, but if you are still looking around for a way to utilize your quad-core better, I just wrote a script to install the dependencies for a port in parallel. Each port being built will still use the 1 core, but will have multiple ports building at once, while being considerate of dependencies.

I wrote this in SH and it does not require the /usr/ports/INDEX file, so this will work on the base system without additional languages or ports.

http://forums.freebsd.org/showthread.php?p=136569#post136569

Let me know how it works, and if you are interested in other features.
 
You must log in or register to reply here.
Back
Top