[HOWTO] buildworld using clang + ccache

Beeblebrox

Beeblebrox

clang is now the default compiler for FreeBSD (instead of gcc-42), and things supposedly run quicker under clang. You can do the buildworld with clang, but including ccache in the process needs some small adjustments:

1. Edit your /etc/src.conf as below. clang is enabled by default anyway, but you may want to add extras to it. Originally this post advised WITHOUT_GCC for a quicker build, but this is ill-advised as explained below.
Code: 
WITH_CLANG= yes
WITH_CLANG_EXTRAS= yes # Build additional clang and llvm tools, such as bugpoint.
WITH_CLANG_IS_CC= yes  # Useful for some buildworld errors
# WITHOUT_GCC= yes

2. In your /usr/local/etc/buildflags.conf set:
Code: 
/usr/src | /usr/src/*{
        CC=     clang
        CPP=    clang-cpp
        CXX=    clang++
        USE_CCACHE
        USE_CCACHE_CPP2
}

You are now set to buildworld with clang + ccache. CCACHE_CPP2 must be enabled because clang dislikes ccache trying extra flags.

If you do not use sysutils/bsdadminscripts, and therefore do not have buildflags.conf, you can set those parameters from command line :
[CMD="/usr/src> #"]make -DUSE_CCACHE -DUSE_CCACHE_CPP2 buildworld[/CMD]
In this situation you of course need these in your /etc/make.conf.
Code: 
CC=     clang
CPP=    clang-cpp
CXX=    clang++

Thanks to Volodymyr Kostyrko for helping me solve this.

EDIT: Apparently as of the date of this post, setting WITHOUT_GCC= yes does not work yet and breaks buildworld. It must be left in /etc/src.conf for now.
 
What ccache environment variables or /etc/make.conf settings are used with this?

Although it will increase compile time, it might be a good idea to leave gcc enabled in src.conf just to have it available.
 
No additional ccache environemnt variables / settings required than when using gcc-4* (CCACHE_PATH, CCACHE_DIR) and setting the size (# ccache -M <size>)

The config file for building ccache now has the options for
Code: 
CLANGLINK
LLVMLINK
These, when selected will automagically create FILE]ccache[/FILE] shortcuts for clang to ccache.

As a Third way of doing things, specifying these in /etc/make.conf should work:
Code: 
CC=/usr/local/libexec/ccache/world/clang
CXX=/usr/local/libexec/ccache/world/clang++
CPP=/usr/local/libexec/ccache/world/clang-cpp

I use lang/gcc46 for compiling my ports. Users who do not have a separate gcc should not disable gcc42 because a good number of ports still require gnu-gcc to compile. Invoke WITHOUT_GCC= yes only if you plan to use a higher version of gcc for your ports.
 
It will become the default compiler in CURRENT but it's not yet done I believe.

It's a good idea to leave GCC in base because of some potential hickups in buildworld. Just recently it was impossible to compile parts of the boot code with clang(1) for a while and if you don't have GCC as the fallback compiler you can't finish the build until the problem is fixed in SVN.
 
@xeube: Yes, clang is default for HEAD only. Its announcement as default is recent. Before that, it was only targeted / planned as default.

@kpa: The ride is certainly far from smooth...
 
UPDATE: I just finished buildworld with settings:
Code: 
WITHOUT_GCC= yes
WITH_CLANG= yes
WITH_CLANG_EXTRAS= yes
WITH_CLANG_IS_CC= yes
However, ccache does not always play well with clang and needs to be disabled if build stops.
This means, start with ccache enabled, but if build stops, disable ccache and <threads> then re-start, OK?

@ ColdfireMC: Olivier Smedts informed me of below - your call:
buildworld is ~= 10% faster with a clang-compiled kernel + world than with a stock (old 4.2) gcc-compiled kernel+world. I'm not speaking of the compiler speed but really the speed of the same build when run under a clang- or a gcc-compiled base system.
Click to expand...
 
wblock@ said:
Supposed to compile faster and have better error messages. Also a BSD-compatible license.
Click to expand...

but...? are there problems about that?

I recently read that binaries are slightly faster, because of newer and better cpu support than gcc4.2 but is near the same versus newer(or not-so-newer) gcc versions

is that true?

well, stop depending on a GNU (or another license with compatiblity issues with BSD license and philosophy) is a huge step towards gaining community independence, keep it up :).
 
UPDATE: I built my clang-only world and kernel with:
Code: 
WITH_CLANG= yes
WITH_CLANG_EXTRAS= yes
WITH_CLANG_IS_CC= yes
WITHOUT_GCC= yes
CLANG_IS_CC is definitely needed if you decide to apply WITHOUT_GCC. I was however, forced to disable ccache at some point so as to continue the build from the point that buildworld broke (using -DNO_CLEAN). Then the build completed without any other problems. Running my clang-only world now :)
 
Last time I tested, ccache really did not make much of an improvement to clang buildworld times. If anyone wants to benchmark, clear the cache ( ccache -C) and run two timed buildworlds. The first should take about as long as without ccache, the second should show the improvement in speed.

Incidentally, setting CCACHE_COMPRESS=yes seems to have no speed penalties and fits more in the cache.
 
Where are all these variables defined? I have been trying to find them in /usr/src, but they're not there.
 
Pardon my ignorance, but $ man ccache and files in /usr/share/doc/ccache folder describe no such variables as USE_CCACHE or USE_CCACHE_CPP2, which is why I am asking. There is CCACHE_CPP2, however. I am not sure, if the approach described in this how-to gives the expected results, because the /usr/local/share/doc/ccache/ccache-howto-freebsd.txt suggests a different way, to use devel/ccache:

Code: 
#
# $FreeBSD: head/devel/ccache/files/ccache-howto-freebsd.txt.in 300896 2012-07-14 13:54:48Z beat $
#

To use ccache add the following to /etc/make.conf.
You can replace cc and c++ with the compilers of your choice.
(remember that only GCC and Clang can build world and kernel)

.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*))
.if !defined(NOCCACHE)
CC:=${CC:C,^cc,/usr/local/libexec/ccache/world/cc,1}
CXX:=${CXX:C,^c\+\+,/usr/local/libexec/ccache/world/c++,1}
.endif
.endif

For Korn/Bourne shells Add the following to /etc/profile:
 export PATH=/usr/local/libexec/ccache:$PATH
 export CCACHE_PATH=/usr/bin:/usr/local/bin

For csh/tcsh Add the following to /etc/csh.cshrc:
 setenv PATH /usr/local/libexec/ccache:$PATH
 setenv CCACHE_PATH /usr/bin:/usr/local/bin

For icc users:
 Add /usr/local/intel_cc_80/bin to CCACHE_PATH

--

To use distcc:

For Korn/Bourne shells Add the following to /etc/profile:
 export CCACHE_PREFIX=distcc
 export DISTCC_HOSTS="localhost host1 host2"

For csh/tcsh Add the following to /etc/csh.cshrc:
 setenv CCACHE_PREFIX distcc
 setenv DISTCC_HOSTS "localhost host1 host2"

--

If you have a problem building world
define NOCCACHE and try again.

If you have a problem building a port
reset PATH=$CCACHE_PATH and try again.
...

Or am I missing something here?
 
wblock@ said:
Last time I tested, ccache really did not make much of an improvement to clang buildworld times. If anyone wants to benchmark, clear the cache ( ccache -C) and run two timed buildworlds. The first should take about as long as without ccache, the second should show the improvement in speed.

Incidentally, setting CCACHE_COMPRESS=yes seems to have no speed penalties and fits more in the cache.
Click to expand...

Yes, that's what I thought, because I didn't see any difference. So now, I made two consecutive buildworlds, but the impact was really great! I don't have sysutils/bsdadminscripts installed, I didn't set the USE_CCACHE or USE_CCACHE_CPP2 variables. What I did is described below.

I installed devel/ccache with these config flags enabled:

Code: 
# make -C /usr/ports/devel/ccache pretty-print-config
+CLANGLINK +DOCS +LLVMLINK -STATIC

My make.conf and my src.conf. Notice that I don't have CCACHE_CPP2 enabled. After that I added these two variables as the /usr/local/share/ccache/ccache-freebsd.txt suggested:

Code: 
# echo "setenv CCACHE_PATH /usr/bin:/usr/local/bin" >> ~/.tcshrc
# echo "setenv CCACHE_LOGFILE ~/.ccache.log" >> ~/.tcshrc
# echo "setenv CCACHE_COMPRESS 1" >> ~/.tcshrc

I decided to keep a log, if something goes wrong, and to confirm that I am really using ccache to do the job, and to make compression (it really paid off -- from ~600 MB it reduced cache size to 80 KB).

After that I ran
Code: 
# cd /usr/src
# time make -j16 buildworld
# time make -j16 buildworld

The first time it took me ~40 min to build (from 22:24 to 22:04). The second time it took considerably less: ~13 minutes (from 22:14 to 22:27).
 
I use a slightly different entry in make.conf:
Code: 
.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*)) && !defined(NOCCACHE)
  CC:=${CC:C,^cc,/usr/local/libexec/ccache/world/cc,1}
  CXX:=${CXX:C,^c\+\+,/usr/local/libexec/ccache/world/c++,1}
.endif

The idea being to avoid the overhead of the extra "if" by short-circuiting. I have not tested whether that's actually an improvement.
 
Thank you, that definitely makes it more readable, but I don't think it has an impact on performance, since I think this file is read only once on the beginning (by accident I once changed the compiler in it during compilation and nothing changed until the compilation ended).

UPDATE:
I noticed, after playing with devel/ccache that world can easily be built without CCACHE_CPP2 option, but the kernel will fail (fun fact).

But what does the devel/ccache actually do? I read through the manual but I don't quite understand how does saving time or binary content speed up the compiling process. Does it reuse old binaries?
 
"Compilation" in this context means only the pre-processor macro definitions that are used abundantly in C. Caching them provides quite significant savings in compilation times because the definitions can be read from a hash table in nearly constant time.
 
Oh, thanks @kpa! That's what was puzzling me, I thought that it was saving compilation results.
 
Last edited by a moderator:
You must log in or register to reply here.
Back
Top