Build clang on FreeBSD

A

ahev

Hello everyone, I would like to build clang on FreeBSD. I know that it requires much memory (RAM). I was advised to have two swap files or partition swaps. Not one big swap, but two swaps. Can I ask if there is an advantage to having such a configuration? Can someone explain why it is more useful / efficient? It may seems that one big swap partition or swap file is enough. Thanks..
 
@dadvv said that having 2gb of RAM and two gigabytes of swap is enough to compile and build clang in the base system. Right now I am trying to make buildworld. I will check if it is possible to compile that with only 8 GB of RAM and no swap.
 
I would also like to know if I can make an incremental build of the in tree clang compiler? Say `cd path/to/clang` and `make`? I tried to do so and failed, because I had not run a tablegen before, therefore I started a `make buildworld `. But the question is what do I need to run to get a MINIMAL build of clang? Thanks.
 
The LLVM/clang in the base tree isn't really meant for building independently of the rest of the tree.

What you probably want is provided instead in one of the devel/llvm* ports.
 
Yes, but the problem is that I am not sure if I can build devel/llvm* ports in theory even. That's because I have tried to do so in Linux and I failed. I do not understand why I am able to build the in-tree version (I have already finished building clang/llvm in buildworld task so it is true) and not able to build the main branch of llvm/clang on Linux (I am suggesting that on FreeBSD the build process will be the same).
 
It's not difficult to build llvm/clang as downloaded on Linux or FreeBSD, I've done so quite a few times. Their documentation and the CMake files contain everything you need.
Nonetheless it's probably a good time to ask you why you're trying to build it outside of the ports framework (or build it at all). I'm guessing you're new to FreeBSD and don't know what a port actually is?
 
Yes, I have never worked with ports.

I would like to *develop* lldb debugger in the first place.

The problem with just building is that I have 8 gigabytes of RAM and no swap partition or file whatsoever, and I am not sure if that is possible with such configuration at all.
 
It should be no problem building LLVM with 2 GB RAM. Just set -j to 1 and wait a long time for it to finish.

Multiple swap areas make sense if they are on different disks, then things go faster overall. They need to have the same priority.
 
ahev said:
The problem with just building is that I have 8 gigabytes of RAM and no swap partition or file whatsoever, and I am not sure if that is possible with such configuration at all.
Click to expand...
Why you think that with 8GB RAM the compilation cannot be done? What if you have 4GB RAM + 4GB swap.
 
There's no way you'll get away with compiling LLVM with just 2 RAM and at least 1G (probably 2-3Gbyte is more realistic swap) without disabling flang even with just one job running.
 
6502 said:
Why you think that with 8GB RAM the compilation cannot be done? What if you have 4GB RAM + 4GB swap.
Click to expand...
I have actually tried compiling FreeBSD from scratch on a laptop that has 8GB of RAM... and while 2 GB of swap is OK, 8GB of RAM is really pushing it when compiling LLVM 12. LLVM stands at 17 or 18 these days. 16 GB of RAM is adequate for compiling that, you gotta be generous with disk space, as well, and to remember to run make clean when done with compilation.

Compilers are generally not the smartest things around when managing available resources while compiling. An out-of-control compilation job can make the whole machine crash, which is why devs tend to favor machines with really beefy and roomy specs.
 
All those RAM numbers for builds don't mean anything without the information of how many compile proceses you run in parallel (make -j <n>).
 
I can not understand the root cause of your anxiousness about compiling with 8GB dram memory and no swap space FreeBSD partition. Add a USB SSD, via SanDisk extreme SSD disk. Place a swap partition there or create a swap file on existing disk space. I did run make build_world and make build_kernel on a Raspberry Pi 4B with 8 gigabytes memory and 8 gigabytes of swap partition space.
You assert having disk space available for the compilation, CORRECT? Then add a single SWAP file of size 2GB or larger like 4GB, or 8GB.
https://docs.freebsd.org/en/books/handbook/config/#create-swapfile
https://www.cyberciti.biz/faq/create-a-freebsd-swap-file/ May 28, 2024 Brand new post for you.
So i’d say, having swap partition/drive is nice if you do not over commit memory usage, system will not become slow due to swapping.

root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # mount
arm64pool/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs)
/dev/da0p1 on /mnt/esp (msdosfs, local)
arm64pool/tftpboot on /tftpboot (zfs, local, noatime, nfsv4acls)
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # df -h
Filesystem Size Used Avail Capacity Mounted on
arm64pool/ROOT/default 763G 345G 419G 45% /
devfs 1.0K 0B 1.0K 0% /dev
/dev/da0p1 3.0G 30M 3.0G 1% /mnt/esp
arm64pool/tftpboot 5.0G 824M 4.2G 16% /tftpboot
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapctl
usage: swapctl [-AghklmsU] [-a file ... | -d file ...]
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapctl -l
Device: 1024-blocks Used:
/dev/gpt/swap 8388608 1828072
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos #
Click to expand...
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapctl -l
Device: 1024-blocks Used:
/dev/gpt/swap 8388608 1828072
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # cat /etc/fstab
# /dev/gpt/rootfs / zfs rw,noatime 1 1
# / dev/gpt/esp /boot/efi msdosfs rw,noatime 0 0
/dev/gpt/swap none swap sw 0 0
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo
Device 1K-blocks Used Avail Capacity
/dev/gpt/swap 8388608 1828072 6560536 22%
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo -g
Device 1G-blocks Used Avail Capacity
/dev/gpt/swap 8 1 6 22%
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo -M
swapinfo: option requires an argument -- M
usage: swapinfo [-ghkm] [-M core [-N system]]
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo -m
Device 1M-blocks Used Avail Capacity
/dev/gpt/swap 8192 1785 6406 22%
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos #


Wish you well, share back what you eventually do, that works for your needs.
Click to expand...
I also ran Poudriere package builder on the Raspberry Pi 4B with 8 gb dram.
https://ghostbsd-arm64.blogspot.com/2023/10/poudriere-setup-to-compile-from.html
https://ghostbsd-arm64.blogspot.com/2023/10/poudriere-simple-tips-on-path-ccache.html
https://ghostbsd-arm64.blogspot.com/2023/07/learning-zfs-on-root-setup-for.html

https://ghostbsd-arm64.blogspot.com/2024/03/compile-gdb-for-aarch64-target-to-use.html
Some of these parameters might be useful to consider when build LLDB.


Wish you good success on your building LLDB also!!
 
cracauer@ said:
All those RAM numbers for builds don't mean anything without the information of how many compile proceses you run in parallel (make -j <n>).
Click to expand...
I avoid using the -j flag when compiling ports. That's because make(1) is not smart enough to manage dependencies in such situations. You have to be absolutely sure deps are satisfied BEFORE even trying to use the -j flag. Otherwise the compilation will crash due to race conditions.
 
wb7odyfred said:
I can not understand the root cause of your anxiousness about compiling with 8GB dram memory and no swap space FreeBSD partition. Add a USB SSD, via SanDisk extreme SSD disk. Place a swap partition there or create a swap file on existing disk space. I did run make build_world and make build_kernel on a Raspberry Pi 4B with 8 gigabytes memory and 8 gigabytes of swap partition space.
You assert having disk space available for the compilation, CORRECT? Then add a single SWAP file of size 2GB or larger like 4GB, or 8GB.
https://docs.freebsd.org/en/books/handbook/config/#create-swapfile
https://www.cyberciti.biz/faq/create-a-freebsd-swap-file/ May 28, 2024 Brand new post for you.
So i’d say, having swap partition/drive is nice if you do not over commit memory usage, system will not become slow due to swapping.
Click to expand...
Hi! First of all, thanks so much for your detailed answer! Let's make this thread a community effort for everyone running on low RAM and willing to compile whether lldb/llvm/clang or run something as big as buildworld or buildkernel.

So let's start writing out what we have.
1. 8 Gbytes of RAM and I am ready to avoid using parallel builds, so I will use Makefiles and -j1.
2. Storage is not a problem, as I have 1TB of HDD.
3. For swap space, an external SSD is faster than USB sticks and HDDs, right? So we filter out these two options.
4. You mention memory over commitment. Does it mean using e. g. -j100 for make?

Given these factors, I am considering bying an external SSD. I will continue this thread when I am ready to share my results. Thank you!
 
diizzy said:
There's no way you'll get away with compiling LLVM with just 2 RAM and at least 1G (probably 2-3Gbyte is more realistic swap) without disabling flang even with just one job running.
Click to expand...
Do you mean that I can disable flang and still compile clang/lldb?
 
diizzy said:
Yes? https://cgit.freebsd.org/ports/tree/devel/llvm15/Makefile#n96 (also an option in later versions)
Click to expand...
How do I make use of that? Could you write an instruction that I can follow to build clang and lldb w/ flang disabled? Thanks.
wb7odyfred said:
I can not understand the root cause of your anxiousness about compiling with 8GB dram memory and no swap space FreeBSD partition. Add a USB SSD, via SanDisk extreme SSD disk. Place a swap partition there or create a swap file on existing disk space. I did run make build_world and make build_kernel on a Raspberry Pi 4B with 8 gigabytes memory and 8 gigabytes of swap partition space.
You assert having disk space available for the compilation, CORRECT? Then add a single SWAP file of size 2GB or larger like 4GB, or 8GB.
https://docs.freebsd.org/en/books/handbook/config/#create-swapfile
https://www.cyberciti.biz/faq/create-a-freebsd-swap-file/ May 28, 2024 Brand new post for you.
So i’d say, having swap partition/drive is nice if you do not over commit memory usage, system will not become slow due to swapping.


root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapctl -l
Device: 1024-blocks Used:
/dev/gpt/swap 8388608 1828072
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # cat /etc/fstab
# /dev/gpt/rootfs / zfs rw,noatime 1 1
# / dev/gpt/esp /boot/efi msdosfs rw,noatime 0 0
/dev/gpt/swap none swap sw 0 0
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo
Device 1K-blocks Used Avail Capacity
/dev/gpt/swap 8388608 1828072 6560536 22%
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo -g
Device 1G-blocks Used Avail Capacity
/dev/gpt/swap 8 1 6 22%
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo -M
swapinfo: option requires an argument -- M
usage: swapinfo [-ghkm] [-M core [-N system]]
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos # swapinfo -m
Device 1M-blocks Used Avail Capacity
/dev/gpt/swap 8192 1785 6406 22%
root@Ghost14-selfbuilt-rpi4B-nginx-tst1:/usr/local/etc/pkg/repos #



I also ran Poudriere package builder on the Raspberry Pi 4B with 8 gb dram.
https://ghostbsd-arm64.blogspot.com/2023/10/poudriere-setup-to-compile-from.html
https://ghostbsd-arm64.blogspot.com/2023/10/poudriere-simple-tips-on-path-ccache.html
https://ghostbsd-arm64.blogspot.com/2023/07/learning-zfs-on-root-setup-for.html

https://ghostbsd-arm64.blogspot.com/2024/03/compile-gdb-for-aarch64-target-to-use.html
Some of these parameters might be useful to consider when build LLDB.


Wish you good success on your building LLDB also!!
Click to expand...
Alright, I cannot plug into my laptop, which is Lenovo 100 15ibd, anything with USB 3.1. There is no port for it. Only USB 3.0. I guess I will try to have a swap on my partitioned HDD.
 
I have also tried to do `buildworld` with 8GiB of RAM and it crashed with the following error:

--- all_subdir_usr.bin/clang/lldb ---
c++: error: unable to execute command: Killed
c++: error: linker command failed due to signal (use -v to see invocation)
2322.21 real 2732.12 user 187.20 sys

Can someone explain why this happened? I am also going to have a swap partition now with 8 GiB of RAM and try again.
 
I was able to buildworld in 2312 seconds, ncups: 4 with a swap file. Thanks! Next, I will try to build llvm/clang using their official instructions.
 
You must log in or register to reply here.
Back
Top