C How to cross-compile ARM64 binary (custom code) from x64 system

[Fest1958]

Hello.

I would like to cross-compile a binary for ARM64 (aarch64) target from my x86-64 system. I don't have native ARM64 system at hand.

Target x86_64 works fine:

$ cc -target x86_64-unknown-freebsd -c -o platform.o platform.c

Target x86 works fine too:

$ cc -target i686-unknown-freebsd -c -o platform.o platform.c

However target aarch64 fails with strange error:

$ cc -target aarch64-unknown-freebsd -c -o platform.o platform.c
In file included from platform.c:13:
In file included from /usr/include/signal.h:43:
/usr/include/sys/_ucontext.h:46:2: error: unknown type name 'mcontext_t'
        mcontext_t      uc_mcontext;
        ^
1 error generated.

Looks like there is an error in the system header file. How can I fix this? Is it even possible, with FreeBSD, to cross-compile ARM64 binary form the x64 system?

Sorry, I'm more familiar with Linux. And, on Linux, installing the crossbuild-essential-arm64 package gives me cross-compiler that will happily create ARM64 binary on the x64 system.

Is there something similar for FreeBSD platform?

Thank you and best regards!


My system is:

FreeBSD  13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64

 

[SirDice]

This should have some useful information: https://wiki.freebsd.org/arm/crossbuild

 

[Fest1958]

SirDice, thank you for your reply.

But this seems to be information on how to cross-build FreeBSD code itself, which uses "the FreeBSD build system".

However, what I need to do is cross-build my "own" code on FreeeBSD. I build straight from the command-line, not using any kind of "build system".

The only thing needed to start a cross-build instead of a native build is to add TARGET_ARCH= to the make command line

I think this only applies to Makefiles used to build FeeBSD itself. To build my "own" code, I need to know what exact command-line option I need to pass to cc (clang).

Using cc -target aarch64-unknown-freebsd probably is the equivalent of passing TARGET_ARCH to the FreeBSD Makefile, isn't it? What am I missing?

Sorry, I'm more used to Linux, but on Linux passing the proper -target option to clang works just fine — this way I can build ARM64 binary from the x64 system:

$ uname -a
Linux 6.2.0-20-generic #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ clang -target aarch64-linux-gnu main.c platform.c

$ file a.out
a.out: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1
👍

Can FreeBSD do this?

Thank you and best regards!

 

[zirias@]

Looks like there is an error in the system header file.

Almost. I don't think there's an error, but it's still the system header for X86 architectures.

You will probably need a build of FreeBSD base for your target architecture and then either build in a jail (using native-xtools to avoid having to emulate the compiler), or try to set up necessary include and library paths when cross-building on your host system.

 

[_martin]

Recently I showed an example here how one can build aarch64 binary on x86 FreeBSD (as opposed to the current issues with MIPS). Standard C library is there so chances are that's all you need.
It all depends on what your code uses. e.g. if it uses some nonstandard, 3rd party library linker won't be able to link your code against it (the same way as it can't do it natively if that library is missing).

 

[Fest1958]

Thank you for the info, _martin! Yes, this seems to work for me:

$ cc -target aarch64-unknown-freebsd --sysroot=/usr/local/freebsd-sysroot/aarch64 -c -o platform.o platform.c

 

[kpedersen]

One approach I would recommend is creating a FreeBSD arch64 Jail on your x86_64 host (binmiscctl + qemu-aarch64-static). Then do the compiling from there.

Unfortunately this thread is a little messy but does detail the steps. This could be useful if dealing with awkward build systems that don't handle cross compilers well.

 

[Fest1958]

kpedersen, thank you, but I think this goes way beyond my understanding. I'm not really familiar with "jails". Can you explain in more detail?

Anyway, I think that I can get along with --sysroot for now...

 

[kpedersen]

I'm not really familiar with "jails". Can you explain in more detail?

A Jail is really just a glorified chroot (so lets simplify by using that instead). All this does is create an environment where / is simply /the/chroot/directory. It just maps it.

This means that you can extract the FreeBSD base tarballs into that directory, "chroot into it" and then you have an isolated FreeBSD install within your original install. Just like a Virtual Machine apart from it shares the same kernel and native hardware (thus tends to be faster to "spin up" and for many applications at runtime.

The cool part is, if you extract a base tarball from another architecture (i.e aarch64) and try to chroot into that; it will fail because when trying to run i.e /bin/sh, this is for a different architecture. Wont work. However by copying across the correct qemu-static binary and a tweak to the OS to use it when a foreign architecture is detected, it will simply "just work".

So in short, the general commands would be:

# pkg install qemu-user-static
# sysrc qemu_user_static_enable=YES
# service qemu_user_static start

# tar -xf /path/to/aarch64/base.txz -C /my/aarch64/chroot
# cp /usr/local/bin/qemu-aarch64-static /my/aarch64/chroot/usr/local/bin

# chroot /my/aarch64/chroot/usr/local/bin

All the qemu_user_static service does is use binmiscctl(8) to tell the kernel to send foreign binaries through our qemu interpreter when it detects them. You can open up the rc.d script and see what it does if you want to do this step manually.

 

[cmoerz]

I found this in my personal notes from back in time when I did cross compile world and then used poudriere to compile aarch64 packages... hope this is of help to you.

1. Install qemu-user-static
2. make buildworld TARGET=arm64
3. make buildkernel TARGET=arm64
4. make installworld TARGET=arm64 DESTDIR=...
5. make distribution TARGET=arm64 DESTDIR=...
6. make installkernel TARGET=arm64 DESTDIR=...
7. sudo mkdir -p ${DESTDIR}/usr/local/bin
8. sudo cp /usr/local/bin/${EMULATOR} ${DESTDIR}/usr/local/bin/${EMULATOR}
9. sudo chroot -u root ${DESTDIR} /usr/local/bin/${EMULATOR} /bin/sh
10. Within system, do service ldconfig start

Further resources:
* https://wiki.freebsd.org/QemuUserModeHowTo
* https://wiki.freebsd.org/action/show/arm/crossbuild?action=show&redirect=FreeBSD/arm/crossbuild

QEMU​

Follow steps as outlined at https://wiki.freebsd.org/arm64/QEMU.

qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt -bios edk2-aarch64-code.fd -serial telnet::4444,server -nographic \
       -hda arm64vol -device virtio-net-pci,netdev=net0 -netdev tap,id=net0,ifname=tap1

This basically starts an aarch64 vm, which you can telnet to console. Once inside, you can compile, work etc. as if it's a native box. Word of warning: it's not particularly fast since it's emulation only.

You can also bless your arm64 binaries under amd64, which allows you to run aarch64 executables while actually operating under amd64. This worked for me, for example: https://forums.freebsd.org/threads/raspberry-pi-3-and-poudriere.63470/

 

[kent_dorfman766]

A little bit off the path but these are some things to consider:

1) use a Makefile! If nothing else it gives you the opportunity to supply configuration and target params in a more generic manner

2) set up a crossbuild toolchain on your platform such that the CC CXX CROSS_COMPILE commands are executed as
$(CROSS_COMPILE)$(CC) in the Makefile

3) CROSS_COMPILE is generally the prefix name of the platform compiler of your target: such as CROSS_COMPILE=arm-gnueabi-hf-

so that the Makefile sees it as arm-gnueabi-hf-cc when CROSS_COMPILE is set. This has the advantage of the crosschain binutils also following that naming convention such as arm-gnueabi-hf-ld or arm-gnueabi-hf-strip, etc.

Granted, most of this stuff I've done has been in the GNU or RTOS world so there may be different conventions under freeBSD I'm not aware of, but if I was attempting it under BSD this is the way I'd do it.