Solved How to cross compile port in AMD64 for ARMV7?

rickyzhang

New Member

Reaction score: 2
Messages: 13

I can't find a good solution to patch a broken port for pfSense which runs on ARMV7 chip.

I already tried/researched the following approaches:

  • Emulate ARMV7 pre-built image in QEMU. I can't make it boot out-of-the-box. It seem that I need to hack the kernel.
  • Find a RPI2 to build it. It is too slow. I built a open-cv in Odroid XU4, which is much faster than RPI2. But it took me a day to compile.
  • Read the cross build wiki. I tried the flag TARGET=arm TARGET_ARCH=armv6. But it doesn't work. It is still amd64 binary.
Code:
[Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ make  -j8 package TARGET=arm TARGET_ARCH=armv6

[Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ file work/stage/usr/local/bin/barnyard2
work/stage/usr/local/bin/barnyard2: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, FreeBSD-style, stripped
How can you guys build port for your embedded device?
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

Where can I get the source code of FreeBSD tool chain?

I pull from svn and then build like the Russian blog instructed.

Code:
svn checkout https://svn.freebsd.org/base/releng/`uname -r | cut -d'-' -f1,1` /usr/src
I did saw some dynamic library built as ARM architecture. But the binary folder only has the following:

Code:
Ricky@freebsd /usr/obj/arm.armv6/usr/src/tmp/usr/bin $ ls -alh
total 14376
drwxr-xr-x   2 root  wheel   512B Aug  5 08:43 .
drwxr-xr-x  13 root  wheel   512B Aug  5 08:38 ..
-rwxr-xr-x   1 root  wheel   781K Aug  5 08:42 addr2line
-rwxr-xr-x   1 root  wheel   1.7M Aug  5 08:42 as
-rwxr-xr-x   1 root  wheel   972K Aug  5 08:43 ctfconvert
-rwxr-xr-x   1 root  wheel   839K Aug  5 08:43 ctfmerge
-rwxr-xr-x   2 root  wheel   1.5M Aug  5 08:42 ld
-rwxr-xr-x   2 root  wheel   1.5M Aug  5 08:42 ld.bfd
-rwxr-xr-x   1 root  wheel   808K Aug  5 08:42 nm
-rwxr-xr-x   2 root  wheel   1.0M Aug  5 08:42 objcopy
-rwxr-xr-x   1 root  wheel   1.6M Aug  5 08:42 objdump
-rwxr-xr-x   1 root  wheel   610K Aug  5 08:42 size
-rwxr-xr-x   1 root  wheel   601K Aug  5 08:42 strings
-rwxr-xr-x   2 root  wheel   1.0M Aug  5 08:42 strip
-rwxr-xr-x   1 root  wheel   540K Aug  5 08:43 sysinit
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

Thanks, I found clang under usr.bin folder.

I'm a Linux guy. I want to patch a broken outdated port Barnyard2 from my recent purchased pfsense router SG-3100 which runs on ARM.

I can't find any latest documentation on how to setup cross build tool chain in FreeBSD. The Russian blog method doesn't work. Because I check the /usr/src/Makefile. The target has been renamed from toolchain to toolchains. When I tried to make it like below:

Code:
TARGET=arm
TARGET_ARCH=armv6
SRC_TREE=/usr/src

cd ${SRC_TREE}
make -j 8 TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} TARGET_CPUARCH=${TARGET} toolchains
It builds all platform below in my /usr/obj:
Code:
    amd64/amd64
    arm/arm
    arm/armeb
    arm/armv6
    arm64/aarch64
    i386/i386
    mips/mipsel
    mips/mips
    mips/mips64el
    mips/mips64
    mips/mipsn32
    pc98/i386
    powerpc/powerpc
    powerpc/powerpc64
    sparc64/sparc64
It doesn't sound right to me.

The cross compile wiki mentioned building cross compile tool chain. But it requires to buildworld step(I suspect it means all ports. Am I right? What is the world in FreeBSD?)

In any case, I found xdev in /usr/src/Makefile. It seems this might be the right target.

I also found people mention using poudriere jail with qemu. I don't know exactly the technical details. I can't find any doc, either. It sounds like running ARM emulation in AMD64. That doesn't seem like a cross compiler. Correct me if I'm wrong.

Cross development process in FreeBSD is not as smooth as Linux.

Disclaimer: I'm not seeking any tech support for my router. I'm just a hobbyist who want to patch my own stuffs.
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

I figured out how to do that in poudriere jail to compile target to ARM in AMD64 platform. But this is not cross compile at all.

TBH, you guys misled the whole world. It should not be called cross compile! All binary files in the armv6 jail is ARM binary. The poudriere bulk is actually emulating ARM version tool chains in AMD64 by qemu. The build speed is slow like a sloth in DMV.

Why can't we use cross compile tool chain? We already have this cross compiler. We can override cc in AMD64 poudriere jail.

In case somebody needs an up-to-date reference, I shared my notes here about how to cross compile a port..
 

Phishfry

Son of Beastie

Reaction score: 1,395
Messages: 4,110

Poudriere is a good way to go. There are lots of tutuorial around the internet.

It builds all platform below in my /usr/obj:
It doesn't sound right to me.
I don't see the problem. You have to use a make.conf to specify that it ONLY builds for a certain platform.
xdev is not required but will speed up your build times.


To set those variables on tcsh:
setenv TARGET arm
setenv TARGET_ARCH armv6
pkg install autoconf automake libtool <<<< These packages are needed to build barnyard2 <<<<
cd /usr/src
make -j3 TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} TARGET_CPUARCH=${TARGET} toolchain

make -j3 TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} TARGET_CPUARCH=${TARGET} buildenv

Entering world for armv6:arm
cd /usr/ports/security/barnyard2
make
 

acheron

Aspiring Daemon

Reaction score: 217
Messages: 548

I figured out how to do that in poudriere jail to compile target to ARM in AMD64 platform. But this is not cross compile at all.

TBH, you guys misled the whole world. It should not be called cross compile! All binary files in the armv6 jail is ARM binary. The poudriere bulk is actually emulating ARM version tool chains in AMD64 by qemu. The build speed is slow like a sloth in DMV.

Why can't we use cross compile tool chain? We already have this cross compiler. We can override cc in AMD64 poudriere jail.

In case somebody needs an up-to-date reference, I shared my notes here about how to cross compile a port..
You should use native-xtools to speed up builds with qemu.
clang is already a cross compiler, not sure why you talk about building a cross compiler.
 

Phishfry

Son of Beastie

Reaction score: 1,395
Messages: 4,110

My amd64 - Arm target build failed in /devel/gmake while trying out /security/barnyard2

Code:
checking whether we are cross compiling... configure: error: in `/usr/ports/devel/gmake/work/make-4.2.1':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
===>  Script "configure" failed unexpectedly.
I think the idea of compiling on Pi2 was a valid one. It might take 3 days but it is native.
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

Phishfry

Can you elaborate how do you use cross compile tool chain? I already built one in /usr/armv6-freebsd.

I'm new to FreeBSD. It is fairly easy to replace tool chain in Linux build. However, the FreeBSD Port Makefile and patching over the original Makefile project make it more complicate in FreeBSD. The FreeBSD porter documentation doesn't explain how to do cross compile.

acheron

Here is my step to use poudriere:

1. Create a poudriere jail.

Code:
sudo poudriere jails -c -j pfsense-port-11-2-armv6 -a arm.armv6 -m svn -v release/11.2.0
2. Create a poudriere local ports tree

Code:
sudo poudriere ports -c -f none -m null -M /home/Ricky/repo/github/pfsense/FreeBSD-ports -p pfsense
3. Build Barnyard2 package

Code:
sudo poudriere bulk -j pfsense-port-11-2-armv6 -p pfsense -f /usr/local/etc/poudriere.d/pfsense-port-11-2-armv6-pfsense-pkglist
The file /usr/local/etc/poudriere.d/pfsense-port-11-2-armv6-pfsense-pkglist only includes one line: security/barnyard2.

The performance really sucks even in my 8 cores i9-9900K CPU.

I enabled the following in /usr/local/etc/poudriere.conf.

Code:
PARALLEL_JOBS=8
ALLOW_MAKE_JOBS=yes
But after 30 minutes run, it is still compiling dependencies of Barnyard2 even with the help from parallelization. Compared to AMD64 native compile, it took me less than 10 minutes to compile Barnyard2 and its dependencies at the first time.

I also found that the compiler cc in the armv6 architecture is actually ARM binary and when poudriere bulk is working, it runs qemu-static-arm with cc ARM binary below. That means when we build anything inside the arm.armv6 jail, we compile everything in ARM emulation under AMD64 platform. It is not going to be fast compared to real cross compile. We should never call this approach "cross compile". I'm surprised that FreeBSD community abuse the term cross compile in this situation.

Code:
Ricky@freebsd ~ $ file /usr/local/poudriere/jails/pfsense-port-11-2-armv6/usr/bin/cc

/usr/local/poudriere/jails/pfsense-port-11-2-armv6/usr/bin/cc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (FreeBSD), statically linked, for FreeBSD 11.2, FreeBSD-style, stripped

Can you explain what native-xtools is? How can I use it in detailed step?
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

malavon

It doesn't sound like "cross compiling" to me. Again, don't abuse the term cross compile. It is emulation.

Code:
 -x          Build    the native-xtools target using the host's /usr/src
          tree and copy    this into the jail.  The use of    /usr/src is
          due to a bug in the native-xtools build which    does not allow
          it to    be built from the jail's own source.  Used exclusively
          for cross building a ports set, typically via    qemu-user
          tools.
 

malavon

Active Member

Reaction score: 117
Messages: 197

It doesn't sound like "cross compiling" to me. Again, don't abuse the term cross compile. It is emulation.
You must have misread something. I never mentioned cross-compilation. I answered a simple question you posed.
That said, I don't cross-compile anything myself. I just compile natively, or emulated if I have to so I can't help you with that part.
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

For using 2nd approach a.k.a emulation, I already figured this out. I wrote that down in my notes. Thanks!

But as I said, it is slow.

I wonder if anyone have try to do the following:

1. Build cross compiler like in 1st approach in my notes here.

2. Create a poudriere jail in AMD64.

3. Copy cross compiler into the jail (I admit I make this up. I don't know if it can work or not)

4. Force make.conf use cross compiler.

In this way, we can speed things up.
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

You should use native-xtools to speed up builds with qemu.
clang is already a cross compiler, not sure why you talk about building a cross compiler.
You are correct about clang. It does supports cross compile to ARM target platform from AMD64.

I saw that when it were building an arm.armv6 architecture jail, it cross compile all source code into ARM binary under AMD64 platform.

But why should we run ARM binary jail then to build the port for ARM target platform? It makes no sense to me to waste our precious development time. Why not do the same in port build?
 

acheron

Aspiring Daemon

Reaction score: 217
Messages: 548

But why should we run ARM binary jail then to build the port for ARM target platform? It makes no sense to me to waste our precious development time. Why not do the same in port build?
Because every ports has its own notion of cross-compiling. I've bootstrapped java, ghc and rust (on 4 differents arches) over the year and none of them do things the same way. There are tons of 'build tools' (autotools, cmake, scons etc) that does thing differently, sometimes the port itself needs to run some natives commands (I don't remember the port name but it happens). We still use gcc on powerpc, and gcc is not a cross-compiler (last time I checked: 5 years ago). So you'll have to put a huge amount of work/kludge/workaround for only 2 arches where qemu is involved (armv6/7 and mips, we build on real hardware on others arches to stress tests the system).
If you are interested, you can pick this project and improve cross-compiling on FreeBSD.
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

Regarding to improve cross platform build process, I think we don't have to use all or nothing strategy. Some of the build are definitely can be done by cross compiling rather slow emulation. I did cross compile Linux kernel target to ARM. But I never did that for the application in rootfs. I only blindly copy rootfs from somewhere. TBH, I really don't know how Fedora Koji use RPM to cross compile its application. So I will poke around and see how difficult the task is.

My current goal is to patch Barnyard2. I will definitely come back on this. Thanks!
 

ralphbsz

Daemon

Reaction score: 1,160
Messages: 1,870

... and gcc is not a cross-compiler ...
Gcc has always been capable of cross-compiling, at least theoretically. I say "at least theoretically" because configuring it for cross-compile is exceedingly difficult. So much so that in many cases the advice has been to not even try, and instead build a native environment to compile in. Even if that requires running on a CPU simulator, it may still be more efficient.
 

kpedersen

Daemon

Reaction score: 535
Messages: 1,438

I had a pretty decent experience at creating a FreeBSD armv6 jail.

A lot of the info can be found in this thread. In particular this specific post was very useful


(Not sure if this helps. You mention armv7 but a lot of what you have tried seems to reference armv6)
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

kpedersen

I figured this out already. My pfsense router CPU is ARM v7 Cortex-A9. But it can runs on ARMV6 architecture.

I documented how to setup FreeBSD development env and use poudriere ARM jail to build a port for ARM target in emulation. See my wiki


Regarding to cross compiling to ARM, I found a blog which implemented my ideas in my previous thread. The author claimed cross compiling is 6x faster than emulation.

Unless we have something like Golang built which requires a lower version Golang to bootstrap, majority of port can use cross compile approach. We can add a flag in Makefile to mark those outliers. Only run QEMU emulation for those.
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

That's just what native-xtools do...
I added -x flag when creating jail for poudriere.

Code:
sudo poudriere jails -c -x -j pfsense-port-11-2-armv6-native-xtools -a arm.armv6 -m svn -v release/11.2.0
Once it is done, I can't find any xtools or native thingy in the jail

Code:
Ricky@t30-freebsd /usr/local/poudriere/jails/pfsense-port-11-2-armv6-native-xtools/usr $ find . | grep xtool
./src/crypto/heimdal/lib/hx509/hxtool-commands.in
./src/crypto/heimdal/lib/hx509/hxtool.c
./src/kerberos5/usr.bin/hxtool
./src/kerberos5/usr.bin/hxtool/Makefile.depend
./src/kerberos5/usr.bin/hxtool/Makefile
./bin/hxtool
./lib/debug/usr/bin/hxtool.debug
Ricky@t30-freebsd /usr/local/poudriere/jails/pfsense-port-11-2-armv6-native-xtools/usr $ find . | grep native
./src/lib/libkvm/kvm_native.3
./src/sys/contrib/octeon-sdk/cvmx-access-native.h
./src/sys/powerpc/aim/moea64_native.c
./src/contrib/elftoolchain/common/native-elf-format
./src/share/doc/papers/malloc/alternatives.ms
./src/gnu/usr.bin/cc/cc_tools/freebsd-native.h
./share/man/man3/kvm_native.3.gz
Do you know any doc about this native-xtool thing? I can't Google it.
 

acheron

Aspiring Daemon

Reaction score: 217
Messages: 548

I added -x flag when creating jail for poudriere.

Code:
sudo poudriere jails -c -x -j pfsense-port-11-2-armv6-native-xtools -a arm.armv6 -m svn -v release/11.2.0
Once it is done, I can't find any xtools or native thingy in the jail

Code:
Ricky@t30-freebsd /usr/local/poudriere/jails/pfsense-port-11-2-armv6-native-xtools/usr $ find . | grep xtool
./src/crypto/heimdal/lib/hx509/hxtool-commands.in
./src/crypto/heimdal/lib/hx509/hxtool.c
./src/kerberos5/usr.bin/hxtool
./src/kerberos5/usr.bin/hxtool/Makefile.depend
./src/kerberos5/usr.bin/hxtool/Makefile
./bin/hxtool
./lib/debug/usr/bin/hxtool.debug
Ricky@t30-freebsd /usr/local/poudriere/jails/pfsense-port-11-2-armv6-native-xtools/usr $ find . | grep native
./src/lib/libkvm/kvm_native.3
./src/sys/contrib/octeon-sdk/cvmx-access-native.h
./src/sys/powerpc/aim/moea64_native.c
./src/contrib/elftoolchain/common/native-elf-format
./src/share/doc/papers/malloc/alternatives.ms
./src/gnu/usr.bin/cc/cc_tools/freebsd-native.h
./share/man/man3/kvm_native.3.gz
Do you know any doc about this native-xtool thing? I can't Google it.
in the src dir: https://svnweb.freebsd.org/base/head/Makefile.inc1?revision=349229&view=markup#l2519
in poudriere: https://github.com/freebsd/poudriere/blob/master/src/share/poudriere/common.sh#L2234
You should have a nxb-bin directory in your poudriere jail and a etc/make.nxb.conf file.
 
OP
OP
rickyzhang

rickyzhang

New Member

Reaction score: 2
Messages: 13

I got it now.

Code:
Ricky@gtx-freebsd /usr/local/poudriere/jails/pfsense-port-11-2-armv6-native-xtools/nxb-bin/usr/bin $ file cc
cc: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 11.2, FreeBSD-style, not stripped
R
The cross compiling is 6x faster than the emulation.

Thanks!
 
Top