Solved ARM or SPARC in FreeBSD Jail on AMD64 host

kpedersen

Daemon

Reaction score: 589
Messages: 1,527

Hi all,

I am working on a small project to allow users to spin up a FreeBSD jail easily via a special SSH "shell" (https://github.com/osen/pavelock).
Currently, users can select between different versions of FreeBSD (9.x, 10.x, 11.x) and between x86 architectures (i386, amd64). The host itself is running on a typical amd64 machine.

Is there any way, via qemu-user-static or otherwise of being able to create and run a Jail for a completely different architecture such as ARM(64) or Sparc(64)? I see that poudriere has experimental support for building ports for different architectures but has anyone here had experience with running (or at least emulating) a full blown Jail?

Some info:
https://forums.freebsd.org/threads/building-arm-packages-with-poudriere-the-simple-way.52994/

Best regards,

Karsten
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,083
Messages: 31,689

No, you can't use jails for this because you can't run SPARC or ARM code natively on an AMD64 system. The QEMU trick that poudriere uses is an emulator, not a jail.
 
OP
OP
K

kpedersen

Daemon

Reaction score: 589
Messages: 1,527

Ah right. I assumed that because poudriere uses the qemu-user-static port rather than the standard qemu, even though it was still emulating the architecture, it was allowing it to integrate with the Jails system.

So is all that poudriere is doing is creating a clean FreeBSD (i.e Sparc) image, running that inside qemu i.e in the background and sending it build commands? How come it requires
Code:
qemu_user_static_enable="YES"
in /etc/rc.conf? Surely it must be doing something a little bit more interesting?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,083
Messages: 31,689

It leverages a neat trick with binmiscctl(8) to execute an ARM executable through the emulator. But you can always take a peek at the code for poudriere, it's actually a massive collection of shell scripts, so it's relatively easy to take it apart and see how it works.
 

tobik@

Daemon
Developer

Reaction score: 1,386
Messages: 1,909

It's really easy to do. Just sysrc qemu_user_static_enable=YES and service qemu_user_static start on the host and then copy /usr/local/bin/qemu-arm-static from the host to the same location in the jail.
 
OP
OP
K

kpedersen

Daemon

Reaction score: 589
Messages: 1,527

Ah, I remember that you can configure binmiscctl(8) to run win32 PE executables through wine in a similar way.
Ok, thanks I will have a scan through the poudriere scripts to see if I can pinch anything useful.

I don't imagine it will be as easy as running a sparc64 build of jexec via qemu-user-static but if I can make it appear to the users that it is smooth and clean, I guess I am happy ;).
 
OP
OP
K

kpedersen

Daemon

Reaction score: 589
Messages: 1,527

It's really easy to do. Just sysrc qemu_user_static_enable=YES and service qemu_user_static start on the host and then copy /usr/local/bin/qemu-arm-static from the host to the same location in the jail.
And this will allow me to jexec into the (extracted sparc base.txz) Jail and immediately run i.e the sparc /bin/ls binary? Wow, if this works, that is an incredibly underrated feature haha.
 

tobik@

Daemon
Developer

Reaction score: 1,386
Messages: 1,909

Hmm, not sure how good QEMU's SPARC emulation is, but it should work fine with aarch64 and armv{6,7}.
 
OP
OP
K

kpedersen

Daemon

Reaction score: 589
Messages: 1,527

Well, I must say, I am very impressed. It works very well for armv6 and aarch64. This seems like a really powerful feature.
I notice that since we are not using qemu to emulate an entire PC (kernel, screen, etc), it actually runs much better than expected. It seems to only take a fraction of time longer to compile up hello world in C than the native host.
Perfect, exactly what I was looking for. Thank you both for your help!

(not a massive issue but qemu-ppc-static (32-bit) unfortunately segfaulted due to a memory issue during the Jail startup. I will try some of the other architectures another time).

Some notes on utilizing an armv6 image for the Jail (since there is no base.txz for this arch unlike aarch64, amd64, etc). These instructions are only useful for testing. The img file would ideally need to be resized to make it useful. I think extracting it entirely is probably the better option.

Mounted "loop back" the armv6 Pi1 image.
Code:
# mdconfig -a -t vnode -u 0 -f FreeBSD-11.1-RELEASE-arm-armv6-RPI-B.img
Checked which partition had the FreeBSD one
Code:
# fdisk /dev/md0
Mounted it to my Jail dir
Code:
# mount -t ufs /dev/md0s2a /usr/jail/armtest

Make jail bin dir and copy qemu-arm-static across
Code:
# mkdir /usr/jail/armtest/usr/local/bin
# cp /usr/local/bin/qemu-arm-static /usr/jail/armtest/usr/local/bin
Started the jail (I required persist (unlike the i386 or amd64 jails or it would immediately terminate))
Code:
# jail -c persist name=armtest path=/usr/jail/armtest mount.devfs host.hostname=armtest.jail command=sh /etc/rc
 

acheron

Aspiring Daemon

Reaction score: 229
Messages: 603

No, you can't use jails for this because you can't run SPARC or ARM code natively on an AMD64 system. The QEMU trick that poudriere uses is an emulator, not a jail.
You're wrong, it's a jail.
 

acheron

Aspiring Daemon

Reaction score: 229
Messages: 603

(not a massive issue but qemu-ppc-static (32-bit) unfortunately segfaulted due to a memory issue during the Jail startup. I will try some of the other architectures another time).
ppc, ppc64 and sparc64 doesn't work due to qemu bugs.
 
OP
OP
K

kpedersen

Daemon

Reaction score: 589
Messages: 1,527

ppc, ppc64 and sparc64 doesn't work due to qemu bugs.
Yep, just tested them earlier today and whilst ppc crashed, the ppc64 and sparc64 just froze spinning at 100% cpu.

It seems a pity but I am happy that aarch64 and armv6 are working well.

I am also intrigued by qemu-system-cheri128 (in ports, developed as research project by University of Cambridge). It doesn't seem to have a qemu-cheri128-static counterpart but I may look further into it when I get some more free time.
 

acheron

Aspiring Daemon

Reaction score: 229
Messages: 603

I am also intrigued by qemu-system-cheri128 (in ports, developed as research project by University of Cambridge). It doesn't seem to have a qemu-cheri128-static counterpart but I may look further into it when I get some more free time
You'll have to port it to qemu-user-static.
 
OP
OP
K

kpedersen

Daemon

Reaction score: 589
Messages: 1,527

I thought I might just add here; the i386 port of qemu-user-static is marked as broken. However, if you remove the BROKEN_i386 flags in the Makefile and build the port it does still work for arm. Just not for any of the others (in this case aarch64).
 
Top