xen Can we virtualize FreeBSD as domU on Linux as dom0 on the Chromebook for arm 32 bit ? How ?

ziomario

ziomario

Hello to everyone.

we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepted upstream by FreeBSD, we will have to find his patches and apply them ourselves to the FreeBSD on arm kernel.

We found these slides :

https://events.static.linuxfound.org/sites/events/files/slides/Porting FreeBSD on Xen on ARM .pdf

Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what we want to find.

It looks like when that slide presentation was written, there were some limitations on FreeBSD Xen guests. For example, for our debian bookworm guest, I am using vcpus = '2' to match the number of real cpus on our Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD guest, so I will need to change that vcpus = '1' in the FreeBSD guest config unless support for 2 or more vcpus was added later, which is possible because that slide presentation is 9 years old.

Here is where I would expect to find the XENHVM FreeBSD on arm kernel config file:

https://cgit.freebsd.org/src/tree/sys/arm/conf

But it is not there so unless I am not understanding something correctly. For now unfortunately conclude that the support for Xen on arm that Julien Grall mentioned in that slide presentation 9 years ago was never added to the official FreeBSD source code. I am searching the web now to see if the patches that Julien Grall wrote are still posted somewhere online. If we cannot find them, we can ask here and on the xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the patches if we cannot find them online.

According to this page from the FreeBSD wiki:

https://wiki.freebsd.org/Xen

I think FreeBSD only supports Xen on x86, not arm. So this is going to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !

I found a slightly newer slide presentation by Julien here:

https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd

It is about the same, but it mentions the GENERIC FreeBSD kernel supports Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on arm 32 bit, which I haven't found online yet.
 
UPDATE :

According with this :

xen(4)

man.freebsd.org man.freebsd.org

To compile hardware-assisted virtualization (HVM) Xen guest support
with para-virtualized drivers into an amd64 or i386 kernel, place the
following lines in your kernel configuration file:

options XENHVM
device xenpci
Click to expand...

I tried :

Code: 
root@marietto:/usr/src # make -DNO_MODULES KERNCONF=GENERIC-xen TARGET=arm TARGET_ARCH=armv7 buildkernel 

[Creating objdir /usr/obj/usr/src/arm.armv7...]
make[1]: "/usr/src/Makefile.inc1" line 341: SYSTEM_COMPILER: libclang will be built for bootstrapping a cross-compiler.
make[1]: "/usr/src/Makefile.inc1" line 346: SYSTEM_LINKER: libclang will be built for bootstrapping a cross-linker.

--------------------------------------------------------------
>>> Kernel build for GENERIC-xen started on Tue Nov 21 14:32:33 CET 2023
--------------------------------------------------------------
===> GENERIC-xen
mkdir -p /usr/obj/usr/src/arm.armv7/sys

--------------------------------------------------------------
>>> stage 1: configuring the kernel
--------------------------------------------------------------
cd /usr/src/sys/arm/conf; PATH=/usr/obj/usr/src/arm.armv7/tmp/bin:/usr/obj/usr/src/arm.armv7/tmp/usr/sbin:/usr/obj/usr/src/arm.armv7/tmp/usr/bin:/usr/obj/usr/src/arm.armv7/tmp/legacy/usr/sbin:/usr/obj/usr/src/arm.armv7/tmp/legacy/usr/bin:/usr/obj/usr/src/arm.armv7/tmp/legacy/bin:/usr/obj/usr/src/arm.armv7/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin  config  -d /usr/obj/usr/src/arm.armv7/sys/GENERIC-xen -I '/usr/src/sys/arm/conf' -I '/usr/src/sys/arm/conf' '/usr/src/sys/arm/conf/GENE
RIC-xen'

/usr/src/sys/arm/conf/GENERIC-xen: unknown option "XENHVM"
*** Error code 1
 
Hello bro'.

thanks. I read it,already. But I wanted to try anyway because I'm not able to find more useful informations. And who knows,maybe the documentation is old :D
 
this is on 14
[titus@hp14 ~]$ grep -r XENHVM /sys/conf/
/sys/conf/options.amd64:XENHVM opt_global.h
/sys/conf/options.i386:XENHVM opt_global.h
so not in options.arm* as of now
 
Who is the maintainer ? I want to "speak" with him. I'm in touch with Julien Grall,I want also be in communication with the FreeBSD maintainer.
 
When I tried Xen on FreeBSD years ago you could do DOM0.
Also there was a port option to enable libvirt support versus default xl.

I see some recent commits too right from Citrix R&D.
This looks interesting:
While there also update SeaBIOS to 1.16.1.
Click to expand...
So this is probably thier version of the bhyve-UEFI firmare for EFI clients.
Instead of EDK2 they are using SeaBIOS..
 
When virtualizing anything, it's important to pay attention to how many bits the host (bare metal) architecture has. If the bare metal only has 32 bits (as in OP's case), then it's only possible to virtualize a 32-bit (or 16-bit) OS. Code for most recent relevant libs tends to be 64-bit, but sources can probably be patched and re-compiled for 32-bit architectures, if you're willing to spend the time and know what you're doing.
 
astyle said:
When virtualizing anything, it's important to pay attention to how many bits the host (bare metal) architecture has. If the bare metal only has 32 bits (as in OP's case), then it's only possible to virtualize a 32-bit (or 16-bit) OS. Code for most recent relevant libs tends to be 64-bit, but sources can probably be patched and re-compiled for 32-bit architectures, if you're willing to spend the time and know what you're doing.
Click to expand...

but in this scenario there isn't a port for arm64. I see only the code for i386,that's not arm.
 
We have found the link to Julien's FreeBSD Xen on arm tarball:

https://xenbits.xen.org/gitweb/?p=p...t;a=snapshot;h=refs/heads/xen-arm-v2.2;sf=tgz

I tried to compile that code on my FreeBSD X64 bit using these instructions :


this is what happened :

Code: 
sudo make TARGET_ARCH=armv6 kernel-toolchain

......

cc -O2 -pipe  -I. -I/usr/freebsd-xen-arm-v2.2-0782e25/gnu/usr.bin/dtc -I/usr/freebsd-xen-arm-v2.2-07
82e25/gnu/usr.bin/dtc/../../../contrib/dtc -I/usr/freebsd-xen-arm-v2.2-0782e25/gnu/usr.bin/dtc/../..
/../sys/contrib/libfdt -std=gnu99 -Qunused-arguments -I/usr/obj/arm.armv6/usr/freebsd-xen-arm-v2.2-
0782e25/tmp/legacy/usr/include -static -L/usr/obj/arm.armv6/usr/freebsd-xen-arm-v2.2-0782e25/tmp/le
gacy/usr/lib -o dtc dtc-parser.tab.o dtc-lexer.lex.o dtc.o checks.o fstree.o livetree.o treesource.o
 data.o flattree.o srcpos.o util.o fdt.o fdt_ro.o fdt_rw.o fdt_strerror.o fdt_sw.o fdt_wip.o -legacy
ld: error: duplicate symbol: yylloc
>>> defined at dtc-parser.tab.c
>>> dtc-parser.tab.o:(yylloc)
>>> defined at dtc-lexer.lex.c
>>> dtc-lexer.lex.o:(.bss+0x58)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

He said that it will work on FreeBSD 11. Probably should I install FreeBSD 11 for X64 bit ? Can I install a different compiler on my FreeBSD 13.2,instead of making a fresh installation of FreeBSD 11 ?

For example now I'm trying to install sshfs,but it is neither in the ports nor in the packages.
 
I've completed the task to prepare the FreeBSD domU / guest OS to boot with xen,installed on my Arm 32 bit Chromebook,following this tutorial :


These are the commands issued : (I've used FreeBSD 11)

Code: 
$ truncate -s 512 xenvm.img
$ sudo mdconfig -f xenvm.img -u 0
$ sudo newfs /dev/md0
$ sudo mount /dev/md0 /mnt

$ sudo make TARGET_ARCH=armv6 kernel-toolchain
$ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
$ sudo make TARGET_ARCH=armv6 DESTDIR=/build KERNCONF=XENVIRT makekernel
$ sudo make TARGET_ARCH=armv6 buildworld
$ sudo make TARGET_ARCH=armv6 DESTDIR=/build installworld distribution

$ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab

$ nano /build/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on
secure")

$ sudo umount /build
$ sudo mdconfig -d -u 0

Then I copied the rootfs and the kernel to DOM 0 on the board.

To boot FreeBSD I need the following configuration file :

Code: 
$ cat freebsd.cfg

kernel="kernel"
memory=64
name="freebsd"
vcpus=1
autoballon="off"
disk=[ 'phy:/dev/loop0,xvda,w' ]

I have also issued these commands :

Code: 
$ losetup /dev/loop0 xenvm.img
$ xl create freebsd.cfg
$ xl console freebsd
$ xl create freebsd.cfg

But I've got an error :

Code: 
Parsing config from freebsd.cfg

xc: error: panic: xg_dom_elfloader.c:63: xc_dom_guest_type: image not capable of booting inside a HVM container: Invalid kernel
libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image failed
libxl: error: libxl_create.c:1640:domcreate_rebuild_done: Domain 4:cannot (re-)build domain: -3
libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 4:Non-existant domain
libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 4:Unable to destroy guest
libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 4:Destruction of domain failed

According with this :

github.com

xen/docs/misc/pvh.pandoc at master · xen-project/xen

Mirror of the Xen Repository (PRs not accepted see: http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches) - xen-project/xen
github.com github.com

this line :

Code: 
ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,          .long,  xen_start32)

needs to be in the ELF header of the FreeBSD ELF kernel image itself.

Someone can explains to me how to do it ? Is this intended to work ? Or something else task is needed ?
 
Oh grazie amico,cazzo e' vero. Avevo dato per scontato fosse per Arm. A questo punto tocca chiedere direttamente a Julien Grall.
 
Please,take a look at this output of the linux kernel that can boot on Xen, and the FreeBSD kernel that cannot :

Code: 
% file zImage-6.1.59-stb-xen-cbe+
zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little-endian)

% file FREEBSD-XENVIRT          
FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not stripped

The FreeBSD kernel that won't boot is in ELF format but the Linux kernel that does boot is in zImage format.

I spent time reading the docs on xenbits.xenproject.org, and according to those docs Xen on arm only knows how to boot a kernel in the zImage format, so the FreeBSD kernel is in a format that modern Xen incorrectly detects as an x86 kernel.

I also watched Julien Grall's 30 minute video presentation of his work to boot FreeBSD/arm on Xen at FOSDEM 2014 here :

https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/

In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting occasional crashes and needed to investigate the problem. He mentioned the zImage ABI that Linux uses, but pointed out FreeBSD does not use that format, and back then it was an open question which format to use to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only supported format is still the zImage format that Linux uses.

It looks like Julien's work back then was using an ELF binary to boot FreeBSD/arm on Xen instead of the supported zImage format that Linux uses and the modern Xen toolstack exits with an error when trying to boot the FreeBSD ELF formatted binary that Julien's patch creates. So the best solution would be to try to port the rules to build a FreeBSD kernel in the zImage format instead of the ELF format. I have been studying the Makefiles in Linux to see how Linux builds the Linux arm kernel in the zImage format, but it is not trivial to understand.
 
You must log in or register to reply here.
Back
Top