qemu I'm trying to virtualize FreeBSD 13.2 for armv7 on my ARM Chromebook (armhf) with qemu-kvm

ZioMario · Aug 14, 2023

Hello to everyone.

I've enabled KVM on a more recent kernel version,to be precise on the 5.4.244,on the old ARM Chromebook model xe303c12,following mainly this tutorial :

http://www.virtualopensystems.com/en/solutions/guides/kvm-on-chromebook/

I have achieved the goal following the procedure explained by the v.o.s website :

Code:

$ sudo ./scripts/sdcard.sh /dev/sdX

This erased all data and created 4 partitions in the sd-card, along with copying the u-boot binary to the first partition:

ChromeOS signed binary (vos chained u-boot)
Partition 2 not used
EXT2 partition for u-boot files (here I have copied your zImage (converted in uImage and your exynos5250-snow.dtb files). I've got the zip file that you created for me one month ago,here :

https://github.com/quarkscript/linux-armv7-xe303c12-only/files/11615040/5.4.244-kvm.zip

I have decompressed it and I've got the zImage file that was stored inside the deb file.

Since I want to use u-boot,I have converted zImage into uImage using the following command :

Code:

mkimage -A arm -O linux -T kernel -C none -a 0x80008000 -e 0x80008000 -n "Linux kernel" -d zImage uImage

4. EXT4 partition for userspace files. Here I have copied the Devuan 4 userland files that I've got here :

https://drive.google.com/u/0/uc?id=1KSgE-3G2GEBzcYhk_JleVYWl6m061GdI&export=download

but I haven't used btrfs fs,but ext4. And I have extended the space on the partition from 1882M to 200 GB.

Code:

sudo dd if=nv_uboot-snow.kpart of=/dev/sdX1 (nv_uboot-snow.kpart taken from the vos website)

I wrote a little bash script that executes the command line :

Code:

dmesg > /root/Desktop/log

as soon as the desktop environment started. This is the content of the log file :

https://pastebin.ubuntu.com/p/GQ59sqt72v/

I read "kvm: Hyp mode initialized successfully". So I'm sure that kvm is working on the kernel version that I've used (5.4.244).

Now let's talk about how to virtualize FreeBSD as a GUEST os. I've created this script :

Code:

UEFICODE=/home/marietto/Dati/Si/AAVMF32_CODE.fd
UEFIVARS=/home/marietto/Dati/Si/AAVMF32_VARS.fd
CD=/home/marietto/Dati/Si/FreeBSD-13.2-RELEASE-arm-armv7-GENERICSD.img
DISK=/home/marietto/Dati/Si/FreeBSD.img

qemu-system-arm \
-enable-kvm -serial stdio \
-m 512 -M virt -cpu cortex-a15 \
-drive if=pflash,format=raw,unit=0,file=$UEFICODE \
-drive if=pflash,format=raw,unit=1,file=$UEFIVARS \
-drive file=$CD,media=disk,format=raw \
-drive file=$DISK,format=raw \
-device virtio-net,netdev=net0,mac="52:54:00:12:34:55" \
-netdev type=user,id=net0 \
-device virtio-gpu-pci \
-usb -device nec-usb-xhci \
-device usb-kbd -device usb-mouse \
-device vmware-svga,id=video0,vgamem_mb=16

what happened when I run it ? You can give a look at the log that I have attached below :

https://pastebin.ubuntu.com/p/RfKvCqyTrD/

The error that prevents FreeBSD from reaching the installation part seems to be :

eval: cannot open /dev/ttyv0: No such file or directory
eval: cannot open /dev/ttyv0: No such file or directory
eval: cannot open /dev/ttyv*: No such file or directory
Fatal kernel mode data abort: 'Alignment Fault' on read

I have no idea about how to fix it.

covacat · Aug 14, 2023

Solved - Kernel panic on armv7 with qemu

I'm trying to run a generic armv7 image (13.2) on an amd64 host (13.2) and I get a panic with the network interface: To start the VM, I use the following command: qemu-system-arm -M virt -m 2048m -nographic -nic tap -bios /usr/local/share/u-boot/u-boot-qemu-arm/u-boot.bin -hda...

forums.freebsd.org

there is a PR and a patch by _martin iirc

ZioMario · Aug 14, 2023

If the problem still persists,I suppose that the _martin patch hasn't been accepted ? Anyway I see that the file that I should patch is the following : /usr/src/sys/dev/virtio/network/if_vtnet.c. But where is located ? Is it inside the "iso" image of FreeBSD 13.2 for armv7 ? How can I patch it if I can't boot and run it ?

covacat · Aug 14, 2023

you need to rebuild the kernel with the patch applied
you can build an arm7 kernel on a x86 freebsd box

ZioMario · Aug 14, 2023

Never did that. It's time to learn a new thing. I need some good documentation (please not the official one,that I'm not able to elaborate correctly),so,if you can,share some external tutorial, focused and that goes straight to the point.

covacat · Aug 14, 2023

if you have a 13.2 box amd64 with the source tree installed
cd /usr/src
make -DNO_MODULES KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 buildkernel
after the build finishes the kernel binary will be at /usr/obj/usr/src/arm.armv7/sys/GENERIC/kernel
you need to copy that to the virtual machine
i
f qemu can emulate rtl8139 you can use that until you fix virtionet, no patching necessary but most likely lower performance

ZioMario · Aug 14, 2023

---> you need to copy that to the virtual machine

where ?

covacat · Aug 14, 2023

if its a stdandard freebsd usb-stick image than mdconfig -t vnode FreeBSD.img
the mount the freebsd partition
mount /dev/md0s2a /mnt
cp the_kernel_file_above /mnt/boot/kernel
umount /mnt
mdconfig -d -u 0

that will be if you don't have other mds connected otherwise it might be md1 md2 etc
and you need change the mount and mdconfig -d accordingly

ZioMario · Aug 14, 2023

For the moment I've used "rtl8139" and I've reached the login prompt. So,this is the proof that It's affected by the same bug. Do you know what's the login and password ? but is the image that I'm using valid to install FreeBSD to an image file ? It does not seems an installation disk. Usually the installation disk jumps to the installation menu.

covacat · Aug 14, 2023

root root i think
or root without password or root / freebsd
anyway you can boot in single user mode and change it if you cant figure what the default one is
some arm images are like a live fs on the install disk

_martin · Aug 14, 2023

My patch is a dirty workaround for OP that was asking about a fix in the link covacat mentioned above. There's nothing wrong with the FreeBSD C-code however compiler is generating incorrect code for given platform.
While I did submit the patch and shared it in PR nobody replied (something I did encounter few times before). Maybe I should have submitted it via reviews but my brain is not able to overcome steep learning curve.

While it's a dirty workaround other drivers do use the same approach to "fix" this (aka "fudge" where they move pointer within data to make sure pointer is properly aligned).

You need to build the kernel from sources. I personally crosscompiled it - apply patch, build kernel, sync to a (virtual) disk and boot it.

ZioMario · Aug 14, 2023

_martin said:
My patch is a dirty workaround for OP that was asking about a fix in the link covacat mentioned above. There's nothing wrong with the FreeBSD C-code however compiler is generating incorrect code for given platform.
While I did submit the patch and shared it in PR nobody replied (something I did encounter few times before). Maybe I should have submitted it via reviews but my brain is not able to overcome steep learning curve.

While it's a dirty workaround other drivers do use the same approach to "fix" this (aka "fudge" where they move pointer within data to make sure pointer is properly aligned).

You need to build the kernel from sources. I personally crosscompiled it - apply patch, build kernel, sync to a (virtual) disk and boot it.

For the moment I'm trying to find an alternative network interface to use. First of all I try the easiest solutions. If they fails,I will try the most complicated route. Below you can see all the network interfaces supported by qemu. I'm trying one by one :

Network devices:

Code:

name "e1000", bus PCI, alias "e1000-82540em", desc "Intel Gigabit Ethernet"
name "e1000-82544gc", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82545em", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000e", bus PCI, desc "Intel 82574L GbE Controller"
name "i82550", bus PCI, desc "Intel i82550 Ethernet"
name "i82551", bus PCI, desc "Intel i82551 Ethernet"
name "i82557a", bus PCI, desc "Intel i82557A Ethernet"
name "i82557b", bus PCI, desc "Intel i82557B Ethernet"
name "i82557c", bus PCI, desc "Intel i82557C Ethernet"
name "i82558a", bus PCI, desc "Intel i82558A Ethernet"
name "i82558b", bus PCI, desc "Intel i82558B Ethernet"
name "i82559a", bus PCI, desc "Intel i82559A Ethernet"
name "i82559b", bus PCI, desc "Intel i82559B Ethernet"
name "i82559c", bus PCI, desc "Intel i82559C Ethernet"
name "i82559er", bus PCI, desc "Intel i82559ER Ethernet"
name "i82562", bus PCI, desc "Intel i82562 Ethernet"
name "i82801", bus PCI, desc "Intel i82801 Ethernet"
name "ne2k_pci", bus PCI
name "pcnet", bus PCI
name "rocker", bus PCI, desc "Rocker Switch"
name "rtl8139", bus PCI
name "tulip", bus PCI
name "usb-net", bus usb-bus
name "virtio-net-device", bus virtio-bus
name "virtio-net-pci", bus PCI, alias "virtio-net"
name "virtio-net-pci-non-transitional", bus PCI
name "virtio-net-pci-transitional", bus PCI
name "vmxnet3", bus PCI, desc "VMWare Paravirtualized Ethernet v3"

yuripv79 · Aug 14, 2023

Just use virtio-net.

ZioMario · Aug 14, 2023

why ?

yuripv79 · Aug 14, 2023

Actually don't

it's "vtnet0" that you are having the problem with.

Otherwise I'd try vmxnet3, another paravirtualized driver, not some ancient h/w emulation.

ZioMario · Aug 14, 2023

I've found the network interface that could work : i82559er,an IP is assigned within the FreeBSD VM. This is how I have configured it :

Code:

qemu :

EFICODE=/home/marietto/Dati/Si/AAVMF32_CODE.fd
UEFIVARS=/home/marietto/Dati/Si/AAVMF32_VARS.fd
CD=/home/marietto/Dati/Si/FreeBSD-13.2-RELEASE-arm-armv7-GENERICSD.img
DISK=/home/marietto/Dati/Si/FreeBSD.img

qemu-system-arm \
-enable-kvm -serial stdio \
-m 512 -M virt -cpu cortex-a15 \
-drive if=pflash,format=raw,unit=0,file=$UEFICODE \
-drive if=pflash,format=raw,unit=1,file=$UEFIVARS \
-drive file=$CD,media=disk,format=raw \
-drive file=$DISK,format=raw \
-device i82559er,netdev=net0,mac="52:54:00:12:34:55" \
-netdev type=user,id=net0 \
-device virtio-gpu-pci \
-usb -device nec-usb-xhci \
-device usb-kbd -device usb-mouse \
-device vmware-svga,id=video0,vgamem_mb=16

/etc/rc.conf

hostname="marietto"
ifconfig_fxp0="DHCP"
local_unbound_enable="YES"
gateway_enable="YES"

/etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search homenet.telecomitalia.it
nameserver 8.8.8.8

root@mario:~ # ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 52:54:00:12:34:55
        inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
        media: Ethernet autoselect (10baseT/UTP <half-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I can ping 10.0.2.15,but not google.com ;

root@mario:~ # ping google.com

PING google.com (142.250.180.142): 56 data bytes
frozen.

Do you see some error in the network configuration ? I want to be sure that i82559er is good.

covacat · Aug 14, 2023

you need to create a tap device on the linux side and bridge it with your ethernet iface
something like here

QEMU/KVM Bridged Network with TAP interfaces

A blog about Samba, SMB, iSCSI and Linux storage software development.

blog.elastocloud.org

ZioMario · Aug 14, 2023

covacat said:
you need to create a tap device on the linux side and bridge it with your ethernet iface
something like here

QEMU/KVM Bridged Network with TAP interfaces

A blog about Samba, SMB, iSCSI and Linux storage software development.

blog.elastocloud.org

the bridging is another method to connect. I didn't used it yet. I know that qemu supports also the "user networking" method,the method that I used,but that I don't know if it is well configured.

ZioMario · Sep 15, 2023

I'm trying to fix the virtio bug using this patch :

Attachment #242118 for bug #271288

bugs.freebsd.org

I don't know why,but I'm making some mistake within the source code and when I try to compile the file,it gives an error. Can someone help me to understand where is the mistake ? Thanks. Please give a look at the screenshots attached below :

the error that it gives is :

covacat · Sep 15, 2023

the patch seems badly applied, some lines of code are missing

Code:

     } else
                        m_head = m_tail = m;
        }

        if (m_tailp != NULL)
                *m_tailp = m_tail;

        return (m_head);

looks like these lines are missing after
m_tail = m;

ZioMario · Sep 15, 2023

It does not compile. Check screens attached,please.

covacat · Sep 15, 2023

this is the the source with the patch applied
it builds with a few warnings

ZioMario · Sep 15, 2023

It can't be the patch since your file does not contain any of the keyword that are written inside the original one. This should be the final version :

Attachment #242127 for bug #271288

bugs.freebsd.org

I've got it on the lastest message posted here : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271288

Where have you got the code of your patch ? I'm not even able to see where is the patched code inside your file.

ZioMario · Sep 15, 2023

This should be your code :

Code:

        m->m_len = size;
        if (m_head != NULL) {
            m_tail->m_next = m;
            m_tail = m;
        } else
            m_head = m_tail = m;
    }

    if (m_tailp != NULL)
        *m_tailp = m_tail;

    return (m_head);
}

this should be the original patch :

It seems completely different. Where is gone the "VTNET_ALIGNMENT" part ?

covacat · Sep 15, 2023

this is the good one

qemu I'm trying to virtualize FreeBSD 13.2 for armv7 on my ARM Chromebook (armhf) with qemu-kvm

ZioMario

covacat

Solved - Kernel panic on armv7 with qemu

ZioMario

covacat

ZioMario

covacat

ZioMario

covacat

ZioMario

covacat

_martin

ZioMario

yuripv79

ZioMario

yuripv79

ZioMario

covacat

QEMU/KVM Bridged Network with TAP interfaces

ZioMario

QEMU/KVM Bridged Network with TAP interfaces

ZioMario

Attachment #242118 for bug #271288

covacat

ZioMario

Attachments

covacat

Attachments

ZioMario

Attachment #242127 for bug #271288

ZioMario

covacat

Attachments