proxmox FreeBSD AARCH64 in Proxmox

gatekeeper

gatekeeper

Hi all,

I have a Proxmox server 8.3-1 (21. November 2024), running on Intel Xeon X5667, and I am trying to install FreeBSD AARCH64 through emulation.
There is a nice guide on how to do this for Debian here.
I follow the guide but, instead of installing Debian, I install FreeBSD 14.2
All goes well, until the system starts - it gets stuck at "Generating RSA host key".
If I CTRL+C, after lots of investigation, I found out more bizarre problems - RSA signatures cannot be validated.
For example, if I issue the command pkg update, it fails completely:
1738946506713.png


For reference, here is how qemu is started by Proxmox:
/usr/bin/qemu-system-aarch64 -id 102 -name test,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/102.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/102.pid -daemonize -smbios type=1,uuid=e1b90329-623c-4bb6-b467-99cf4522cf8d -drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.fd -drive if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/zvol/rpool/data/vm-102-disk-2,size=67108864 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -nographic -cpu cortex-a57 -m 4192 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f -device usb-ehci,id=ehci,bus=pcie.0,addr=0x1 -chardev socket,id=serial0,path=/var/run/qemu-server/102.serial0,server=on,wait=off -serial chardev:serial0 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3,free-page-reporting=on -iscsi initiator-name=iqn.1993-08.org.debian:01:b894dabdb95 -device virtio-scsi-pci,id=scsihw0,bus=pcie.0,addr=0x5 -drive file=/dev/zvol/rpool/data/vm-102-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=101 -drive if=none,id=drive-scsi1,media=cdrom,aio=io_uring -device scsi-cd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,bootindex=100 -netdev type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device virtio-net-pci,mac=BC:24:11:7E:82:8B,netdev=net0,bus=pcie.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102 -machine accel=tcg,type=virt+pve0
Click to expand...

Hmmm... it could be a bug in qemu by the Proxmox project but then, why does Debian Bookworm work without problems....
Additionally, if I try to run the very same FreeBSD distribution in qemu under MacOS with UTM, all runs fine - so, it is not a problem of the installation media.

Any ideas what I might be missing?


Thanks!
 
I've have this minimal setup:
Code: 
qemu-system-aarch64 -M virt -cpu cortex-a55 -smp 4 -m 8192 -nographic \
-drive if=pflash,format=raw,read-only=on,file=./AAVMF_CODE.fd \
-drive if=pflash,format=raw,file=./AAVMF_VARS.fd \
-device virtio-blk-pci,drive=hd0 -drive if=none,id=hd0,format=raw,file=FreeBSD-14.2-RELEASE-arm64-aarch64-BASIC-CI.raw
Where image is downloaded from FreeBSD web.

In my home LAB one of two nodes are Odroid h2 (J4105) and h4+ultra (i3-N305). Both are on the same patch level, use the same guest image. On the older one I can replicate your issue.

This is not about SSH keys (those are just server side ssh keys). They are victim of the issue though. Some part of crypto support is failing for some reason.

Example:
Code: 
root@freebsd:~ # fetch https://google.com
Certificate verification failed for /C=US/O=Google Trust Services/CN=WR2
00208185759B0000:error:0200008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:/usr/src/crypto/openssl/crypto/rsa/rsa_pk1.c:75:
00208185759B0000:error:02000072:rsa routines:rsa_ossl_public_decrypt:padding check failed:/usr/src/crypto/openssl/crypto/rsa/rsa_ossl.c:598:
00208185759B0000:error:1C880004:Provider routines:rsa_verify:RSA lib:/usr/src/crypto/openssl/providers/implementations/signature/rsa_sig.c:774:
00208185759B0000:error:06880006:asn1 encoding routines:ASN1_item_verify_ctx:EVP lib:/usr/src/crypto/openssl/crypto/asn1/a_verify.c:219:
00208185759B0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
fetch: https://google.com: Authentication error
root@freebsd:~ #

root@freebsd:~ # ssh 192.0.2.1
Fssh_ssh_dispatch_run_fatal: Connection to 192.0.2.1 port 22: incorrect signature

RSA test failed too: openssl speed rsa.

I tried several different cpus, different random sources (-device virtio-rng-device), but I was not able to find what is a problem.
 
I've a debian 12.9 with the qemu. I copied the /usr/bin/qemu-system-aarch64 from debian to proxmox (pve-manager/8.3.2/3e76eec21c4a14a7 (running kernel: 6.8.12-6-pve). Only libcapstone4 was missing on proxmox.

My "run" script looks like (rng device is optional, used it later in tests though):
Code: 
LIB="/usr/share/kvm"

/root/qemu-system-aarch64 -L ${LIB} -M virt -cpu cortex-a57 -smp 4 -m 8192 -nographic \
-drive if=pflash,format=raw,read-only=on,file=./AAVMF_CODE.fd \
-drive if=pflash,format=raw,file=./AAVMF_VARS.fd \
-device virtio-blk-pci,drive=hd0 -drive if=none,id=hd0,format=raw,file=FreeBSD-14.2-RELEASE-arm64-aarch64-BASIC-CI.raw \
-device virtio-rng-device

I had to add LIB as location of the fw is different in proxmox to debian. I changed the cpu to a57 as a55 was not in the list in debian version. I tested it on fresh image again and the results:

Using proxmox qemu-system-aarch64:
Code: 
root@freebsd:~ # openssl speed rsa
Doing 512 bits private rsa's for 10s: 1993 512 bits private RSA's in 10.03s
RSA verify setup failure.  No RSA verify will be done.
0020812C9F830000:error:0200008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:/usr/src/crypto/openssl/crypto/rsa/rsa_pk1.c:75:
0020812C9F830000:error:02000072:rsa routines:rsa_ossl_public_decrypt:padding check failed:/usr/src/crypto/openssl/crypto/rsa/rsa_ossl.c:598:
0020812C9F830000:error:1C880004:Provider routines:rsa_verify:RSA lib:/usr/src/crypto/openssl/providers/implementations/signature/rsa_sig.c:833:
Doing 1024 bits private rsa's for 10s: ^C

root@freebsd:~ # fetch https://google.com
Certificate verification failed for /C=US/O=Google Trust Services/CN=WR2
0020E111608E0000:error:0200008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:/usr/src/crypto/openssl/crypto/rsa/rsa_pk1.c:75:
0020E111608E0000:error:02000072:rsa routines:rsa_ossl_public_decrypt:padding check failed:/usr/src/crypto/openssl/crypto/rsa/rsa_ossl.c:598:
0020E111608E0000:error:1C880004:Provider routines:rsa_verify:RSA lib:/usr/src/crypto/openssl/providers/implementations/signature/rsa_sig.c:774:
0020E111608E0000:error:06880006:asn1 encoding routines:ASN1_item_verify_ctx:EVP lib:/usr/src/crypto/openssl/crypto/asn1/a_verify.c:219:
0020E111608E0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1890:
fetch: https://google.com: Authentication error
root@freebsd:~ #

Using debian qemu-system-aarch64:
Code: 
root@freebsd:~ # openssl speed rsa
Doing 512 bits private rsa's for 10s: 13742 512 bits private RSA's in 9.99s
Doing 512 bits public rsa's for 10s: 149995 512 bits public RSA's in 10.00s
Doing 1024 bits private rsa's for 10s: ^C

root@freebsd:~ # fetch https://google.com
fetch: https://google.com: size of remote file is not known
google.com                                              20 kB 1206 kBps    00s
root@freebsd:~ #

Works just fine. Something on proxmox side of qemu is not ok.

For comparison I set the exact VM but with debian image, otherwise the same image (proxmox qemu):
Code: 
root@localhost:~# openssl speed rsa
Doing 512 bits private rsa's for 10s: 5001 512 bits private RSA's in 10.04s
Doing 512 bits public rsa's for 10s: 53339 512 bits public RSA's in 10.02s
Doing 1024 bits private rsa's for 10s: ^C
root@localhost:~#

root@localhost:~# wget https://google.com
--2025-02-14 15:31:53--  https://google.com/
Resolving google.com (google.com)... 142.251.208.174, 2a00:1450:400d:803::200e
Connecting to google.com (google.com)|142.251.208.174|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/ [following]
--2025-02-14 15:31:55--  https://www.google.com/
Resolving www.google.com (www.google.com)... 142.251.36.100, 2a00:1450:400d:80a::2004
Connecting to www.google.com (www.google.com)|142.251.36.100|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’

index.html.1            [ <=>                ]  20.03K  --.-KB/s    in 0.03s

2025-02-14 15:31:55 (689 KB/s) - ‘index.html.1’ saved [20510]

root@localhost:~#
Was working (though rsa test was worse than on FreeBSD).

It semes that something on FreeBSD (libcrypto?) with a particular version of CPU and qemu doesn't like each other.
 
Before trying to hunt down the root cause of the bug there's easier way to test things first: use the test repo of proxmox to see if this was already addressed or fixed as a result of a different fix.

I've udpated promox with the test repo, updated system:
Code: 
# pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-8-pve)
..
..
pve-qemu-kvm: 9.2.0-1
..

And started the fresh FreeBSD image again:
Code: 
root@freebsd:~ # openssl speed rsa
Doing 512 bits private rsa's for 10s: 5617 512 bits private RSA's in 10.05s
Doing 512 bits public rsa's for 10s: ^C
root@freebsd:~ #

root@freebsd:~ # pkg update
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:aarch64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.21.3...
Extracting pkg-1.21.3: 100%
..

This bug is fixed in higher versions of qemu.

But what is stunning is how slow qemu9 is. That surprised me a lot. Same FreeBSD image, rsa test 5617 512 bits private RSA's in 10.05s vs 13888 512 bits private RSA's in 10.11s.
And even the boot of VM itself is way faster on qemu7.
 
Awesome!

I can confirm that your solution also works for me.
So... you are absolutely right - it seems like a problem with qemu in Proxmox rather than with FreeBSD.
What I find strange is that I had version 9.2.0-5 (which should be newer), but version 9.2.0-1 is the one that works... That version gets installed with the test repo.

Thanks a lot for your help and replies :-)
 
gatekeeper said:
What I find strange is that I had version 9.2.0-5 (which should be newer), but version 9.2.0-1 is the one that works...
Click to expand...
Yeah, that versioning is a bit confusing.
When I'll have time I want to do additional tests and comparisons. Depending on results I was considering opening a bug report to proxmox.

edit: To the version thing: I checked mine, on prod one I have 9.0.2-5, test one has 9.2.0-1. So all OK in that regard.
 
You must log in or register to reply here.
Back
Top