bhyve pcie passthrough: base or size not page aligned, failed to initialize BARs

P

pgzh1

Hi,

on my new FreeBSD 15.0-RELEASE server based on an AsRock X570D4U with Ryzen 7 5700X I get the following error message when trying to start bhyve with one of the chipsets SATA controllers passed through to the guest:
Code: 
bhyve: passthru device 43/0/0 BAR 5: base 0xfc500000 or size 0x800 not page aligned
 
bhyve: failed to initialize BARs for PCI 43/0/0
Device emulation initialization error: Invalid argument

I invoke bhyve with
Code: 
bhyve -AHPSDu \
        -c cpus=4,sockets=1,cores=2,threads=2 \
        -m 6144M \
        -s 0,amd_hostbridge \
        -s 2,nvme,/dev/zvol/zroot/vm/nas \
        -s 4,passthru,43/0/0 \
        -s 6,passthru,45/0/16 \
        -s 8,ahci-cd,/mnt/usb/debian-13.3.0-amd64-netinst.iso \
        -s 28,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \
        -s 29,xhci,tablet \
        -s 30,virtio-rnd \
        -s 31,lpc \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/vm/nas/BHYVE_UEFI_VARS.fd \
        nas

The X570s chipset sports four SATA controllers. Two are connected to the two (currently unpopulated) M.2 slots on the mainboard, the other two are wired to 2x4 SATA ports on the board.
I want to pass through one of these (4 SATA ports) to a vm.
Code: 
# pciconf -l
[...]
ppt2@pci0:43:0:0:       class=0x010601 rev=0x51 hdr=0x00 vendor=0x1022 device=0x7901 subvendor=0x1022 subdevice=0x7901
ahci0@pci0:44:0:0:      class=0x010601 rev=0x51 hdr=0x00 vendor=0x1022 device=0x7901 subvendor=0x1022 subdevice=0x7901
[...]

Code: 
# cat /boot/loader.conf
[...]
vmm_load="YES"
pptdevs="43/0/0 38/0/0 39/0/0"
hw.vmm.amdvi.enable=1

The pci devices 38 and 39 are onboard-nics that I plan to pass through other vms in the future.

What could be the cause for the failed initialization?
I have a feeling I have to tweak something in the BIOS, but am not sure...
What BIOS settings could be of importance here?
What else could cause this problem?

I've passed one SATA controller of a X570 chipset to a VM on an AsRock X570 Pro4 board with QEMU / Linux KVM before, so technically it is possible to do this.

Any advice would be much appreciated!

Peter
 
Since nobody seems to have a solution for this, I'd like to share what I've found out myself:
The PCI BAR size of the SATA controllers is 2M, which is less than the page size (4M).
The FreeBSD kernel does not allow any PCI device with a BAR size not perfectly aligned to the page size to be passed through to a bhyve VM (currently).

There is a patch that may work around that, but I am not sure if this causes security or stability issues and have not tried it.
H

bhyve Thread 'Patch: allow any MM BARs passthru + without interrupts on bhyve'

When I tried to passthru simple CompactPCI (basically PCI) cards with PLX9030 and PLX9050 chip, having:
  • memory mapped BARs only
  • no interrupts (and thus no MSI(-X) support)
I encountered unexpected fatal problems with BHYVE (FreeBSD 14.3 RELEASE):
  • it does not allow MM BARs with size that is not multiple of page size (4096 bytes)
  • it does not allow read/write access to MM BARs at all (!) - only I/O BARs are implemented
  • it does not allow card that has NO interrupts (so absence of MSI/MSI-X is not a problem)
As workaround I created experimental patch that:
  1. ...

I wanted to set up a Linux fileserver inside that VM and will try to create a jail'ed fileserver based directly on FreeBSD instead.
This way I can attach the disks in my mirror to two different controllers and maybe get a small performance boost from that as well.
 
You must log in or register to reply here.
Back
Top