Bhyve crashes system during PCI passthrough

[arun]

When I try to pass WiFi PCI device rtl8723be, the OS crashes and reboots. The guest boots normally when passthru is removed.

# bhyve -S -c 2 -m 1G -w -H \
-s 0,hostbridge -s 3,ahci-cd,/linux.iso \
-s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
-s 30,xhci,tablet -s 31,lpc -l com1,stdio \
-s 5,passthru,2/0/0 \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0

/boot/loader.conf:

vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
pptdevs="2/0/0"
# kern.vty=vt
hw.psm.synaptics_support="1"
drm.i915.enable_rc6=7

OS version: FreeBSD 12.0 uefi boot
Guest OS: ArchLinux
Processor: i3 6006

 

[SirDice]

Requirements

    CPU supports Intel IOMMU (a.k.a. VT-d) feature
    PCI device (and driver) supports MSI/MSI-x interrupts 

Host VT-d support can be determined by searching for a DMAR table in the ACPI tables with acpidump -t | grep DMAR

PCI card MSI/MSI-x support can be determined with pciconf -lc | grep MSI

bhyve/pci_passthru - FreeBSD Wiki

 

[arun]

$ sudo acpidump -t | grep DMAR
  DMAR: Length=168, Revision=1, Checksum=37,

$ sudo pciconf -lc | grep MSI
    cap 05[ac] = MSI supports 1 message 
    cap 05[90] = MSI supports 1 message 
    cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message
    cap 05[80] = MSI supports 1 message 
    cap 05[8c] = MSI supports 1 message, 64 bit 
    cap 05[80] = MSI supports 1 message enabled with 1 message
    cap 05[80] = MSI supports 1 message 
    cap 05[80] = MSI supports 1 message 
    cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 05[50] = MSI supports 1 message, 64 bit 
    cap 10[70] = PCI-Express 2 endpoint MSI 1 max data 128(128) RO
    cap 11[b0] = MSI-X supports 4 messages, enabled
    cap 05[50] = MSI supports 1 message, 64 bit

ppt0@pci0:2:0:0:    class=0x028000 card=0x81c1103c chip=0xb72310ec rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[50] = [B]MSI supports 1 message, 64 bit[/B] 
    cap 10[70] = PCI-Express 2 endpoint max data 128(128) RO
                 link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 0023b7feff4ce000
    ecap 0018[150] = LTR 1
    ecap 001e[158] = unknown 1

 

[Phishfry]

I want to try this out as well. Passing through Wifi with bhyve. Has anybody else done this successfully?
My first thought is why not prove it is possible first with the standard Atheros cards.
Any other wifi devices on FreeBSD can be challenging. Intel IWM seems solid too.

 

[arun]

PCI passthrough is supported on my hardware. I have successfully passed that wifi device into Linux guest on Linux host.
-----------------------------------------------------------------------------------------------------------------------------------------------

 

[Phishfry]

But Linux supports the device that is why it works. Linix is loading the drivers firmware blob on boot.
On FreeBSD all wireless modules need a firmware blob to work.
By using an unidentified/unsupported card in FreeBSD there is no blob to load.
Hence when you go to start it in a VM it crashes. The blob needs to start on actual bootup.
This is all speculation on my part.
I am going to try an supported card like Atheros and will report back.

 

[arun]

But Linux supports the device that is why it works. Linix is loading the drivers firmware blob on boot.
On FreeBSD all wireless modules need a firmware blob to work.
By using an unidentified/unsupported card in FreeBSD there is no blob to load.
Hence when you go to start it in a VM it crashes. The blob needs to start on actual bootup.
This is all speculation on my part.
I am going to try an supported card like Atheros and will report back.

Firmware needed for pass-through? I never heard something like that. Unsupported wifi chip passthrough

It's probably a bug similar to this - bhyve bug

 

[arun]

Just confirmed. It's vmm.ko bug. I have applied those patches, now vm booted and it loads driver modules and firmware without crashing.
but it doesn't scan WiFi networks though.