bhyve host crashes on VM start with USB passthru

patpro

patpro

Hello,

I’m running a FreeBSD 14.4-RELEASE server as a bhyve host for a Win11 VM. I want this VM to get access to a USB device. I’ve identified a USB port that seems alone up to the root XHCI device it depends on.
When passing-thru this device, and starting the VM, the host reboots. I’m using vm-bhyve.

I’ve identified the USB I target is seen as xhci0:
Code: 
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfc000000-0xfc0fffff at device 0.3 on pci5
xhci0: 64 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0

and before configuring passthru in loader.conf I read:
Code: 
# vm passthru|grep xhci0
xhci0      6/0/3        No           -

after configuring passthru in loader.conf I read:
Code: 
# vm passthru|grep -E "xhci0|ppt"
ppt0       6/0/3        Yes          -
xhci0      6/0/4        No           -

usbus0 is still on xhci0, but xhci0 as jumped from pci0:6:0:3 to pci0:6:0:4 and pci0:6:0:3 is for ppt0 alone.
Not sure if it’s an expected result.

Any help greatly appreciated.

CPU is:
Code: 
CPU: AMD Ryzen 7 7840U w/ Radeon  780M Graphics      (3294.03-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0xa70f41  Family=0x19  Model=0x74  Stepping=1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ef8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x75c237ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX,ADMSKX>
  Structured Extended Features=0xf1bf97a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,AVX512CD,SHA,AVX512BW,AVX512VL>
  Structured Extended Features2=0x405fde<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,RDPID>
  Structured Extended Features3=0x10000010<FSRM,L1DFL>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x791ef257<CLZERO,IRPerf,XSaveErPtr,RDPRU,BE,WBNOINVD,IBPB,INT_WBINVD,IBRS,STIBP,STIBP_ALWAYSON,PREFER_IBRS,SAMEMODE_IBRS,NOLMSLE,SSBD,CPPC,PSFD,BTC_NO,IBPB_RET>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics

The output of sysctl hw.vmm.amdvi is:
Code: 
hw.vmm.amdvi.domain_id: 0
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 1
hw.vmm.amdvi.count: 1

/boot/loader.conf is:
Code: 
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.racct.enable="1"

vmm_load="YES"
cryptodev_load="YES"
zfs_load="YES"

amdtemp_load="YES"
autoboot_delay="2"

accf_data_load="YES"
accf_http_load="YES"

mac_portacl_load="YES"
cpuctl_load="YES"

#passthrough
hw.vmm.amdvi.enable=1
pptdevs="6/0/3"

# https://m4c.pl/blog/freebsd-audio-setup-bitperfect-equalizer-realtime/#freebsd_equalizer
hint.pcm.4.eq=1

VM config file is:
Code: 
loader="uefi"
graphics="yes"
graphics_port="5799"
xhci_mouse="yes"
cpu=2
memory=2G

passthru0="6/0/3"

# put up to 8 disks on a single ahci controller.
# without this, adding a disk pushes the following network devices onto higher slot numbers,
# which causes windows to see them as a new interface
ahci_device_limit="8"

# ideally this should be changed to virtio-net and drivers installed in the guest
# e1000 works out-of-the-box
network0_type="e1000"
network0_switch="noinet"

disk0_type="ahci-hd"
disk0_name="disk0.img"

# windows expects the host to expose localtime by default, not UTC
utctime="no"
uuid="8898e0f0-9a3d-11f1-9289-d063b40622a6"
network0_mac="58:9c:fc:01:6e:51"
 
Greetings.

Sorry, I can't offer a solution or where to start to debug the problem, just wanted to share my experience passing through USB controller devices to a vm-bhyve machine on a AMD laptop I tested a few days ago and today again.

No joy. My system doesn't reboot, the passed through USB controller devices show up in the guest (FreeBSD-15.1-RELEASE), but no plugged in devices (USB wifi adapter, USB disks) are recognized. I successfully passed through a built-in PCI wifi adapter to a bhyve guest, though. The issued in this specific case (AMD systems) could be affecting USB related devices only.

The test setup is on a 15.1-RELEASE-p2 host. The /boot/loader.conf passthrough configuration is basically the same as on your system, passing all available xhci controller.

I noticed in the guest on boot several hundred messages spamming the console, comming from the xhci devices:
Rich (BB code): 
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x800000000-0x8000fffff at device 6.0 on pci0
xhci0: 64 bytes context size, 64-bit DMA
xhci0: xECP capabilities <VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEN
D(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND( [continue 100+]

xhci1: <XHCI (generic) USB 3.0 controller> mem 0x800000000-0x8000fffff at device 6.0 on pci0
xhci1: 64 bytes context size, 64-bit DMA
xhci1: xECP capabilities <VEND(ff),VEND(ff),VEND(ff),,VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),
VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(ff),VEND(  [continue 100+]

I didn't try debugging the issue on bhyve(8), used VirtualBox instead. No problem there.

If it's important to have it working on your 14.4 system, open a PR on https://bugs.freebsd.org .

If the file system is ZFS, you could try 16-CURRENT in a boot environment.
 
Thank you for your feed back. I’ve spent 6 hours trying to do the same USB passthru to a VM on an Intel NUC: it barely works. To be honest, sometime — I don’t know why — I got it working with a thumbdrive. The behavior of bhyve and FreeBSD (15.1 in that particular case) is so random it pissed me off really hard.
One example amongst many: passthru active, I plug the thumbdrive and boot the VM -> it shows up on VM, then I remove the thumbdrive, reboot the VM, plug the thumbdrive -> it shows up on host.
I’ve had far more fails than successes.

At least on the NUC, the PCI device behavior was better than on AMD:

without pptdevs in loader:
xhci1 57/0/0 No JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 2C 2018]

with
pptdevs in loader:
ppt0 57/0/0 Yes JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 2C 2018]

Anyway, the NUC had few crashes too.

I’m giving up on bhyve.
 
You must log in or register to reply here.
Back
Top