Hey folks,
I'm running into an issue for a few days now that is absolutely driving me nuts.
A few weeks ago I setup a Windows 11 Pro guest using sysutils/vm-bhyve. Installation and configuration of the VM worked without any problems.
Then I started to passthru a PCIe USB controller card. This also worked without any problems.
However, after a few days, the VM refused to boot. It gets stuck eternally somewhere in the Windows bootloader (when the Windows loader circle appears just after TianoCore finished). At this point there's nothing I can do other than to
After some dicking around I figured out that the VM boots fine 100% of the time if I simply don't pass thru the PCIe USB controller. And once I add the passthru again, it locks up exactly the same.
I'm completely out of ideas. This used to work well for a few days after I initially created the VM and fails consistently ever since then.
Scenario:
/boot/loader.conf
VNC screenshot of when/where the guest gets stuck (notice the segment of the Windows bootloader progress thingy - it stops moving):
Any ideas?
I have since updated to a more recent stable/14 commit twice - no changes in behavior.
Changes to VM configuration such as CPU count, CPU topology or memory have no effect.
Also enabling/disabling the network config in the VM config doesn't change anything.
I'm running into an issue for a few days now that is absolutely driving me nuts.
A few weeks ago I setup a Windows 11 Pro guest using sysutils/vm-bhyve. Installation and configuration of the VM worked without any problems.
Then I started to passthru a PCIe USB controller card. This also worked without any problems.
However, after a few days, the VM refused to boot. It gets stuck eternally somewhere in the Windows bootloader (when the Windows loader circle appears just after TianoCore finished). At this point there's nothing I can do other than to
kill the bhyve process. I even need to kill -9.After some dicking around I figured out that the VM boots fine 100% of the time if I simply don't pass thru the PCIe USB controller. And once I add the passthru again, it locks up exactly the same.
I'm completely out of ideas. This used to work well for a few days after I initially created the VM and fails consistently ever since then.
Scenario:
- FreeBSD 14-STABLE host
- Windows 11 Pro guest
vm-bhyve config:
Code:
loader="uefi"
graphics="yes"
xhci_mouse="yes"
# cpu
cpu=8
cpu_sockets=1
cpu_cores=8
cpu_threads=1
# memory
memory=16G
# AHCI
ahci_device_limit="8"
# networking
network0_type="virtio-net"
network0_switch="public"
# disk
disk0_type="nvme"
disk0_name="disk0.img"
# Windows expects the host to expose localtime by default, not UTC
utctime="no"
# Passtrhu
#passthru0="6/0/0" <---- Uncommenting this makes the VM getting stuck at Windows boot :(
uuid="151ec8ab-cdb6-11ee-8d6b-000acd2d4844"
network0_mac="xx:xx:xx:xx:xx:xx" # Redacted for public post vm passthru
Code:
DEVICE BHYVE ID READY DESCRIPTION
hostb0 0/0/0 No 8th Gen Core Processor Host Bridge/DRAM Registers
pcib1 0/1/0 No 6th-10th Gen Core Processor PCIe Controller (x16)
pcib2 0/1/1 No Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8)
xhci0 0/20/0 No 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
none0 0/22/0 No 200 Series PCH CSME HECI
ahci0 0/23/0 No 200 Series PCH SATA controller [AHCI mode]
pcib3 0/27/0 No 200 Series PCH PCI Express Root Port
pcib4 0/27/4 No 200 Series PCH PCI Express Root Port
pcib10 0/28/0 No 200 Series PCH PCI Express Root Port
pcib11 0/28/1 No 200 Series PCH PCI Express Root Port
pcib15 0/28/4 No 200 Series PCH PCI Express Root Port
pcib16 0/29/0 No 200 Series PCH PCI Express Root Port
isab0 0/31/0 No Z370 Chipset LPC/eSPI Controller
none1 0/31/2 No 200 Series/Z370 Chipset Family Power Management Controller
hdac1 0/31/3 No 200 Series PCH HD Audio
ichsmb0 0/31/4 No 200 Series/Z370 Chipset Family SMBus Controller
em0 0/31/6 No Ethernet Connection (2) I219-V
vgapci0 1/0/0 No GP104GL [Quadro P5000]
hdac0 1/0/1 No GP104 High Definition Audio Controller
nvme0 2/0/0 No NVMe SSD Controller PM9A1/PM9A3/980PRO
nvme1 3/0/0 No NVMe SSD Controller SM981/PM981/PM983
pcib5 4/0/0 No PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
pcib6 5/1/0 No PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
pcib7 5/2/0 No PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
pcib8 5/3/0 No PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
pcib9 5/4/0 No PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
ppt0 6/0/0 Yes uPD720202 USB 3.0 Host Controller
ppt1 7/0/0 Yes uPD720202 USB 3.0 Host Controller
ppt2 8/0/0 Yes uPD720202 USB 3.0 Host Controller
ppt3 9/0/0 Yes uPD720202 USB 3.0 Host Controller
pcib12 11/0/0 No PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch
pcib13 12/1/0 No PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch
pcib14 12/2/0 No PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch
re0 13/0/0 No RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
re1 14/0/0 No RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
xhci1 15/0/0 No ASM2142/ASM3142 USB 3.1 Host Controller/boot/loader.conf
Code:
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
kern.racct.enable=1
autoboot_delay="5"
# Don't wait on USB enumeration
hw.usb.no_boot_wait="1"
# CPU microcode updates
cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"
# Use usbhid(4)
hw.usb.usbhid.enable="1" # Still necessary on FreeBSD 13.1+ ?
# VFS
vfs.usermount=1
# VMM
vmm_load="YES"
# PCI passthrough (bhyve)
pptdevs="6/0/0 7/0/0 8/0/0 9/0/0"VNC screenshot of when/where the guest gets stuck (notice the segment of the Windows bootloader progress thingy - it stops moving):
Any ideas?
I have since updated to a more recent stable/14 commit twice - no changes in behavior.
Changes to VM configuration such as CPU count, CPU topology or memory have no effect.
Also enabling/disabling the network config in the VM config doesn't change anything.
