bhyve bhyve PCI Passthrough Hard drives and PCI device, I can't do both at same time

I am using "churchers" bhyve on FreeBSD 12.2-RELEASE-p3 running on a x570 board with Ryzen R9 CPU with 64GB of ram, I want to have my "windows vm" access both a pass true device and 4 physical hard drives. I can get it to do one or the other but not both at once. Below is my config for the VM: (as is my physical hard drive do not show up, If I comment out passthru0="6/0/0" my drives show in explorer.)
Code:
loader="uefi"
cpu=8
memory=12G
bhyve_options="-S"
graphics="yes"
xhci_mouse="yes"
graphics_res="1600x900"
network0_type="virtio-net"
network0_switch="deplorable"
ahci_device_limit="8"
debug="yes"
disk0_opts="sectorsize=512"
disk0_type="ahci-hd"
disk0_name="BackupR7.img"
disk1_dev="custom"
disk1_opts="sectorsize=512"
disk1_type="ahci-hd"
disk1_name="/dev/ada0,hd:/dev/ada1"
disk2_dev="custom"
disk2_opts="sectorsize=512"
disk2_type="ahci-hd"
disk2_name="/dev/ada3,hd:/dev/ada4"
uuid="762f78e9-270b-11e8-87a4-309c230c9abc"
network0_mac="02:15:5d:45:32:01"
passthru0="6/0/0"

This results of pciconf -lv
Code:
hostb0@pci0:0:0:0: class=0x060000 card=0x87c01043 chip=0x14801022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse Root Complex'
class = bridge
subclass = HOST-PCI
none0@pci0:0:0:2: class=0x080600 card=0x87c01043 chip=0x14811022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse IOMMU'
class = base peripheral
subclass = IOMMU
hostb1@pci0:0:1:0: class=0x060000 card=0x00000000 chip=0x14821022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
pcib1@pci0:0:1:1: class=0x060400 card=0x87c01043 chip=0x14831022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib2@pci0:0:1:2: class=0x060400 card=0x87c01043 chip=0x14831022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse GPP Bridge'
class = bridge
subclass = PCI-PCI
hostb2@pci0:0:2:0: class=0x060000 card=0x00000000 chip=0x14821022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
hostb3@pci0:0:3:0: class=0x060000 card=0x00000000 chip=0x14821022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
pcib13@pci0:0:3:1: class=0x060400 card=0x87c01043 chip=0x14831022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse GPP Bridge'
class = bridge
subclass = PCI-PCI
hostb4@pci0:0:4:0: class=0x060000 card=0x00000000 chip=0x14821022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
hostb5@pci0:0:5:0: class=0x060000 card=0x00000000 chip=0x14821022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
hostb6@pci0:0:7:0: class=0x060000 card=0x00000000 chip=0x14821022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
pcib14@pci0:0:7:1: class=0x060400 card=0x87c01043 chip=0x14841022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]'
class = bridge
subclass = PCI-PCI
hostb7@pci0:0:8:0: class=0x060000 card=0x00000000 chip=0x14821022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
pcib15@pci0:0:8:1: class=0x060400 card=0x87c01043 chip=0x14841022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]'
class = bridge
subclass = PCI-PCI
intsmb0@pci0:0:20:0: class=0x0c0500 card=0x87c01043 chip=0x790b1022 rev=0x61 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH SMBus Controller'
class = serial bus
subclass = SMBus
isab0@pci0:0:20:3: class=0x060100 card=0x87c01043 chip=0x790e1022 rev=0x51 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH LPC Bridge'
class = bridge
subclass = PCI-ISA
hostb8@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x14401022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 0'
class = bridge
subclass = HOST-PCI
hostb9@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x14411022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 1'
class = bridge
subclass = HOST-PCI
hostb10@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x14421022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 2'
class = bridge
subclass = HOST-PCI
hostb11@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x14431022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 3'
class = bridge
subclass = HOST-PCI
hostb12@pci0:0:24:4: class=0x060000 card=0x00000000 chip=0x14441022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 4'
class = bridge
subclass = HOST-PCI
hostb13@pci0:0:24:5: class=0x060000 card=0x00000000 chip=0x14451022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 5'
class = bridge
subclass = HOST-PCI
hostb14@pci0:0:24:6: class=0x060000 card=0x00000000 chip=0x14461022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 6'
class = bridge
subclass = HOST-PCI
hostb15@pci0:0:24:7: class=0x060000 card=0x00000000 chip=0x14471022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Device 24: Function 7'
class = bridge
subclass = HOST-PCI
nvme0@pci0:1:0:0: class=0x010802 card=0x2263126f chip=0x2263126f rev=0x03 hdr=0x00
vendor = 'Silicon Motion, Inc.'
device = 'SM2263EN/SM2263XT SSD Controller'
class = mass storage
subclass = NVM
pcib3@pci0:2:0:0: class=0x060400 card=0x00000000 chip=0x57ad1022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse Switch Upstream'
class = bridge
subclass = PCI-PCI
pcib4@pci0:3:1:0: class=0x060400 card=0x87c01043 chip=0x57a31022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib5@pci0:3:2:0: class=0x060400 card=0x87c01043 chip=0x57a31022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib6@pci0:3:3:0: class=0x060400 card=0x87c01043 chip=0x57a31022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib7@pci0:3:4:0: class=0x060400 card=0x87c01043 chip=0x57a31022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib8@pci0:3:5:0: class=0x060400 card=0x87c01043 chip=0x57a31022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib9@pci0:3:6:0: class=0x060400 card=0x87c01043 chip=0x57a31022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib10@pci0:3:8:0: class=0x060400 card=0x87c01043 chip=0x57a41022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib11@pci0:3:9:0: class=0x060400 card=0x87c01043 chip=0x57a41022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib12@pci0:3:10:0: class=0x060400 card=0x87c01043 chip=0x57a41022 rev=0x00 hdr=0x01
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse PCIe GPP Bridge'
class = bridge
subclass = PCI-PCI
nvme1@pci0:4:0:0: class=0x010802 card=0x50121987 chip=0x50121987 rev=0x01 hdr=0x00
vendor = 'Phison Electronics Corporation'
device = 'E12 NVMe Controller'
class = mass storage
subclass = NVM
ppt0@pci0:5:0:0: class=0x0c0330 card=0xffffffff chip=0x00141912 rev=0x03 hdr=0x00
vendor = 'Renesas Technology Corp.'
device = 'uPD720201 USB 3.0 Host Controller'
class = serial bus
subclass = USB
ppt1@pci0:6:0:0: class=0x0c0330 card=0x2104174c chip=0x10421b21 rev=0x00 hdr=0x00
vendor = 'ASMedia Technology Inc.'
device = 'ASM1042 SuperSpeed USB Host Controller'
class = serial bus
subclass = USB
ahci0@pci0:7:0:0: class=0x010601 card=0x10601b21 chip=0x06121b21 rev=0x01 hdr=0x00
vendor = 'ASMedia Technology Inc.'
device = 'ASM1062 Serial ATA Controller'
class = mass storage
subclass = SATA
re0@pci0:8:0:0: class=0x020000 card=0x86771043 chip=0x816810ec rev=0x15 hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet
re1@pci0:9:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x07 hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet
none1@pci0:10:0:0: class=0x130000 card=0x87c01043 chip=0x14851022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse Reserved SPP'
class = non-essential instrumentation
xhci0@pci0:10:0:1: class=0x0c0330 card=0x87c01043 chip=0x149c1022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse USB 3.0 Host Controller'
class = serial bus
subclass = USB
xhci1@pci0:10:0:3: class=0x0c0330 card=0x148c1022 chip=0x149c1022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse USB 3.0 Host Controller'
class = serial bus
subclass = USB
ahci1@pci0:11:0:0: class=0x010601 card=0x79011022 chip=0x79011022 rev=0x51 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH SATA Controller [AHCI mode]'
class = mass storage
subclass = SATA
ahci2@pci0:12:0:0: class=0x010601 card=0x79011022 chip=0x79011022 rev=0x51 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH SATA Controller [AHCI mode]'
class = mass storage
subclass = SATA
vgapci0@pci0:13:0:0: class=0x030000 card=0x853f1043 chip=0x13c210de rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GM204 [GeForce GTX 970]'
class = display
subclass = VGA
hdac0@pci0:13:0:1: class=0x040300 card=0x853f1043 chip=0x0fbb10de rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GM204 High Definition Audio Controller'
class = multimedia
subclass = HDA
none2@pci0:14:0:0: class=0x130000 card=0x87c01043 chip=0x148a1022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse PCIe Dummy Function'
class = non-essential instrumentation
none3@pci0:15:0:0: class=0x130000 card=0x87c01043 chip=0x14851022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse Reserved SPP'
class = non-essential instrumentation
none4@pci0:15:0:1: class=0x108000 card=0x87c01043 chip=0x14861022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse Cryptographic Coprocessor PSPCPP'
class = encrypt/decrypt
xhci2@pci0:15:0:3: class=0x0c0330 card=0x87c01043 chip=0x149c1022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Matisse USB 3.0 Host Controller'
class = serial bus
subclass = USB
hdac1@pci0:15:0:4: class=0x040300 card=0x87bb1043 chip=0x14871022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Starship/Matisse HD Audio Controller'
class = multimedia
subclass = HAD






Note: This Is passed to another windows VM and works well.
Code:
ppt0@pci0:5:0:0: class=0x0c0330 card=0xffffffff chip=0x00141912 rev=0x03 hdr=0x00
vendor = 'Renesas Technology Corp.'
device = 'uPD720201 USB 3.0 Host Controller'
class = serial bus
subclass = USB
 
I'm not sure, do not think the vm is attached to specific controller (I did not passthrough like I did with the pcie card). they are accessed with this:


disk1_dev="custom"
disk1_opts="sectorsize=512"
disk1_type="ahci-hd"
disk1_name="/dev/ada0,hd:/dev/ada1"
disk2_dev="custom"
disk2_opts="sectorsize=512"
disk2_type="ahci-hd"
disk2_name="/dev/ada3,hd:/dev/ada4"

I assume here: (OS is on a nvme) They seem to be the only ones on the MB itself)

ahci1@pci0:11:0:0: class=0x010601 card=0x79011022 chip=0x79011022 rev=0x51 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH SATA Controller [AHCI mode]'
class = mass storage
subclass = SATA
ahci2@pci0:12:0:0: class=0x010601 card=0x79011022 chip=0x79011022 rev=0x51 hdr=0x00
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH SATA Controller [AHCI mode]'
class = mass storage
subclass = SATA

camcontrol devlist
<HL-DT-ST BD-RE BH16NS40 1.03> at scbus1 target 0 lun 0 (cd0,pass0) on separate controller (add in card)
<TOSHIBA MD04ACA400 FP2A> at scbus2 target 0 lun 0 (ada0,pass1) this
<MD4000GSA6472E 01.01AX1> at scbus3 target 0 lun 0 (ada1,pass2) this
<SAMSUNG HD103UJ 1AA01113> at scbus5 target 0 lun 0 (ada2,pass3) is also connected but not used by the vm
<HGST HDN726040ALE614 APGNW7JH> at scbus6 target 0 lun 0 (ada3,pass4)
this
<ST5000DM000-1FK178 CC44> at scbus7 target 0 lun 0 (ada4,pass5) this
<Generic STORAGE DEVICE 9451> at scbus8 target 0 lun 0 (pass6,da0)

Does any of this help?
 
Last edited:
Having another look at the passthrough settings but 6/0/0 appears to be a USB host controller, not a disk controller:
Code:
ppt1@pci0:6:0:0: class=0x0c0330 card=0x2104174c chip=0x10421b21 rev=0x00 hdr=0x00
vendor = 'ASMedia Technology Inc.'
device = 'ASM1042 SuperSpeed USB Host Controller'
class = serial bus
subclass = USB[

Besides the NVMe controller (nvme1@pci0:4:0:0) and the two FCH AMD controllers (ahci1@pci0:11:0:0 and ahci1@pci0:12:0:0) you also have:
Code:
ahci0@pci0:7:0:0: class=0x010601 card=0x10601b21 chip=0x06121b21 rev=0x01 hdr=0x00
vendor = 'ASMedia Technology Inc.'
device = 'ASM1062 Serial ATA Controller'
class = mass storage
subclass = SATA

You need to configure the VM for two controllers, one is the passthrough and the other is a virtual controller that's connected to ada0 to ada3. I'm honestly not sure if vm(8) allows you to do that. From bhyve(8)'s point of view it should be possible but it might be that the vm(1) scripts simply assume there's always just one disk controller.

Edit: I just realized I have one VM with something similar. The trick seems to be to use passthru0="2/0/0=2:0". The =2:0 tells it to use a second controller for this. You will need to adjust the 2/0/0 for your situation of course.
 
Back
Top