bhyve IOMMU groups

bileslav · 2025-01-10T20:49:22+0000

On Linux, if you want to pass through a PCI(e) device to a VM, one of the first things to do is to check the IOMMU groups. If the device is not properly isolated, one can apply some hacky ACS patch to "fix" that. The Internet is full of information about all this.

In the FreeBSD world, on the other hand, there is almost no information of such thing as IOMMU groups. You are only told that you need to enable IOMMU in the firmware.

From lspci(8) (sysutils/pciutils):

IOMMU group that this device is part of (optional, Linux only).

I don't remember the details, but when I checked my IOMMU groups on Linux, it was pretty sad. My devices are not isolated as required. However, I successfully run Windows 10 and Linux in bhyve with these pptdevs:

Code:

ppt0@pci0:7:0:0:    class=0x010802 rev=0x01 hdr=0x00 vendor=0x2646 device=0x5013 subvendor=0x2646 subdevice=0x5013
    vendor     = 'Kingston Technology Company, Inc.'
    device     = 'KC3000/FURY Renegade NVMe SSD [E18]'
    class      = mass storage
    subclass   = NVM
ppt1@pci0:8:0:0:    class=0x0c0330 rev=0x04 hdr=0x00 vendor=0x1033 device=0x0194 subvendor=0x144d subdevice=0xc0a5
    vendor     = 'NEC Corporation'
    device     = 'uPD720200 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt2@pci0:10:0:0:    class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x220a subvendor=0x10de subdevice=0x1616
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 [GeForce RTX 3080 12GB]'
    class      = display
    subclass   = VGA
ppt3@pci0:10:0:1:    class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1aef subvendor=0x10de subdevice=0x1616
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA

IOMMU groups is something OS-agnostic, isn't it? Can anybody shed some light on how bhyve/FreeBSD handles this?

Phishfry · 2025-01-10T21:42:48+0000

I am not familiar with groups. I do bhyve passthru on a few machines. All Supermicro servers.
I don't see why passthru would need a group.
You are simply intercepting the pci device at boot and telling the system to not use it. Reserve for VMM.

I do have IIO assignments in BIOS under North Bridge.
IIO1 and IIO2
These are linked to each CPU in a dual processor machine.
This is where you set bifurcation of the PCIe slots.

When building a big NVMe 10 disk array I pondered how to RAID across the two IIO CPU groups.
One array per CPU group or span across for raid.

Utilizing the Intel® Xeon® Processor Scalable Family IIO Performance...

This article provides information on the Intel® Xeon® Processor Scalable Family IIO PerfMon events that count data bandwidth, explains how to run Int

www.intel.com

Phishfry · 2025-01-10T22:00:06+0000

After reading this I have a grasp. I can tell you we have nothing of the sort that I know of.

IOMMU Groups

It's not bad. For example I have some USB3 cards I pass thru and they have 4 controllers onboard all with pcie address.
It can be a pain finding which one goes where.

I guess you could group the whole USB3 card so you know what block of addresses it uses.

The video card example is a good one.

Indeed you need to pass thru the HDMI video and HDMI Audio. But its not that much extra work to add the PPT's.

Maxnix · 2025-01-10T22:11:57+0000

IIUC these groups are just a way to logically organize PCI(e) devices, nothing more nor less. According to the Arch wiki, your reserved devices should be organized in "groups" in this way:

ppt0, ppt1, ppt2 are separate groups (buses), and can be passed independently from each other.
ppt3 is part of ppt2, hence are part of the same group (share the same bus).

So to be able to use ppt3, you should include ppt2 too.
However, shouldn't ppt2 still be usable without ppt3? If on Linux you need to pass a whole group of devices isn't this less flexible? What If I need to just passthru the video card and do not need the associated audio controller (don't know if the host could still use it, maybe not...)?

P.S.
I see you are passing your NVMe SSD instead of assigning to the VM(s) the raw device in /dev, is this for performance reasons?

bileslav · 2025-01-10T22:18:23+0000

Phishfry, thanks for your input. So far, it looks like Linux is throwing a spanner in the works where FreeBSD is not. If this is indeed the case, that's great, but I'd like to make sure.

The aforementioned ACS patch causes Linux to lie that devices are properly isolated for passthrough, when in fact they are not. I guess that FreeBSD might do something similar by default.

Phishfry said:
Indeed you need to pass thru the HDMI video and HDMI Audio. But its not that much extra work to add the PPT's.

The problem is that not parts of the same device, but different devices, can end up in the same group. For example, two GPUs. Or a GPU and some expansion card. On Linux, my RTX 3080 is in the largest group, which has a ton of other stuff that I couldn't transfer to the VM. I don't have such problem in FreeBSD.

Phishfry · 2025-01-10T22:24:59+0000

We do not support VFIO from what I see.

What IOMMU seems to offer is isolation of devices on bus.

But with individual ppt addresses like we have each device is fully isolated I would think.
Grouping might speed that process up though.

bileslav · 2025-01-10T22:25:42+0000

Maxnix said:
I see you are passing your NVMe SSD instead of assigning to the VM(s) the raw device in /dev, is this for performance reasons?

Initially I passed it as /dev/nda1, but then I thought: why virtualize the NVMe thing when it's possible to pass through the raw PCI device?

Phishfry · 2025-01-10T22:28:42+0000

VFIO also seems to be the way the graphics cards are passed through on KVM/Linux.
I found this amusing that we are patching to look like KVM. How ridiculous.

https://dflund.se/~getz/Notes/2024/freebsd-gpu/

bileslav · 2025-01-10T22:33:08+0000

Phishfry said:
I found this amusing that we are patching to look like KVM. How ridiculus.

I agree. This patching is needed to make an NVIDIA driver work in Linux guests. In Windows 10, it works without patching. Windows 11 is just broken.

Phishfry · 2025-01-10T22:46:02+0000

I had sucess early with older x8 video cards GT710/720/730 and Bhyve passthrough.
I only use FreeBSD VM's so it worked. Maybe the old NVidia driver did not check for KVM sig. i dunno.

bileslav · 2025-01-10T22:58:29+0000

Phishfry said:
I had sucess early with older x8 video cards GT710/720/730 and Bhyve passthrough.
I only use FreeBSD VM's so it worked. Maybe the old NVidia driver did not check for KVM sig. i dunno.

I forgot to say, the NVIDIA driver in FreeBSD guests work without patching bhyve too, yep. I have one.

bileslav · 2025-01-10T23:13:20+0000

So, I just booted EndeavourOS (basically Arch Linux) live ISO.

Code:

IOMMU Group 0:
    00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
    00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
    01:00.0 Non-Volatile memory controller [0108]: INNOGRIT Corporation NVMe SSD Controller IG5236 [1dbe:5236] (rev 01)
    02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller [1022:43ee]
    02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller [1022:43eb]
    02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port [1022:43e9]
    03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
    03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
    03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
    03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
    04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c7)
    05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
    06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 XT/6500M] [1002:743f] (rev c7)
    06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
    07:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD E18 [2646:5013] (rev 01)
    08:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] (rev 04)
    09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 1:
    00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 2:
    00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
    00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
    0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080 12GB] [10de:220a] (rev a1)
    0a:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
IOMMU Group 3:
    00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
    00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
    00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 6:
    00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 7:
    00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
    00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 9:
    00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
    00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 10:
    00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
    00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
    00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
    00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
    00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
    00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
    00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
    00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 11:
    0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 12:
    0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 13:
    0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 14:
    0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

Code:

hostb0@pci0:0:0:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1480 subvendor=0x1022 subdevice=0x1480
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse Root Complex'
    class      = bridge
    subclass   = HOST-PCI
amdiommu0@pci0:0:0:2:    class=0x080600 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1481 subvendor=0x1022 subdevice=0x1481
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse IOMMU'
    class      = base peripheral
    subclass   = IOMMU
hostb1@pci0:0:1:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
pcib1@pci0:0:1:1:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1483 subvendor=0x1022 subdevice=0x1453
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse GPP Bridge'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:1:2:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1483 subvendor=0x1022 subdevice=0x1453
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse GPP Bridge'
    class      = bridge
    subclass   = PCI-PCI
hostb2@pci0:0:2:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
hostb3@pci0:0:3:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
pcib10@pci0:0:3:1:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1483 subvendor=0x1022 subdevice=0x1453
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse GPP Bridge'
    class      = bridge
    subclass   = PCI-PCI
hostb4@pci0:0:4:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
hostb5@pci0:0:5:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
hostb6@pci0:0:7:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
pcib11@pci0:0:7:1:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1484 subvendor=0x1022 subdevice=0x1484
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]'
    class      = bridge
    subclass   = PCI-PCI
hostb7@pci0:0:8:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
pcib12@pci0:0:8:1:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1484 subvendor=0x1022 subdevice=0x1484
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]'
    class      = bridge
    subclass   = PCI-PCI
intsmb0@pci0:0:20:0:    class=0x0c0500 rev=0x61 hdr=0x00 vendor=0x1022 device=0x790b subvendor=0x1849 subdevice=0xffff
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'FCH SMBus Controller'
    class      = serial bus
    subclass   = SMBus
isab0@pci0:0:20:3:    class=0x060100 rev=0x51 hdr=0x00 vendor=0x1022 device=0x790e subvendor=0x1849 subdevice=0xffff
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'FCH LPC Bridge'
    class      = bridge
    subclass   = PCI-ISA
hostb8@pci0:0:24:0:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1440 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 0'
    class      = bridge
    subclass   = HOST-PCI
hostb9@pci0:0:24:1:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1441 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 1'
    class      = bridge
    subclass   = HOST-PCI
hostb10@pci0:0:24:2:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1442 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 2'
    class      = bridge
    subclass   = HOST-PCI
hostb11@pci0:0:24:3:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1443 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 3'
    class      = bridge
    subclass   = HOST-PCI
hostb12@pci0:0:24:4:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1444 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 4'
    class      = bridge
    subclass   = HOST-PCI
hostb13@pci0:0:24:5:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1445 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 5'
    class      = bridge
    subclass   = HOST-PCI
hostb14@pci0:0:24:6:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1446 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 6'
    class      = bridge
    subclass   = HOST-PCI
hostb15@pci0:0:24:7:    class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1447 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse/Vermeer Data Fabric: Device 18h; Function 7'
    class      = bridge
    subclass   = HOST-PCI
nvme0@pci0:1:0:0:    class=0x010802 rev=0x01 hdr=0x00 vendor=0x1dbe device=0x5236 subvendor=0x1dbe subdevice=0x5236
    vendor     = 'INNOGRIT Corporation'
    device     = 'NVMe SSD Controller IG5236'
    class      = mass storage
    subclass   = NVM
xhci0@pci0:2:0:0:    class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x43ee subvendor=0x1b21 subdevice=0x1142
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = '500 Series Chipset USB 3.1 XHCI Controller'
    class      = serial bus
    subclass   = USB
ahci0@pci0:2:0:1:    class=0x010601 rev=0x00 hdr=0x00 vendor=0x1022 device=0x43eb subvendor=0x1b21 subdevice=0x1062
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = '500 Series Chipset SATA Controller'
    class      = mass storage
    subclass   = SATA
pcib3@pci0:2:0:2:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x43e9 subvendor=0x1b21 subdevice=0x0201
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = '500 Series Chipset Switch Upstream Port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:3:0:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x43ea subvendor=0x1b21 subdevice=0x3308
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:3:4:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x43ea subvendor=0x1b21 subdevice=0x3308
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    class      = bridge
    subclass   = PCI-PCI
pcib8@pci0:3:7:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x43ea subvendor=0x1b21 subdevice=0x3308
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    class      = bridge
    subclass   = PCI-PCI
pcib9@pci0:3:8:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x43ea subvendor=0x1b21 subdevice=0x3308
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:4:0:0:    class=0x060400 rev=0xc7 hdr=0x01 vendor=0x1002 device=0x1478 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Navi 10 XL Upstream Port of PCI Express Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:5:0:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1002 device=0x1479 subvendor=0x1002 subdevice=0x1479
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Navi 10 XL Downstream Port of PCI Express Switch'
    class      = bridge
    subclass   = PCI-PCI
vgapci0@pci0:6:0:0:    class=0x030000 rev=0xc7 hdr=0x00 vendor=0x1002 device=0x743f subvendor=0x1da2 subdevice=0xe458
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Navi 24 [Radeon RX 6400/6500 XT/6500M]'
    class      = display
    subclass   = VGA
hdac0@pci0:6:0:1:    class=0x040300 rev=0x00 hdr=0x00 vendor=0x1002 device=0xab28 subvendor=0x1002 subdevice=0xab28
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Navi 21/23 HDMI/DP Audio Controller'
    class      = multimedia
    subclass   = HDA
ppt0@pci0:7:0:0:    class=0x010802 rev=0x01 hdr=0x00 vendor=0x2646 device=0x5013 subvendor=0x2646 subdevice=0x5013
    vendor     = 'Kingston Technology Company, Inc.'
    device     = 'KC3000/FURY Renegade NVMe SSD [E18]'
    class      = mass storage
    subclass   = NVM
ppt1@pci0:8:0:0:    class=0x0c0330 rev=0x04 hdr=0x00 vendor=0x1033 device=0x0194 subvendor=0x144d subdevice=0xc0a5
    vendor     = 'NEC Corporation'
    device     = 'uPD720200 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
re0@pci0:9:0:0:    class=0x020000 rev=0x05 hdr=0x00 vendor=0x10ec device=0x8125 subvendor=0x1849 subdevice=0x8125
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8125 2.5GbE Controller'
    class      = network
    subclass   = ethernet
ppt2@pci0:10:0:0:    class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x220a subvendor=0x10de subdevice=0x1616
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 [GeForce RTX 3080 12GB]'
    class      = display
    subclass   = VGA
ppt3@pci0:10:0:1:    class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1aef subvendor=0x10de subdevice=0x1616
    vendor     = 'NVIDIA Corporation'
    device     = 'GA102 High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA
none0@pci0:11:0:0:    class=0x130000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x148a subvendor=0x1022 subdevice=0x148a
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse PCIe Dummy Function'
    class      = non-essential instrumentation
none1@pci0:12:0:0:    class=0x130000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1485 subvendor=0x1022 subdevice=0x1485
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse Reserved SPP'
    class      = non-essential instrumentation
none2@pci0:12:0:1:    class=0x108000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1486 subvendor=0x1022 subdevice=0x1486
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Starship/Matisse Cryptographic Coprocessor PSPCPP'
    class      = encrypt/decrypt
xhci1@pci0:12:0:3:    class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x149c subvendor=0x1849 subdevice=0xffff
    vendor     = 'Advanced Micro Devices, Inc. [AMD]'
    device     = 'Matisse USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB

Memory failed me when I said that my RTX 3080 is in the "bad" IOMMU group on Linux. As you can see, it's actually in IOMMU Group 2, and it's properly isolated. But these two are in IOMMU Group 0, which is where the whole bunch of stuff is, and that wouldn't let me pass through them without patching Linux:

Code:

07:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD E18 [2646:5013] (rev 01)
08:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] (rev 04)

I find it suspicious that FreeBSD just doesn't have this problem. This is what I'm trying to figure out. I don't think VFIO thing has anything to do with this.

bileslav · 2025-01-10T23:19:34+0000

IIUC, that Linux ACS patch only leads to security issues. I've never tried passthrough on Linux, too lazy to deal with KVM after bhyve.

bileslav · 2025-01-11T12:50:05+0000

In the past, I've seen here some very useful insights on bhyve from PMc: perhaps you could give some advice?

If no one can clarify, I plan to write to freebsd-virtualization@freebsd.org, and if I get a response, I'll post it in this thread.