VT-d supported, but no IOMMU?

I

ibrewster

I have an intel DZ77GA-70K board with an intel core i7-3770 processor. Everything I can find online says this setup supports VT-d, for example the intel ark page on that processor (https://www.intel.com/content/www/u...r-8m-cache-up-to-3-90-ghz/specifications.html), and this post about the motherboard: https://www.overclock.net/threads/vt-d-compatible-motherboards.1338063/. I checked the bios and "Intel Virtualization Technology" and "Intel VT for Directed I/O (VT-d)" are both enabled. /boot/loader.conf has the line
Code: 
intel_iommu="on"

However, in spite of all of this, running
Code: 
acpidump -t | grep DMAR
returns only
Code: 
acpidump: DSDT is corrupt

I'm trying to use PCI passthrough with a PCI USB card and bhyve to give one of my VM's access to a USB device, but the lack of the DMAR tables seems to be the sticking point at the moment.

What am I missing?

I'm sure I have omitted some vital information, so please ask for anything I missed. Thank you!
 
ibrewster said:
What am I missing?

I'm sure I have omitted some vital information, so please ask for anything I missed. Thank you!
Click to expand...
Hi. Missing here is any information about your having read any FreeBSD-related sources on how to enable/use passthru on FreeBSD. Good chances are, then, that your failure here is due to not properly using FreeBSD tools for that. Like this, for example.
I, personally, am not aware of a setting like "intel_iommu=on", although I've been using pci passthru for a while now... where did you take it from, I wonder?
EDIT: oh I see, it's from Linux :D
 
free-and-bsd said:
Missing here is any information about your having read any FreeBSD-related sources on how to enable/use passthru on FreeBSD
Click to expand...
You're right - I didn't mention the many multiple hours I spent reading over every page and document I could find on the subject, most of which were quite unhelpful. For example, at the beginning of the document you linked, it states:
Host VT-d support can be determined by searching for a DMAR table in the ACPI tables with acpidump -t | grep DMAR
Click to expand...
However, as I pointed out in my post, this doesn't work:
ibrewster said:
running
Code: 
acpidump -t | grep DMAR
returns only
Code: 
acpidump: DSDT is corrupt
Click to expand...
even though my host *DOES* support VT-d, as I pointed out (also showing I spent considerable time researching this before posting). Then, later in that same document, it gives directions for various flags to pass to the bhyve command, but no indication of what to put in the "vm configure" for the VM so you can just start it using "vm start <vmname>". I had to piece that together myself from various OTHER documents I read on the subject (apparently "vm" is a separate tool for managing bhyve vm's, but you can also use the "bhyve" command directly, which is all that document refers to). But that's not the issue, the lack of the DMAR table is.

free-and-bsd said:
I, personally, am not aware of a setting like "intel_iommu=on", although I've been using pci passthru for a while now... where did you take it from, I wonder?
EDIT: oh I see, it's from Linux :D
Click to expand...
Then why was it in my /boot/loader.conf? I certainly didn't put it there! You're right though - *most* of the references I found for this setting _were_ linux - probably because in FreeBSD it is on by default so you don't need to mess with it.
 
ibrewster said:
I'm trying to use PCI passthrough with a PCI USB card and bhyve to give one of my VM's access to a USB device
Click to expand...
What is the USB card or chipset?

Does it show up in pciconf as a ppt device after you pass it thru?

Just for clarification is this an actual PCI USB card or PCIe USB card?
 
Much like this thread I am afraid your DMAR quest may be in vein. Not needed.

Emrion said:
acpidump -t | grep DMAR gives nothing with a i5-3570 and a MSI B75MA-P45 mainboard. Yet, this processor has VT-d capability and passthru in bhyve works.
Click to expand...
 
Phishfry said:
What is the USB card or chipset?
Click to expand...
pciconf shows it as a "VL805/806 xHCI USB 3.0 Controller"
Phishfry said:
Does it show up in pciconf as a ppt device after you pass it thru?
Click to expand...
Yes. From pciconf -lv:
ppt0@pci0:55:0:0: class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x1106 device=0x3483 subvendor=0x1106 subdevice=0x3483
vendor = 'VIA Technologies, Inc.'
device = 'VL805/806 xHCI USB 3.0 Controller'
class = serial bus
subclass = USB
Click to expand...

And also "vm passthru":
DEVICE BHYVE ID READY DESCRIPTION
...
ppt0 55/0/0 Yes VL805/806 xHCI USB 3.0 Controller
Click to expand...

Also, for completeness sake, "pciconf -lc | grep MSI":

cap 05[90] = MSI supports 1 message
cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message
cap 05[8c] = MSI supports 1 message, 64 bit
cap 05[8c] = MSI supports 1 message, 64 bit
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
cap 05[80] = MSI supports 1 message
cap 05[80] = MSI supports 1 message
cap 05[80] = MSI supports 1 message
cap 05[80] = MSI supports 1 message enabled with 1 message
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint MSI 1 max data 128(256)
cap 11[b0] = MSI-X supports 32 messages, enabled
cap 05[d0] = MSI supports 1 message, 64 bit
cap 11[a0] = MSI-X supports 5 messages, enabled
cap 05[48] = MSI supports 4 messages, 64 bit, vector masks
cap 05[48] = MSI supports 4 messages, 64 bit, vector masks
cap 05[48] = MSI supports 4 messages, 64 bit, vector masks
cap 05[48] = MSI supports 4 messages, 64 bit, vector masks
cap 05[48] = MSI supports 4 messages, 64 bit, vector masks
cap 05[48] = MSI supports 4 messages, 64 bit, vector masks
cap 05[90] = MSI supports 4 messages, 64 bit
Click to expand...
So that doesn't appear to be the issue. Unfortunately, when attempting to start the vm using the command "vm start <vmname>", the start attempt fails almost immediately, with the log showing the following:

Dec 24 07:23:11: initialising
Dec 24 07:23:11: [loader: uefi]
Dec 24 07:23:11: [cpu: 2]
Dec 24 07:23:11: [memory: 4GB]
Dec 24 07:23:11: [hostbridge: standard]
Dec 24 07:23:11: [com ports: com1]
Dec 24 07:23:11: [uuid: fbd5d9c1-97ba-11ee-a9c6-00224d9d7b76]
Dec 24 07:23:11: [debug mode: no]
Dec 24 07:23:11: [primary disk: disk0.img]
Dec 24 07:23:11: [primary disk dev: file]
Dec 24 07:23:11: fatal; pci passthrough not supported on this system (no VT-d or amdvi)
Click to expand...

Which is what led me to that first line in the bhyve pci_passthru document, and my search for the DMAR tables.

If, in fact, that line in the documentation is in error, then I am really at a loss as to what I am missing here.
 
No, all PCIe on the board, and this is a PCIe device (https://www.amazon.com/dp/B07FRFJ165) - although I notice now that the listing says "mini PCI-E", perhaps that's an issue? But then why am I getting the error about "no VT-d support"?. I have no idea how the bus numbers are assigned, but here is the full output from vm passthru, if that helps anything, where I notice many devices up that high:

Bash: 
DEVICE     BHYVE ID     READY        DESCRIPTION
hostb0     0/0/0        No           Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
vgapci0    0/2/0        No           IvyBridge GT2 [HD Graphics 4000]
xhci0      0/20/0       No           7 Series/C210 Series Chipset Family USB xHCI Host Controller
none0      0/22/0       No           7 Series/C216 Chipset Family MEI Controller
none1      0/22/1       No           7 Series/C210 Series Chipset Family MEI Controller
em0        0/25/0       No           82579V Gigabit Network Connection
ehci0      0/26/0       No           7 Series/C216 Chipset Family USB Enhanced Host Controller
hdac0      0/27/0       No           7 Series/C216 Chipset Family High Definition Audio Controller
pcib1      0/28/0       No           7 Series/C216 Chipset Family PCI Express Root Port 1
pcib2      0/28/4       No           7 Series/C210 Series Chipset Family PCI Express Root Port 5
pcib3      0/28/5       No           7 Series/C210 Series Chipset Family PCI Express Root Port 6
ehci1      0/29/0       No           7 Series/C216 Chipset Family USB Enhanced Host Controller
isab0      0/31/0       No           Z77 Express Chipset LPC Controller
ahci0      0/31/2       No           7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
ichsmb0    0/31/3       No           7 Series/C216 Chipset Family SMBus Controller
re0        1/0/0        No           RTL8125 2.5GbE Controller
em1        50/0/0       No           82574L Gigabit Network Connection
pcib4      51/0/0       No           PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
pcib5      52/1/0       No           PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
pcib6      52/4/0       No           PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
pcib7      52/5/0       No           PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
pcib8      52/7/0       No           PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
pcib9      52/9/0       No           PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
ppt0       55/0/0       Yes          VL805/806 xHCI USB 3.0 Controller
pcib10     57/0/0       No           IT8892E PCIe to PCI Bridge
none2      58/2/0       No           TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
 
ibrewster said:
"mini PCI-E", perhaps that's an issue?
Click to expand...
No that is same bus standard different form factor.

This SkullTrail board should have all the bells and whistles for a desktop/gaming board.

Normally Intel tops out at 0:31:x for motherboard devices like USB jacks.

I see with PEX it shoves stuff up.
It seems to start normal at 1:0:0 (These are your traditional PCIe address's for stuff)
Weirdly instead of next device starting at 2:0:0 it jumps to device 50:0:0
That I have never ever seen. It could be the problem.
I think that PEX chip is messing with PCI bus addresses.
 
Question: Is the ethernet contoller em1 on 50:0:0 onboard or in a PCIe slot or PCI slot?
Everything jumps to a high address after this device.
 
ibrewster said:
You're right - I didn't mention the many multiple hours I spent reading over every page and document I could find on the subject, most of which were quite unhelpful. For example, at the beginning of the document you linked, it states:

However, as I pointed out in my post, this doesn't work:

even though my host *DOES* support VT-d, as I pointed out (also showing I spent considerable time researching this before posting). Then, later in that same document, it gives directions for various flags to pass to the bhyve command, but no indication of what to put in the "vm configure" for the VM so you can just start it using "vm start <vmname>". I had to piece that together myself from various OTHER documents I read on the subject (apparently "vm" is a separate tool for managing bhyve vm's, but you can also use the "bhyve" command directly, which is all that document refers to). But that's not the issue, the lack of the DMAR table is.


Then why was it in my /boot/loader.conf? I certainly didn't put it there! You're right though - *most* of the references I found for this setting _were_ linux - probably because in FreeBSD it is on by default so you don't need to mess with it.
Click to expand...
How would anybody know you did it all if you don't say anything? Nor how the "iommu" directive ended up in your /boot/loader.conf .

Anyway, as yours is a PCIe USB card, those cards don't ALL play well with bhyve passthru -- which I know from personal experience. First I tried to passthru my on-board VIA USB controller -- at the time it caused my machine to reboot!! Had to buy an expansion PCIe USB card, which worked for a while... until it started causing problems in its turn. I removed it, but luckily for me, the first one got more support this time (the bhyve development was going on, thnx to the devs).

So the problem may very well be not in your mobo, but in the card itself.
 
It looks like at least part of the issue is with vm-bhyve. After digging through the code that gives me the error, it appears that because the acpidump command doesn't produce the expected response, vm-bhyve simply outputs the error and exits, never even trying.

So instead I tried the following bhyve command directly (which probably has some errors in it):

Bash: 
bhyve -c 2 -m 4GB -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-S -U fbd5d9c1-97ba-11ee-a9c6-00224d9d7b76 \
-u -S \
-s 0,hostbridge \
-s 31,lpc \
-s 4:0,ahci-hd,/zroot/vm/home-assistant/disk0.img \
-s 5:0,virtio-net,tap2,mac=58:9c:fc:0a:da:ee \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1280,h=720,wait \
-s 7,passthru,55/0/0 \
-l com1,stdio home-assistant

This didn't give me any errors, but also didn't actually boot the VM... first I got the output "fbuf frame buffer base: 0x19cb0fc00000 [sz 16777216]", then when I tried to connect via VNC, I got a brief flash of the grub boot loader followed by "Booting `Slot A (rescue shell)'"
Though if I remove the passthru option from the bhyve command above, then after trying to connect VNC it does boot. So maybe there *IS* something with this particular USB card. Sigh.
 
Did you try to passthru the on-board USB bus? At least to see how it would work. I'm sure the bhyve message about VT-d "not being supported" doesn't mean much. It only reports the results of its own efforts to do what it's meant to DO with whatever resources are made available to it by other mechanisms. Among other things, by the vmm driver (am I right?), which is why it has to be loaded at boot via /boot/loader.conf .
 
free-and-bsd said:
Did you try to passthru the on-board USB bus? At least to see how it would work
Click to expand...
That's a thought - just use the card for the "host" usb, and passthrough the on-board USB for the VM...if that works, of course! Let me try it and get back to you.
free-and-bsd said:
I'm sure the bhyve message about VT-d "not being supported" doesn't mean much
Click to expand...
Yeah, probably not...except for the fact that when it gets that error, it immediately terminates the attempt to start the VM. The relevant code block from vm-run is this:

Bash: 
    if [ -n "${_passdev}" ] && ! util::check_bhyve_iommu; then
        util::log "guest" "${_name}" "fatal; pci passthrough not supported on this system (no VT-d or amdvi)"
        exit 15
    fi
 
free-and-bsd said:
Did you try to passthru the on-board USB bus? At least to see how it would work
Click to expand...
Well, wudja know...that seems to work! Hallelujah! Thank you, thank you, thank you!

I can live with that easily enough. Let the VM use the onboard USB, and the host can use the card. Not the original intent, but it works, so I'm not going to complain!

Thanks again!

EDIT: Actually, further testing shows I only need to passthrough one of the three on-board USB devices (some threads seemed to indicate they were a unit and would either all need to be passed through or none), which means I don't even need the PCIe card - between the three controllers, I have plenty of USB ports to go around.
 
ibrewster said:
Well, wudja know...that seems to work! Hallelujah! Thank you, thank you, thank you!

I can live with that easily enough. Let the VM use the onboard USB, and the host can use the card. Not the original intent, but it works, so I'm not going to complain!

Thanks again!

EDIT: Actually, further testing shows I only need to passthrough one of the three on-board USB devices (some threads seemed to indicate they were a unit and would either all need to be passed through or none), which means I don't even need the PCIe card - between the three controllers, I have plenty of USB ports to go around.
Click to expand...
Exactly! And I am also using one of those z77 mobos, and passing thru of USB 3.0 part works fine!
And BTW, you can click Thanks there LOL.
 
You must log in or register to reply here.
Back
Top